{"id":30699951,"url":"https://github.com/structuredllm/itergen","last_synced_at":"2025-09-02T11:43:26.634Z","repository":{"id":283533836,"uuid":"937740578","full_name":"structuredllm/itergen","owner":"structuredllm","description":"Iterate on LLM-based structured generation forward and backward","archived":false,"fork":false,"pushed_at":"2025-03-20T04:21:48.000Z","size":4948,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-20T18:49:05.773Z","etag":null,"topics":["grammar","llm","llm-inference","parser"],"latest_commit_sha":null,"homepage":"https://structuredllm.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/structuredllm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-23T19:33:30.000Z","updated_at":"2025-03-20T17:08:52.000Z","dependencies_parsed_at":"2025-03-20T18:49:17.653Z","dependency_job_id":"4b583491-ef9a-4214-8a6b-14781769a102","html_url":"https://github.com/structuredllm/itergen","commit_stats":null,"previous_names":["structuredllm/itergen"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/structuredllm/itergen","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/structuredllm%2Fitergen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/structuredllm%2Fitergen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/structuredllm%2Fitergen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/structuredllm%2Fitergen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/structuredllm","download_url":"https://codeload.github.com/structuredllm/itergen/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/structuredllm%2Fitergen/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273279809,"owners_count":25077318,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-02T02:00:09.530Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["grammar","llm","llm-inference","parser"],"created_at":"2025-09-02T11:43:23.420Z","updated_at":"2025-09-02T11:43:26.596Z","avatar_url":"https://github.com/structuredllm.png","language":"Python","funding_links":[],"categories":["Libraries"],"sub_categories":[],"readme":"## IterGen: Iterative structured LLM Generation with Backtracking\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"400\" alt=\"syncode\" src=\"https://github.com/user-attachments/assets/0845333b-8c54-4887-abd0-b4a7550fb8a6\" /\u003e\n\u003c/p\u003e\n\nIterGen enables precise control of text generation by integrating grammar rules and offering three core functions: `forward`, `backward`, and `view`. These functions allow programs to navigate the generation process with fine-grained control while decoding with Large Language Models (LLMs).\n\n## Key Features\n\n1. **Grammar-Aware Decoding**  \n   Define custom grammars to structure generation using terminal and non-terminal symbols. IterGen is built on top of [SynCode](https://github.com/uiuc-focal-lab/syncode), a grammar-constrained generation framework.\n\n2. **Forward Generation**  \n   The `forward` function generates content up to a specified count of grammar symbols (e.g., sentences, words) or until meeting stopping conditions such as EOS tokens or a maximum token limit. Each call supports customized decoding parameters like temperature and sampling methods.\n\n3. **Backtracking Support**  \n   The `backward` function removes generated content by a specified count of grammar symbols, allowing corrections and refinements in real time.\n\n4. **Output Inspection**  \n   The `view` function retrieves parts of the generated text corresponding to specific grammar symbols, enabling validation or further analysis.\n\n---\n### Python Installation and Usage Instructions\nSimply install SynCode via PyPi using the following command:\n``` bash\npip install git+https://github.com/uiuc-arc/itergen.git\n```\n\n\n## 📜 Citation\n\u003cp\u003e\n    \u003ca href=\"https://arxiv.org/abs/2410.07295\"\u003e\u003cimg src=\"https://img.shields.io/badge/Paper-arXiv-blue\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n```\n@misc{ugare2024itergeniterativestructuredllm,\n      title={IterGen: Iterative Structured LLM Generation}, \n      author={Shubham Ugare and Rohan Gumaste and Tarun Suresh and Gagandeep Singh and Sasa Misailovic},\n      year={2024},\n      eprint={2410.07295},\n      archivePrefix={arXiv},\n      primaryClass={cs.SE},\n      url={https://arxiv.org/abs/2410.07295}, \n}\n```\n\n## Example: Using IterGen \n\n```python\nfrom itergen.main import IterGen\n\n# Define the grammar\ngrammar = \"\"\"\nparagraph: sentence+\nsentence: word+ sentence_end\nword: /[a-zA-Z0-9]+/ | other_punctuations\nsentence_end: \".\" | \"!\" | \"?\"\nother_punctuations: \",\" | \";\" | \":\" | \"'\" | \"\\\"\"\n%ignore \" \"\n\"\"\"\n\n# Initialize IterGen with the grammar and a model with Hugging Face model ID\niter_gen = IterGen(grammar=grammar, model_id=\"microsoft/Phi-3-mini-128k-instruct\")\nprompt = \"Once upon a time\"\n\n# Start generation\niter_gen.start(prompt)\n\n# Generate one sentence\ngenerated_sentence = iter_gen.forward(stop_symbol=\"sentence\", num=1)\nprint(\"Generated Sentence:\", generated_sentence)\n\n# Backtrack by 2 words\niter_gen.backward(\"word\", num=2)\n\n# Inspect all words in the current generation\ncurrent_words = iter_gen.view(\"word\")\nprint(\"Current Words:\", current_words)\n```\n\n\n## Case Studies\nThe following case studies demonstrate the utility of IterGen in various applications as described in the [paper](https://arxiv.org/abs/2410.07295).\n\n\u003cdetails\u003e\n  \u003csummary\u003e SQL Case Study\u003c/summary\u003e\n\nIterGen can be used to improve SQL generation by backtracking and correcting errors when the table names and column names are not present in the database schema. The SQL case study demonstrates how IterGen improves the accuracy of SQL generation by correcting errors in the generated SQL code.\n\n```python\n\nTo run all SQL experiments, use the following command:\n```bash\npython3 case_studies/sql/eval_sql.py\n```\n\n### Results\nThe results of the experiments will be stored in the `results/sql_results/results.jsonl` file. Each line in this file contains detailed information about the performance of a model on different SQL tasks.\n\nField Explanation Table:\n\n| Field | Description |\n| --- | --- |\n| `model_id` | Model id used for the experiment |\n| `evaluation_mode` | Evaluation mode used for the experiment |\n| `temperature` | Temperature used for the experiment |\n| `recurrence_penalty` | Recurrence penalty used for the experiment |\n| `seed` | Seed used for the experiment |\n| `execution_accuracy` | Accuracy of the model on different difficulty levels (easy, medium, hard, extra, all) |\n| `Counts` | Number of problems in each difficulty level |\n| `error_types` | Error types of the model |\n| `results_jsonl_file` | Path to the result jsonl file |\n| `avg_num_tokens` | Average number of tokens in the generated code |\n| `num_of_problems` | Number of problems in the experiment |\n| `Average time` | Average time taken to generate the code |\n\nAn example of a result entry looks like this:\n```javascript\n{\n    'model_id': 'model_id', \n    'evaluation_mode': 'itergen', \n    'temperature': None, \n    'recurrence_penalty': 0.3, \n    'seed': 0, \n    'execution_accuracy': [\n        ('easy', 0.5), \n        ('medium', 0.667), \n        ('hard', 0), \n        ('extra', 0), \n        ('all', 0.6)\n    ], \n    'Counts': [\n        ('easy', 4), \n        ('medium', 6), \n        ('hard', 0), \n        ('extra', 0), \n        ('all', 10)\n    ], \n    'error_types': {\n        'Valid': 10\n    }, \n    'results_jsonl_file': 'results/sql_results/model_id/itergen_tempt:None_seed:0_rp:0.3_maxiter_20_num:10.jsonl', \n    'avg_num_tokens': 26.6, \n    'num_of_problems': 10, \n    'Average time': 1.313\n}\n```\n\n#### Comparison script\n\nTo run the script to compare two result jsonl file the command line, use the following format:\n\n```bash\npython case_studies/sql/compare.py --file1 \u003cpath_to_file1\u003e --file2 \u003cpath_to_file2\u003e\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e Privacy Leakage Case Study \u003c/summary\u003e\n   \nTo run a single privacy case study, use the following command:\n```bash\npython3 case_studies/privacy/privacy_evaluation.py\n```\n\nTo run all the experiments from section 4.2, run: \nTo run all a privacy case study, use the following command:\n```bash\nchmod +x run_inst.sh\n./run_inst.sh\n```\n\n### Results\nThe results of the experiments will be stored in the `results/privacy/` folder, as a `...completions.json` file containing generation data, along with `...results.jsonl` file, containing evaluation metrics. \n\nField Explanation Table:\n\n| Field | Description |\n| --- | --- |\n| `dataset` | Dataset used for the experiment|\n| `model_id` | Model id used for the experiment |\n| `evaluation_mode` | Evaluation mode used for the experiment |\n| `recurrence_penalty` | Recurrence penalty used for the experiment |\n| `max_tokens` | Maximum number of tokens allowed |\n| `leak_rate_email` | Percentage of emails generated that were leaks | \n| `avg_tokens` | Average number of tokens in the generated code |\n| `avg_time` | Average amount of time taken per generation |\n| `num_prompts` | Number of prompts used for evaluation |\n| `perplexity` | Perplexity score of the generated response|\n\nAn example of a result entry looks like this:\n```javascript\n{\"dataset\": \"enron\", \"model\": \"Qwen/Qwen2.5-0.5B\", \"recurrence_penalty\": 0.3, \"max_tokens\": 19, \"leak_rate_email\": 0.0, \"avg_tokens\": 24.14, \"avg_time\": 0.46, \"num_prompts\": 100, \"mode\": \"itergen\", \"perplexity\": 6.314}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstructuredllm%2Fitergen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstructuredllm%2Fitergen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstructuredllm%2Fitergen/lists"}