{"id":21120405,"url":"https://github.com/tiger-ai-lab/program-of-thoughts","last_synced_at":"2025-08-22T13:05:09.841Z","repository":{"id":63771653,"uuid":"565941671","full_name":"TIGER-AI-Lab/Program-of-Thoughts","owner":"TIGER-AI-Lab","description":"Data and Code for Program of Thoughts (TMLR 2023)","archived":false,"fork":false,"pushed_at":"2024-05-15T02:23:06.000Z","size":22182,"stargazers_count":275,"open_issues_count":3,"forks_count":23,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-06-07T05:39:48.152Z","etag":null,"topics":["llm","program","prompting"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TIGER-AI-Lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-14T16:40:02.000Z","updated_at":"2025-06-03T16:10:58.000Z","dependencies_parsed_at":"2024-05-15T18:22:25.817Z","dependency_job_id":null,"html_url":"https://github.com/TIGER-AI-Lab/Program-of-Thoughts","commit_stats":null,"previous_names":["tiger-ai-lab/program-of-thoughts","wenhuchen/program-of-thoughts"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TIGER-AI-Lab/Program-of-Thoughts","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FProgram-of-Thoughts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FProgram-of-Thoughts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FProgram-of-Thoughts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FProgram-of-Thoughts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TIGER-AI-Lab","download_url":"https://codeload.github.com/TIGER-AI-Lab/Program-of-Thoughts/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FProgram-of-Thoughts/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271643470,"owners_count":24795440,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-22T02:00:08.480Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llm","program","prompting"],"created_at":"2024-11-20T03:09:35.201Z","updated_at":"2025-08-22T13:05:09.721Z","avatar_url":"https://github.com/TIGER-AI-Lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Program of Thoughts\nThis is code repository for the TMLR2023 paper [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://arxiv.org/abs/2211.12588). \n\nWe propose to disentangle computation and reasoning from in the problem solving process. The large language model only needs to express the thoughts using Python program, the computation and solving process is accomplished via an external Python Interpreter. \n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/intro.001.jpeg\" width=\"1000\"\u003e\n\u003c/p\u003e\n\n1. We outperform few-shot CoT by an average of 12% on all the datasets evaluated.\n2. We outperform zero-shot CoT also by an average of 12% on all the datasets evaluated.\n3. We achieve SoTA performance with self-consistency decoding on all the evaluated math word problem datasets (GSM8K, AQuA, SVAMP, TabMWP, MultiArith).\n\nComparison with Few-shot CoT:\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/FS_comparison.png\" width=\"600\"\u003e\n\u003c/p\u003e\n\nComparison with Few-shot CoT with self-consistency:\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/SC_comparison.png\" width=\"600\"\u003e\n\u003c/p\u003e\n\nComparison with Zero-shot CoT:\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/ZS_comparison.png\" width=\"600\"\u003e\n\u003c/p\u003e\n\n## News\n\n1. Added CoT evaluation for AQuA QA for GPT4, the accuracy is 72.7%\n2. Adding [Benchmark](https://github.com/wenhuchen/Program-of-Thoughts/blob/main/benchmark.md)\n\n## Running the code\n\nFirst you need to specify your OPENAI key\n```\nexport OPENAI_KEY = [YOUR_KEY]\n```\n\n- Few-shot + Greedy\n```\npython run_gsm8k.py --greedy\npython run_aqua.py --greedy\n...\n```\n- Few-shot + Self-Consistency\n```\npython run_gsm8k.py\npython run_aqua.py\n...\n```\n-  Zero-shot\n```\npython run_gsm8k_zs.py\npython run_aqua_zs.py\n...\n```\n\nThe prediction file will be dumped in the outputs/ folder, let's say gsm8K_s0_e-1_11_17_10_20.jsonl, or gsm8K_sc_s0_e-1_11_08_21_14.jsonl, or  gsm8K_zs_s0_e-1_11_19_09_55.jsonl.\n\n- Evaluation\n```\ncd outputs\npython compute_score.py --inputs gsm8K_s0_e-1_11_17_10_20.jsonl\npython compute_score.py --inputs aqua_s0_e-1_11_06_18_38.jsonl\npython compute_score.py --inputs svamp_s0_e-1_11_06_21_11.jsonl\n....\n```\n\n## Few-shot Results\n\n1. GSM8K\n- Number of Test Examples: 1318\n- Output: outputs/gsm8K_s0_e-1_11_17_10_20.jsonl\n- EM Score: 0.716\n\n- Output: outputs/gsm8K_sc_s0_e-1_11_08_21_14.jsonl\n- EM Score: 0.799\n\n2. AQuA\n- Number of Test Examples: 253\n- Output: outputs/aqua_s0_e-1_11_06_18_38.jsonl\n- EM Score: 0.541\n\n- Output: aqua_sc_s0_e-1_11_07_20_49.jsonl\n- EM Score: 0.582\n\n3. SVAMP\n- Number of Test Examples: 1000\n- Output: outputs/svamp_s0_e-1_11_24_14_38.jsonl\n- EM Score: 0.852\n\n- Output: outputs/svamp_sc_s0_e-1_11_24_15_54.jsonl\n- EM Score: 0.891\n\n4. TabWMP\n- Number of Test Examples: 7861\n- Output: outputs/tabmwp_s0_e-1_11_06_22_55.jsonl\n- EM Score: 0.732\n\n- Output: outputs/tabmwp_sc_s0_e-1_11_08_18_21.jsonl\n- EM Score: 0.818\n\n5. FinQA\n- Number of Test Examples: 1147 \n- Ouptut: outputs/finqa_s0_e-1_11_16_13_29.jsonl\n- EM Score: 0.647\n\n- Output: outputs/finqa_sc_s0_e-1_11_09_13_00.jsonl\n- EM SCore: 0.682\n\n\n6. ConvFinQA\n- Number of Test Examples: 421 \n- Ouptut: outputs/convfinqa_s0_e-1_11_12_01_38.jsonl\n- EM Score: 0.665\n\n- Output: outputs/convfinqa_sc_s0_e-1_11_12_02_27.jsonl\n- EM SCore: 0.714\n\n7. TATQA\n- Number of Test Examples: 1668 \n- Output: outputs/tatqa_8shot_11_06_19_53.json\n- EM Score: 0.689\n\n- Output: outputs/tatqa_8shot_11_06_19_53.json\n- EM Score: 0.702\n\n\n## Zero-shot Results\n\n\n1. GSM8K\n- Number of Test Examples: 1318\n- Output: outputs/gsm8K_zs_s0_e-1_11_19_09_55.jsonl\n- EM Score: 0.569\n\n2. AQuA\n- Number of Test Examples: 253\n- Output: outputs/aqua_zs_s0_e-1_11_19_11_56.jsonl\n- EM Score: 0.438\n```\npython compute_score.py --inputs aqua_zs_s0_e-1_11_19_11_56.jsonl --relaxed\n```\n3. SVAMP\n- Number of Test Examples: 1000\n- Output: outputs/svamp_zs_s0_e-1_11_18_20_12.jsonl\n- EM Score: 0.708\n\n4. MultiArith\n- Number of Test Examples: 600\n- Output: outputs/multiarith_zs_s0_e-1_11_19_20_12.jsonl\n- EM Score: 0.922\n\n5. TabMWP\n- Output: outputs/tabmwp_zs_s0_e-1_11_19_20_01.jsonl\n- EM Score: 0.646\n\n\n## Cite our Work\n```\n@article{chen2022program,\n  title = {Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks},\n  author = {Wenhu Chen and Xueguang Ma and Xinyi Wang and William W. Cohen},\n  journal={Transactions on Machine Learning Research},\n  year = {2023},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftiger-ai-lab%2Fprogram-of-thoughts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftiger-ai-lab%2Fprogram-of-thoughts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftiger-ai-lab%2Fprogram-of-thoughts/lists"}