{"id":13429762,"url":"https://github.com/reasoning-machines/CoCoGen","last_synced_at":"2025-03-16T04:31:11.580Z","repository":{"id":69335030,"uuid":"507240498","full_name":"reasoning-machines/CoCoGen","owner":"reasoning-machines","description":"Language Models of Code are Few-Shot Commonsense Learners (EMNLP 2022)","archived":false,"fork":false,"pushed_at":"2023-03-20T01:55:19.000Z","size":5961,"stargazers_count":86,"open_issues_count":2,"forks_count":12,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-10T02:09:11.539Z","etag":null,"topics":["code-generation","commonsense-reasoning","few-shot-learning","large-language-models","reasoning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/reasoning-machines.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-06-25T07:08:18.000Z","updated_at":"2024-09-21T12:20:36.000Z","dependencies_parsed_at":null,"dependency_job_id":"837300a8-38a0-4220-bfd6-762fde9f8a0f","html_url":"https://github.com/reasoning-machines/CoCoGen","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reasoning-machines%2FCoCoGen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reasoning-machines%2FCoCoGen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reasoning-machines%2FCoCoGen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reasoning-machines%2FCoCoGen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/reasoning-machines","download_url":"https://codeload.github.com/reasoning-machines/CoCoGen/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243826788,"owners_count":20354220,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code-generation","commonsense-reasoning","few-shot-learning","large-language-models","reasoning"],"created_at":"2024-07-31T02:00:44.915Z","updated_at":"2025-03-16T04:31:07.759Z","avatar_url":"https://github.com/reasoning-machines.png","language":"Python","funding_links":[],"categories":["3 Reasoning Tasks","Python"],"sub_categories":["3.1 Commonsense Reasoning"],"readme":"# Language Models of Code are Few-Shot Commonsense Learners\n\nThe official repository for [\"Language Models of Code are Few-Shot Commonsense Learners\"](https://arxiv.org/abs/2210.07128) (Madaan et al., EMNLP'2022).\n\nThis paper addresses the general task of *structured* commonsense reasoning: \ngenerate a *graph* given a natural language input.\nWe address these family of tasks by framing the problem as a **code generation** task, and prompting large language models of **code** (e.g., Codex).\n\n| \u003cimg alt=\"image\" width=\"70%\" src=\"https://user-images.githubusercontent.com/15002544/194352035-f0ae52f9-8fe3-47f7-af2e-2c1dc2461ad2.png\"\u003e |\n|-|\n\n| \u003cimg width=\"70%\" alt=\"image\" src=\"https://user-images.githubusercontent.com/15002544/194354502-460dcda7-b497-44b2-bdca-75240cb0664c.png\"\u003e |\n|-|\n\n## Table of contents\n\n- [Running CoCoGen](#running-cocogen)\n- [Sample Outputs](#sample-outputs)\n- [End-to-end workflow](#end-to-end-workflow)\n- [Creating dynamic prompts (KST)](#creating-dynamic-prompts-kst)\n\n\n## Running CoCoGen\n\nAn OpenAI API key is required to run the jobs. To get an API key, register at [https://openai.com/blog/openai-codex/](https://openai.com/blog/openai-codex/).\n\nThe key should be exported in the environment variable `OPENAI_API_KEY`.\n\nPlease note that as of Oct 2022, codex is free to use for non-commercial purposes with a key.\n\n\n\n1. Proscript script generation:\n\n```\npython -u src/api/query_openai_over_tasks.py --task_file_path data/proscript_script_gen/dev.jsonl --num_tasks -1 \\\n                                                 --output_file_path data/proscript_script_gen/dev_outputs.jsonl \\\n                                                 --prompt_path  data/proscript_script_gen/prompt.txt --job_type proscript-literal \\\n                                                  --engine text-davinci-002 --max_requests_per_min 10\n```\n\n\nHere:\n\n    * `--task_file_path`: path to the file containing the tasks to be run\n    * `--num_tasks`: number of tasks to be run. If -1, all tasks in the file will be run.\n    * `--output_file_path`: path to the file to which the outputs will be written\n    * `--prompt_path`: path to the file containing the prompt. The prompt is created from the train split.\n    * `--job_type`: the type of job to run. This is used to determine the code converter. Currently, the following converters are supported: \"explagraphs\",         \"proscript-hashmap-init\", \"proscript-hashmap-sep\", \"proscript-networkx\", \"proscript-literal\", \"proscript-oo\"\n\n    * `--engine`: the API engine to use. See the complete list of available at https://beta.openai.com/docs/api-reference/introduction.\n\n\n\n2. Proscript edge prediction:\n\n```\npython -u src/api/query_openai_over_tasks.py --task_file_path data/proscript_edge_prediction/dev.jsonl --num_tasks -1 \\\n                                                 --output_file_path data/proscript_edge_prediction/dev_outputs.jsonl \\\n                                                 --prompt_path  data/proscript_edge_prediction/prompt.txt --job_type \"proscript-edge-pred-literal\" \\\n                                                  --engine text-davinci-002 --max_requests_per_min 10\n```\n\n\n* Evaluation:\n```sh\npython src/eval/edge_pred_eval.py  data/proscript_edge_prediction/dev_outputs.jsonl\n```\n\n3. Explagraphs:\n\n```\npython -u src/api/query_openai_over_tasks.py --task_file_path data/explagraphs/dev.jsonl --num_tasks -1 \\\n                                                 --output_file_path data/explagraphs//dev_outputs.jsonl \\\n                                                 --prompt_path  data/explagraphs//prompt.txt --job_type \"explagraphs-literal\" \\\n                                                  --engine text-davinci-002 --max_requests_per_min 10\n```\n\n\n* Evaluate:\n```sh\nexport CUDA_VISIBLE_DEVICES=0,1 \u0026\u0026 python src/eval/explagraph_eval.py data/explagraphs//dev_outputs.jsonl tmp\n```\n\n4. Propara:\n\n```\npython -u src/api/query_openai_over_tasks.py --task_file_path data/propara/test.jsonl --num_tasks -1 \\\n                                                 --output_file_path data/propara//test_outputs.jsonl \\\n                                                 --prompt_path  data/propara//prompt.txt --job_type \"propara-func\" \\\n                                                  --engine text-davinci-002 --max_requests_per_min 10\n```\n\n* Evaluate:\n```sh\npython src/eval/propara/eval_results.py --raw_output_file data/propara//test_outputs.jsonl --output_file  data/propara/test_output_predictions.jsonl\n```\n\n\n---\n\n## Sample Outputs\n\nSample outputs for each task are located in `outputs`.\n\n```\noutputs/\n├── explagraphs\n├── propara\n├── proscript_edge_pred\n└── proscript_script_generation\n```\n\n Since we are not permitted by the authors of proscript to release the test split, we remove the reference outputs for proscript_script_generation and proscript_edge_prediction.\n\n\n\n### Output format\nEach output file is a jsonl, where each line is a json object with several common fields:\n\n```js\n{\n\n    \"prompt\": \"the dynamic prompt created for the task\"\",\n    \"reference_graph\":  \"the graph/table for the task (G in paper),\n    \"reference_code\": \"python code for the graph (T + G_c in paper))\",\n    \"codex_response\": { \n        response from codex\n    },\n    \"generated_code\": \"code generated by codex\",\n    \"generated_graph\": {\n        same format as reference_graph, obtained by parsing the generated code\n    }\n}\n```\n\nFor example, for edge prediction:\n\n```js\n{\n    \"scenario\": \"have bowl for cut ingredients\",\n    \"closest_queries\": [\n        \"mix ingredients together\",\n        \"add the measured ingredients to bowl\",\n        \"take out a cutting board and knife\",\n        \"gather the ingredients\",\n        \"measure out ingredients\",\n        \"put all ingredients into bowl and mix\",\n        \"prepare the ingredients with a knife\",\n        \"make a list of ingredients needed\",\n        \"copy list from website to paper\",\n        \"put dry ingredients in one bowl\",\n        \"prepare a meal\"\n    ],\n    \"prompt\": \"the dynamic prompt created for the task\"\",\n        \"reference_graph\": {\n        \"title\": \"have bowl for cut ingredients\",\n        \"num_steps\": 7,\n        \"schema\": [\n            \"step0: Move toward where work is to be done\",\n            \"step1: Find the location of desired bowl\",\n            \"step2: Grab the bowl firmly\",\n            \"step3: Set the bowl down\",\n            \"step4: Walk toward the bowls location\",\n            \"step5: get out ingredients\",\n            \"step6: have bowl for cut ingredients\"\n        ],\n        \"relations\": [\n            \"step1 -\u003e step4\",\n            \"step4 -\u003e step2\",\n            \"step2 -\u003e step0\",\n            \"step0 -\u003e step3\",\n            \"step3 -\u003e step6\",\n            \"step5 -\u003e step1\"\n        ]\n    },\n    \"reference_code\": \"class HaveBowlForCutIngredients:\\n\\n    title = \\\"have bowl for cut ingredients\\\"\\n    steps = 7\\n\\n    def step0(self):\\n        return \\\"Move toward where work is to be done\\\"\\n\\n    def step1(self):\\n        return \\\"Find the location of desired bowl\\\"\\n\\n    def step2(self):\\n        return \\\"Grab the bowl firmly\\\"\\n\\n    def step3(self):\\n        return \\\"Set the bowl down\\\"\\n\\n    def step4(self):\\n        return \\\"Walk toward the bowls location\\\"\\n\\n    def step5(self):\\n        return \\\"get out ingredients\\\"\\n\\n    def step6(self):\\n        return \\\"have bowl for cut ingredients\\\"\\n\\n    def get_relations(self):\\n        return [\\n            \\\"step1 -\u003e step4\\\",\\n            \\\"step4 -\u003e step2\\\",\\n            \\\"step2 -\u003e step0\\\",\\n            \\\"step0 -\u003e step3\\\",\\n            \\\"step3 -\u003e step6\\\",\\n            \\\"step5 -\u003e step1\\\",\\n        ]\\n\",\n    \"input_prompt_code\": \"class HaveBowlForCutIngredients:\\n\\n    title = \\\"have bowl for cut ingredients\\\"\\n    steps = 7\\n\\n    def step0(self):\\n        return \\\"Move toward where work is to be done\\\"\\n\\n    def step1(self):\\n        return \\\"Find the location of desired bowl\\\"\\n\\n    def step2(self):\\n        return \\\"Grab the bowl firmly\\\"\\n\\n    def step3(self):\\n        return \\\"Set the bowl down\\\"\\n\\n    def step4(self):\\n        return \\\"Walk toward the bowls location\\\"\\n\\n    def step5(self):\\n        return \\\"get out ingredients\\\"\\n\\n    def step6(self):\\n        return \\\"have bowl for cut ingredients\\\"\\n\\n\",\n    \"codex_response\": {\n        \"id\": \"\",\n        \"object\": \"text_completion\",\n        \"created\": ,\n        \"model\": \"code-davinci:002\",\n        \"choices\": [\n            {\n                \"text\": \"    def get_relations(self):\\n        return [\\n            \\\"step4 -\u003e step1\\\",\\n            \\\"step1 -\u003e step2\\\",\\n            \\\"step2 -\u003e step3\\\",\\n            \\\"step3 -\u003e step6\\\",\\n            \\\"step5 -\u003e step0\\\",\\n            \\\"step5 -\u003e step4\\\",\\n        ]\\n\",\n                \"index\": 0,\n                \"logprobs\": null,\n                \"finish_reason\": \"stop\"\n            }\n        ]\n    },\n    \"generated_code\": \"class HaveBowlForCutIngredients:\\n\\n    title = \\\"have bowl for cut ingredients\\\"\\n    steps = 7\\n\\n    def step0(self):\\n        return \\\"Move toward where work is to be done\\\"\\n\\n    def step1(self):\\n        return \\\"Find the location of desired bowl\\\"\\n\\n    def step2(self):\\n        return \\\"Grab the bowl firmly\\\"\\n\\n    def step3(self):\\n        return \\\"Set the bowl down\\\"\\n\\n    def step4(self):\\n        return \\\"Walk toward the bowls location\\\"\\n\\n    def step5(self):\\n        return \\\"get out ingredients\\\"\\n\\n    def step6(self):\\n        return \\\"have bowl for cut ingredients\\\"\\n\\n    def get_relations(self):\\n        return [\\n            \\\"step4 -\u003e step1\\\",\\n            \\\"step1 -\u003e step2\\\",\\n            \\\"step2 -\u003e step3\\\",\\n            \\\"step3 -\u003e step6\\\",\\n            \\\"step5 -\u003e step0\\\",\\n            \\\"step5 -\u003e step4\\\",\\n        ]\\n\",\n    \"generated_graph\": {\n        \"title\": \"have bowl for cut ingredients\",\n        \"num_steps\": 7,\n        \"schema\": [\n            \"step0: Move toward where work is to be done\",\n            \"step1: Find the location of desired bowl\",\n            \"step2: Grab the bowl firmly\",\n            \"step3: Set the bowl down\",\n            \"step4: Walk toward the bowls location\",\n            \"step5: get out ingredients\",\n            \"step6: have bowl for cut ingredients\"\n        ],\n        \"relations\": [\n            \"step4 -\u003e step1\",\n            \"step1 -\u003e step2\",\n            \"step2 -\u003e step3\",\n            \"step3 -\u003e step6\",\n            \"step5 -\u003e step0\",\n            \"step5 -\u003e step4\"\n        ]\n    },\n    \"elapsed_time\": 0.0003092289\n}\n```\n\n\n---\n\n\n## End-to-end workflow\n\n\n- To give an overview of our end-to-end workflow, we provide all the files required to run propara end-to-end.\n\n\n#### Step 1: Create task files\n\n\n```sh\npython src/prompting/make_task_file.py --inpath data/propara/train.json --outpath data/propara/code/func/train.jsonl --job_type \"propara-func\"\n\npython src/prompting/make_task_file.py --inpath data/propara/test.json --outpath data/propara/code/func/test.jsonl --job_type \"propara-func\"\n\npython src/prompting/make_task_file.py --inpath data/propara/dev.json --outpath data/propara/code/func/dev.jsonl --job_type \"propara-func\"\n```\n\n\n#### Step 2: Create the prompt\n\n```sh\npython src/prompting/make_codex_prompt.py data/propara/code/func/train.jsonl 6 \u003e data/propara/code/func/prompt.txt \u0026\u0026 cat data/propara/code/func/prompt.txt|it2copy\n```\n\n#### Step 3: Run inference\n\n\n```sh\npython -u src/codexapi/query_openai_over_tasks.py --task_file_path data/propara/code/func/test.jsonl --num_tasks -1 --output_file_path data/propara/code/func/test_output.jsonl --prompt_path data/propara/code/func/prompt.txt --job_type \"propara-func\" --engine code-davinci-002 --max_requests_per_min 10  --max_tokens 800\n```\n\n* The max tokens is set to 800 (vs. 280 default) because the generations are long. The prompt fits about 5 examples.\n\n#### Step 4: Evaluate\n\n```sh\npython src/eval/propara/eval_results.py --raw_output_file data/propara/code/func/test_output.jsonl --output_file  data/propara/predictions/test_output.json \n```\n\nEvaluate `data/propara/code/func/test_output.jsonl`\n\n\n---\n\n## Creating dynamic prompts\n\n- Note that when we run a job with a fixed random prompt, the `prompt_path` is supplied as an argument to `query_openai_over_tasks.py`. In case of dynamic prompts, we will create a prompt for *each test example*, and store it in the task jsonl file with each example in the `prompt` field.\n\n- In summary, we will call `query_openai_over_tasks.py` as before, but using a different task file and without specifying a prompt_path.\n\n- To create this file, we use the following command:\n\n```sh\npython src/prompting/knnprompt/make_knn_like_prompt.py\\\n       --train_file_path ${TRAIN_EXAMPLES_PATH}\\\n       --test_file_path ${TEST_FILE_PATH} \\\n       --k $K \n       --output_file_path ${KNN_TEST_FILE_PATH} \\\n       --query_field ${QFIELD}\n```\n\nHere:\n* `TRAIN_EXAMPLES_PATH` is the path to the training examples used to create the prompt. Note that this file should be the same file that is used for creating the prompts (it's the file we get after running `make_task_file`).\n\n* `TEST_FILE_PATH` is the path to the test file. A prompt will be created for each example in this file, and written to the output file ${KNN_TEST_FILE_PATH}.\n\n* `K` is the number of examples (nearest neighbors) to use for creating the prompt.\n\n* `QUERY_FIELD` is the field in the test file that will be used to do knn search.\n\n\nBy default, we use \"all-mpnet-base-v2\" as our similarity model, but it can be changed using `--retrieval_model_name`\n\n\n* An example with explagraph is:\n\n```sh\npython src/prompting/knnprompt/make_knn_like_prompt.py --train_file_path data/explagraphs/train.jsonl --test_file_path data/explagraphs/dev.jsonl --k 21 --output_file_path data/explagraphs/dev_kst_task.jsonl  --query_field belief\n```\n\nOnce this command finishes, we can query codex as usual using the following:\n\n```sh\npython -u src/codexapi/query_openai_over_tasks.py --task_file_path data/explagraphs/dev_kst_task.jsonl --num_tasks -1 --output_file_path data/explagraphs/dev_kst_task_OUTPUT.jsonl  --job_type explagraph-relation --engine code-davinci-002\n\n```\n\nNote that no prompt path is specified, as the task file `data/explagraphs/dev_kst_task.jsonl` contains a prompt for each example.\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freasoning-machines%2FCoCoGen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freasoning-machines%2FCoCoGen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freasoning-machines%2FCoCoGen/lists"}