{"id":23500287,"url":"https://github.com/zorazrw/odex","last_synced_at":"2025-04-15T18:41:19.947Z","repository":{"id":115680060,"uuid":"580471241","full_name":"zorazrw/odex","owner":"zorazrw","description":"[EMNLP'23] Execution-Based Evaluation for Open Domain Code Generation","archived":false,"fork":false,"pushed_at":"2023-12-22T20:14:33.000Z","size":605,"stargazers_count":47,"open_issues_count":0,"forks_count":6,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-28T23:43:50.032Z","etag":null,"topics":["code-generation","evaluation","execution","open-domain"],"latest_commit_sha":null,"homepage":"https://code-eval.github.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc-by-sa-4.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zorazrw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-12-20T16:36:41.000Z","updated_at":"2025-03-15T19:41:08.000Z","dependencies_parsed_at":null,"dependency_job_id":"cdce6f7f-98d5-4563-ba00-b7f84df7e847","html_url":"https://github.com/zorazrw/odex","commit_stats":{"total_commits":10,"total_committers":2,"mean_commits":5.0,"dds":"0.19999999999999996","last_synced_commit":"8335b426b01e8f3454cd7e218a1ca509dc53ba5e"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zorazrw%2Fodex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zorazrw%2Fodex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zorazrw%2Fodex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zorazrw%2Fodex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zorazrw","download_url":"https://codeload.github.com/zorazrw/odex/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249130894,"owners_count":21217630,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code-generation","evaluation","execution","open-domain"],"created_at":"2024-12-25T06:44:09.661Z","updated_at":"2025-04-15T18:41:19.922Z","avatar_url":"https://github.com/zorazrw.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Execution-based Evaluation for Open Domain Code Generation\n\n[![CC BY-SA 4.0][cc-by-sa-shield]][cc-by-sa]\n\nThis repository contains the data and code for the work [Execution-based Evaluation for Open Domain Code Generation](https://arxiv.org/pdf/2212.10481.pdf).\n\nThis work is licensed under a\n[Creative Commons Attribution-ShareAlike 4.0 International License][cc-by-sa].\n\n[cc-by-sa]: http://creativecommons.org/licenses/by-sa/4.0/\n[cc-by-sa-image]: https://licensebuttons.net/l/by-sa/4.0/88x31.png\n[cc-by-sa-shield]: https://img.shields.io/badge/License-CC%20BY--SA%204.0-lightgrey.svg\n\nIf you find our paper or code useful, please cite the paper\n\n```\n@article{wang2022execution,\n  title={Execution-Based Evaluation for Open-Domain Code Generation},\n  author={Zhiruo Wang, Shuyan Zhou, Daniel Fried, Graham Neubig},\n  journal={arXiv preprint arXiv:2212.10481},\n  year={2022}\n}\n```\n\n## Install\n\n```bash\npip install -r requirements.txt\n```\n\n## Dataset\n\nWe split the dataset by which natural language is of the corresponding intent.\n\n```markdown\n.\n├── README.md\n├── data\n│   ├── en_test.jsonl\n│   ├── es_test.jsonl\n│   ├── ja_test.jsonl\n│   └── ru_test.jsonl\n```\n\nEach line contains a serialized json object, an example looks like:\n\n```\n{\n    'task_id': 3844801,\n    'intent': \"check if all elements in list `myList` are identical\", \n    'prompt': \"def f_3844801(myList):\\n\\treturn \",\n    'canonical_solution': \"all(x == myList[0] for x in myList)\",\n    'suffix': \"\",\n    'test_start': \"\\ndef check(candidate):\",\n    'test': [\n        \"\\n    assert candidate([1,2,3]) == False\\n\", \n        \"\\n    assert candidate([1,1,1,1,1,1]) == True\\n\",\n        \"\\n    assert candidate([1]) == True\\n\",\n        \"\\n    assert candidate(['k','k','k','k','k']) == True\\n\",\n        \"\\n    assert candidate([None,'%$#ga',3]) == False\\n\"\n    ],\n    'entry_point': \"f_3844801\",\n}\n```\n\nwhere:\n\n1. `task_id` is the post id of the original StackOverflow post where the sample is constructed from;\n2. `intent` is the natural language description rewritten by human annotators with qualified specificity;\n3. `prompt` is the function prefix (definition, input arguments, etc.) to properly execute the code snippet;\n4. `canonical_solution` is the reference solution (verified by human annotators) of the coding problem;\n5. `suffix` is the function suffix (return values, if any) to proper to execute the code;\n6. `test_start` is the definition of test functions, also, including library imports if necessitated by the program;\n7. `test` is the list of test cases created by human annotators;\n8. `entry_point` is the function name that should be called for 'check' during evaluation.\n\nTo correctly execute the (canonical) code snippets, one needs to install all involved libraries, as listed in the `./library/` directory.\n\n\n## Evaluating Code Generation Models\n\nWe provide code to evaluate on two state-of-the-art code generation models: CodeX and CodeGen. To perform the NL-to-Code generation task and collect model predictions:\n\nFor __CodeX__, run\n\n```bash\npython nl2code_codex.py --language en \\\n--model_name \"code-davinci-002\" \\\n--openai_api_key ${YOUR_API_KEY} \\\n```\n\nchange the `model_name` argument to \"code-cushman-001\" or \"code-davinci-001\" to try other model variants.\n\nFor __CodeGen__, run\n\n```bash\npython nl2code_codegen.py --language en \\\n--model_size 350M --model_data mono \n```\n\nOther valid options for `model_size` include: \"2B\", \"6B\", and \"16B\", which correspond to the 2.7B, 6.1B, and 16.1B CodeGen models.\n\nFor `model_data`, other options include \"multi\" and \"nl\".\n\n### Evaluation\n\nOur default evaluation metric is the execution pass rate.\nBefore the evaluation, make sure your environment has all required libraries installed, and better imported as in the code samples. To do this, you can:\n\n```bash\npip install -r ./library/requirements.txt \npython ./library/imports.py\n```\n\nThen we can perform the execution by running:\n\n```bash\npython eval_passk.py --language en --prediction_path ${MODEL_PRED_PATH}\n```\n\nWe also support five other non-execution metrics: BLEU, ROUGE, METEOR, ChrF, and CodeBLEU.\nFor example, to evaluate with the BLEU metric, run:\n\n```bash\npython eval_nonex.py --language en --prediction_path ${MODEL_PRED_PATH} --eval_metric bleu\n```\n\nSpecifying the `eval_metric` argument with \"rouge\"/\"meteor\"/\"chrf\"/\"codebleu\" to use other metrics.\n\n### Detailed Analysis\n\n#### Open-Domain versus Closed-Domain\n\nTo evaluate on the subset of open-domain or closed-domain samples, you only need to add another argument at evaluation time (when running `eval_passk.py` or `eval_nonex.py`), by\n\n```bash\n--library_usage \"open\"   # or \"closed\"\n```\n\n#### Few-shot Prompting\n\nTo include more prompt-solution pairs for in-context prompting learning, specify the `num_examples` at inference time (when running `nl2code_codex.py` and `nl2code_codegen.py`), by\n\n```bash\n--num_examples 1    # 2, 3, ... \n```\n\n#### Number of Input Test Cases\n\nTo add exemplar test cases in the prompt inputs, specify the `num_tests` at inference time, by\n\n```bash\n--num_tests 1   # 2, 3, ...\n```\n\n#### Number of Evaluation Test Cases\n\nTo use different numbers of test cases for execution-based evaluation, specify the `num_tests_eval` when running `eval_passk.py`, for example\n\n```bash\npython eval_passk.py --language en --prediction_path ${MODEL_PRED_PATH} --num_tests_eval 1 \n```\n\n#### Semantics of Function Names\n\nOur paper explores three methods to create function names in the wrapping context:\n\n* \"id\": `f_${ID}`, simple string formatting using the StackOverflow post ID\n* \"constant\": `function`, using the same string constant for all samples\n* \"intent\": heuristic-based extraction from the paired NL intent\n\nTo experiment with different function names, specify the `function_name` at inference time, by\n\n```bash\n--function_name \"intent\"   # \"id\" \"constant\"\n```\n\n#### Metric Correlation\n\nWe also provide code to compare execution-based and non-execution evaluation metrics on a sample-wise basis. Take the execution and BLEU score as an example, one can run:\n\n```bash\npython metric_corr.py --language en \\\n--prediction_file ${MODEL_PRED_PATH} \\\n--eval_metric \"bleu\"\n```\n\nTo get visualizations in violin plots and histograms, add `--do_plot_violin` or `do_plot_stacked_hist`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzorazrw%2Fodex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzorazrw%2Fodex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzorazrw%2Fodex/lists"}