{"id":13994001,"url":"https://github.com/amazon-science/mxeval","last_synced_at":"2025-04-12T13:10:37.836Z","repository":{"id":62180199,"uuid":"549744904","full_name":"amazon-science/mxeval","owner":"amazon-science","description":null,"archived":false,"fork":false,"pushed_at":"2024-07-17T21:36:22.000Z","size":9342,"stargazers_count":102,"open_issues_count":4,"forks_count":22,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-01-13T14:17:34.300Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amazon-science.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-11T17:05:42.000Z","updated_at":"2024-12-06T01:50:44.000Z","dependencies_parsed_at":"2024-06-19T03:01:24.462Z","dependency_job_id":"127d2406-dae2-4a84-9f89-ba1bb0fe0394","html_url":"https://github.com/amazon-science/mxeval","commit_stats":null,"previous_names":["amazon-science/mxeval"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fmxeval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fmxeval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fmxeval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fmxeval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amazon-science","download_url":"https://codeload.github.com/amazon-science/mxeval/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234808946,"owners_count":18890088,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-09T14:02:39.713Z","updated_at":"2025-01-20T15:14:42.298Z","avatar_url":"https://github.com/amazon-science.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Execution-based evaluation of code in 10+ languages\n\nThis repository contains code to perform execution-based multi-lingual evaluation of code generation capabilities and the corresponding data, namely, a multi-lingual benchmark MBXP, multi-lingual MathQA and multi-lingual HumanEval. Results and findings can be found in the paper \"Multi-lingual Evaluation of Code Generation Models\" (https://arxiv.org/abs/2210.14868).\n\n\n## Paper summary\n\nOur paper describes the language conversion framework, the synthetic solution generation, and many other types of evaluation beyond the traditional function completion evaluation such as translation, code insertion, summarization, and robustness evaluation. \n\n![Paper summary](./graphics/paper_summary.png)\n\n\n## Language conversion of execution-based function completion datasets\n\nBelow we demonstrate the language conversion (component A above) for the conversion from Python to Java (abridged example for brevity). \n\n![Example conversion to Java](./graphics/mbxp_java_conversion.png)\n\n\n# Installation\n\nCheck out and install this repository:\n```\ngit clone https://github.com/amazon-science/mxeval.git\npip install -e mxeval\n```\n\n\n\n## Dependencies\nWe provide scripts to help set up programming language dependencies that are used to execute and evaluate using datasets in MBXP.\n\n### Amazon Linux AMI\n```\nbash language_setup/amazon_linux_ami.sh\n```\n### Ubuntu\n```\nbash language_setup/ubuntu.sh\n```\n\n# Usage\n\n**This program exists to run untrusted model-generated code. Users are strongly\nencouraged not to do so outside of a robust security sandbox. See the comment in\n`execution.py` for more information and instructions.**\n\nEach sample is formatted into a single line:\n```\n{\"task_id\": \"Corresponding task ID\", \"completion\": \"Completion only without the prompt\",\n\"language\": \"programming language name\"}\n```\nWe provide `data/mbxp/examples/mbxp_samples.jsonl` to illustrate the format. \n\nHere is nearly functional example code (you just have to provide\n`generate_one_completion` to make it work) that saves generated completions to\n`samples.jsonl`.\n```\nfrom mxeval.data import write_jsonl, read_problems\n\nproblems = read_problems()\n\nnum_samples_per_task = 200\nsamples = [\n    dict(task_id=task_id, language=problems[task_id][\"language\"], completion=generate_one_completion(problems[task_id][\"prompt\"]))\n    for task_id in problems\n    for _ in range(num_samples_per_task)\n]\nwrite_jsonl(\"samples.jsonl\", samples)\n```\n\nTo evaluate the samples for, e.g., Java MBJP evaluation, run \n```\nevaluate_functional_correctness data/mbxp/examples/mbjp_samples.jsonl --problem_file data/mbxp/mbjp_release_v1.jsonl\n```\nor to run all languages\n```\nfor lang in mbcpp mbcsp mbgp mbjp mbjsp mbkp mbphp mbplp mbpp mbrbp mbscp mbswp mbtsp;\n  do evaluate_functional_correctness --problem_file data/mbxp/${lang}_release_v1.jsonl data/mbxp/examples/${lang}_samples.jsonl;\ndone\n```\nYou can check the programming-language dependency installation by running the above example for each MBXP dataset. You should obtain the following results for the mbxp_samples.jsonl files provided:\n\n\n| Dataset | pass@1 |\n|---------|--------|\n| MBCPP   | 79.60% |\n| MBCSP   | 63.63% |\n| MBGP    | 39.19% |\n| MBJP    | 85.30% |\n| MBJSP   | 78.67% |\n| MBKP    | 63.77% |\n| MBPHP   | 72.77% |\n| MBPLP   | 38.41% |\n| MBPP    | 82.24% |\n| MBRBP   | 58.90% |\n| MBSCP   | 42.96% |\n| MBSWP   | 29.40% |\n| MBTSP   | 87.29% |\n\n\n\nNote: Because there is no unbiased way of estimating pass@k when there are fewer\nsamples than k, the script does not evaluate pass@k for these cases. To\nevaluate with other k values, pass `--k \u003ccomma-separated-values-here\u003e`. For\nother options, see\n```\n$ evaluate_functional_correctness --help\n```\nHowever, we recommend that you use the default values for the rest.\n\n### Example usage with non-default values\n```\nevaluate_functional_correctness  data/mbxp/samples/mbjp_samples.jsonl --problem_file data/mbxp/mbjp_release_v1.jsonl --n_workers 63 --k 1,5,10,100\n```\n\n### Known Issues\n\nWhile evaluation uses very little memory, you might see the following error\nmessage when the system is running out of RAM. Since this may cause some\ncorrect programs to fail, we recommend that you free some memory and try again.\n```\nmalloc: can't allocate region\n```\n\nSome system might require longer compilation timeout. If you see that the execution fails due to compilation timeout reason, this number should be increased accordingly.\n\n### Canonical solutions release\n\nWe have released canonical solutions for certain popular langauges (v1.2). The detailed numbers of the solutions for each langauge are listed below. \n\n| Dataset | # solutions | # problems |\n|---------|-------------|------------|\n| MBCPP   |     773     |     848    |\n| MBCSP   |     725     |     968    |\n| MBJP    |     874     |     966    |\n| MBJSP   |     938     |     966    |\n| MBKP    |     796     |     966    |\n| MBPHP   |     950     |     966    |\n| MBPP    |     960     |     974    |\n| MBRBP   |     784     |     966    |\n| MBTSP   |     967     |     968    |\n\n\n### Future release\nWe plan to release synthetic canonical solutions as well as processed datasets for other evaluation tasks such as code-insertion, code-translation, etc. \n\n\n\n\n### Credits\nWe adapted OpenAI's human-eval package (https://github.com/openai/human-eval) for the multi-lingual case. We thank OpenAI for their pioneering effort in this field including the release of the original HumanEval dataset, which we convert to the multi-lingual versions. We also thank Google for their release of the original MBPP Python dataset (https://github.com/google-research/google-research/tree/master/mbpp), which we adapt and convert to other programming languages.\n\n### Citation\n\nPlease cite using the following bibtex entry:\n\n```\n@article{mbxp_athiwaratkun2022,\n  title = {Multi-lingual Evaluation of Code Generation Models},\n  author = {Athiwaratkun, Ben and\n   Gouda, Sanjay Krishna and\n   Wang, Zijian and\n   Li, Xiaopeng and\n   Tian, Yuchen and\n   Tan, Ming\n   and Ahmad, Wasi Uddin and\n   Wang, Shiqi and\n   Sun, Qing and\n   Shang, Mingyue and\n   Gonugondla, Sujan Kumar and\n   Ding, Hantian and\n   Kumar, Varun and\n   Fulton, Nathan and\n   Farahani, Arash and\n   Jain, Siddhartha and\n   Giaquinto, Robert and\n   Qian, Haifeng and\n   Ramanathan, Murali Krishna and\n   Nallapati, Ramesh and\n   Ray, Baishakhi and\n   Bhatia, Parminder and\n   Sengupta, Sudipta and\n   Roth, Dan and\n   Xiang, Bing},\n  doi = {10.48550/ARXIV.2210.14868},\n  url = {https://arxiv.org/abs/2210.14868},\n  keywords = {Machine Learning (cs.LG), Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},\n  publisher = {arXiv},\n  year = {2022},\n  copyright = {Creative Commons Attribution 4.0 International}\n}\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famazon-science%2Fmxeval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famazon-science%2Fmxeval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famazon-science%2Fmxeval/lists"}