{"id":13427558,"url":"https://github.com/openai/human-eval","last_synced_at":"2025-05-14T03:11:26.841Z","repository":{"id":38428920,"uuid":"383601290","full_name":"openai/human-eval","owner":"openai","description":"Code for the paper \"Evaluating Large Language Models Trained on Code\"","archived":false,"fork":false,"pushed_at":"2025-01-17T18:22:17.000Z","size":48,"stargazers_count":2685,"open_issues_count":38,"forks_count":383,"subscribers_count":129,"default_branch":"master","last_synced_at":"2025-04-13T00:38:34.556Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-06T21:23:45.000Z","updated_at":"2025-04-12T12:45:15.000Z","dependencies_parsed_at":"2025-01-24T23:10:29.481Z","dependency_job_id":null,"html_url":"https://github.com/openai/human-eval","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Fhuman-eval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Fhuman-eval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Fhuman-eval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Fhuman-eval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openai","download_url":"https://codeload.github.com/openai/human-eval/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254059519,"owners_count":22007771,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T01:00:31.580Z","updated_at":"2025-05-14T03:11:21.833Z","avatar_url":"https://github.com/openai.png","language":"Python","funding_links":[],"categories":["Paper List","Evaluation","Datasets \u0026 Benchmarks","Domain-Specific Benchmarks","Python","A01_文本生成_文本对话","Benchmarks \u0026 Evaluation","Learning Resources","Benchmark","Evaluation and Monitoring","Coding and Software Engineering","Benchmarks \u0026 Datasets","🛠️ AI 工具与框架","📝 DeepReview Agent","Benchmarks","11. Benchmarks \u0026 Leaderboards","📈 Benchmarks \u0026 Leaderboards","📊 AI Evaluation \u0026 Benchmarks","Benchmarks and Datasets","Browse The Shelves","Coding Agent Evaluation","Prompts \u0026 Datasets"],"sub_categories":["Transformer-based","Others","Coding and Software Engineering","大语言对话模型及数据","Code Benchmarks","Papers \u0026 Research","Code","Code Generation","Domain-Specific Benchmarks","模型评估","Software Code Review","Domain","Data \u0026 Alignment Tools","YouTube Channels","Usage Tips","Agent evals","Benchmark Datasets","Frameworks \u0026 Tools"],"readme":"# HumanEval: Hand-Written Evaluation Set \n\nThis is an evaluation harness for the HumanEval problem solving dataset\ndescribed in the paper \"[Evaluating Large Language Models Trained on\nCode](https://arxiv.org/abs/2107.03374)\".\n\n## Installation\n\nMake sure to use python 3.7 or later:\n```\n$ conda create -n codex python=3.7\n$ conda activate codex\n```\n\nCheck out and install this repository:\n```\n$ git clone https://github.com/openai/human-eval\n$ pip install -e human-eval\n```\n\n## Usage\n\n**This program exists to run untrusted model-generated code. Users are strongly\nencouraged not to do so outside of a robust security sandbox. The [execution\ncall](https://github.com/openai/human-eval/blob/master/human_eval/execution.py#L48-L58)\nin `execution.py` is deliberately commented out to ensure users read this\ndisclaimer before running code in a potentially unsafe manner. See the comment in\n`execution.py` for more information and instructions.**\n\nAfter following the above instructions to enable execution, generate samples\nand save them in the following JSON Lines (jsonl) format, where each sample is\nformatted into a single line like so:\n```\n{\"task_id\": \"Corresponding HumanEval task ID\", \"completion\": \"Completion only without the prompt\"}\n```\nWe provide `example_problem.jsonl` and `example_solutions.jsonl` under `data`\nto illustrate the format and help with debugging.\n\nHere is nearly functional example code (you just have to provide\n`generate_one_completion` to make it work) that saves generated completions to\n`samples.jsonl`.\n```\nfrom human_eval.data import write_jsonl, read_problems\n\nproblems = read_problems()\n\nnum_samples_per_task = 200\nsamples = [\n    dict(task_id=task_id, completion=generate_one_completion(problems[task_id][\"prompt\"]))\n    for task_id in problems\n    for _ in range(num_samples_per_task)\n]\nwrite_jsonl(\"samples.jsonl\", samples)\n```\n\nTo evaluate the samples, run\n```\n$ evaluate_functional_correctness samples.jsonl\nReading samples...\n32800it [00:01, 23787.50it/s]\nRunning test suites...\n100%|...| 32800/32800 [16:11\u003c00:00, 33.76it/s]\nWriting results to samples.jsonl_results.jsonl...\n100%|...| 32800/32800 [00:00\u003c00:00, 42876.84it/s]\n{'pass@1': ..., 'pass@10': ..., 'pass@100': ...}\n```\nThis script provides more fine-grained information in a new file ending in\n`\u003cinput_path\u003e_results.jsonl`. Each row now contains whether the completion\n`passed` along with the execution `result` which is one of \"passed\", \"timed\nout\", or \"failed\".\n\nAs a quick sanity-check, the example samples should yield 0.5 pass@1.\n```\n$ evaluate_functional_correctness data/example_samples.jsonl --problem_file=data/example_problem.jsonl\nReading samples...\n6it [00:00, 3397.11it/s]\nRunning example suites...\n100%|...| 6/6 [00:03\u003c00:00,  1.96it/s]\nWriting results to data/example_samples.jsonl_results.jsonl...\n100%|...| 6/6 [00:00\u003c00:00, 6148.50it/s]\n{'pass@1': 0.4999999999999999}\n```\n\nBecause there is no unbiased way of estimating pass@k when there are fewer\nsamples than k, the script does not evaluate pass@k for these cases. To\nevaluate with other k values, pass `--k=\u003ccomma-separated-values-here\u003e`. For\nother options, see\n```\n$ evaluate_functional_correctness --help\n```\nHowever, we recommend that you use the default values for the rest.\n\n## Known Issues\n\nWhile evaluation uses very little memory, you might see the following error\nmessage when the system is running out of RAM. Since this may cause some\ncorrect programs to fail, we recommend that you free some memory and try again.\n```\nmalloc: can't allocate region\n```\n\n## Citation\n\nPlease cite using the following bibtex entry:\n\n```\n@article{chen2021codex,\n  title={Evaluating Large Language Models Trained on Code},\n  author={Mark Chen and Jerry Tworek and Heewoo Jun and Qiming Yuan and Henrique Ponde de Oliveira Pinto and Jared Kaplan and Harri Edwards and Yuri Burda and Nicholas Joseph and Greg Brockman and Alex Ray and Raul Puri and Gretchen Krueger and Michael Petrov and Heidy Khlaaf and Girish Sastry and Pamela Mishkin and Brooke Chan and Scott Gray and Nick Ryder and Mikhail Pavlov and Alethea Power and Lukasz Kaiser and Mohammad Bavarian and Clemens Winter and Philippe Tillet and Felipe Petroski Such and Dave Cummings and Matthias Plappert and Fotios Chantzis and Elizabeth Barnes and Ariel Herbert-Voss and William Hebgen Guss and Alex Nichol and Alex Paino and Nikolas Tezak and Jie Tang and Igor Babuschkin and Suchir Balaji and Shantanu Jain and William Saunders and Christopher Hesse and Andrew N. Carr and Jan Leike and Josh Achiam and Vedant Misra and Evan Morikawa and Alec Radford and Matthew Knight and Miles Brundage and Mira Murati and Katie Mayer and Peter Welinder and Bob McGrew and Dario Amodei and Sam McCandlish and Ilya Sutskever and Wojciech Zaremba},\n  year={2021},\n  eprint={2107.03374},\n  archivePrefix={arXiv},\n  primaryClass={cs.LG}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenai%2Fhuman-eval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenai%2Fhuman-eval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenai%2Fhuman-eval/lists"}