{"id":13576358,"url":"https://github.com/bin123apple/AutoCoder","last_synced_at":"2025-04-05T05:31:37.070Z","repository":{"id":239662559,"uuid":"799753842","full_name":"bin123apple/AutoCoder","owner":"bin123apple","description":"We introduced a new model designed for the Code generation task. Its test accuracy on the HumanEval base dataset surpasses that of GPT-4 Turbo (April 2024) and GPT-4o. ","archived":false,"fork":false,"pushed_at":"2024-07-06T21:00:06.000Z","size":27022,"stargazers_count":810,"open_issues_count":6,"forks_count":69,"subscribers_count":15,"default_branch":"main","last_synced_at":"2024-11-05T12:33:29.469Z","etag":null,"topics":["code-generation","code-interpreter","humaneval","llm","nlp","nlp-machine-learning","text-generation"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2405.14906","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bin123apple.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-13T03:06:16.000Z","updated_at":"2024-11-05T09:22:55.000Z","dependencies_parsed_at":"2024-05-13T22:50:42.552Z","dependency_job_id":"33da21c2-774d-4d93-96d6-3e58efffaafb","html_url":"https://github.com/bin123apple/AutoCoder","commit_stats":null,"previous_names":["bin123apple/autocoder"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bin123apple%2FAutoCoder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bin123apple%2FAutoCoder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bin123apple%2FAutoCoder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bin123apple%2FAutoCoder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bin123apple","download_url":"https://codeload.github.com/bin123apple/AutoCoder/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247294462,"owners_count":20915334,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code-generation","code-interpreter","humaneval","llm","nlp","nlp-machine-learning","text-generation"],"created_at":"2024-08-01T15:01:09.595Z","updated_at":"2025-04-05T05:31:32.036Z","avatar_url":"https://github.com/bin123apple.png","language":"Python","readme":"# AutoCoder\n\n## News :fire: \n\nA new model [AutoCoder_QW_7B](https://huggingface.co/Bin12345/AutoCoder_QW_7B) is uploaded. In this model, We fixed the previous problem that the model will only start the code interpreter when you ask it to *verify* its code. \n\nThe base model of AutoCode_QW_7B is [CodeQwen1.5-7b](https://huggingface.co/Qwen/CodeQwen1.5-7B-Chat).\n\n## Introduction :mega:\nWe introduced a new model designed for the Code generation task. Its test accuracy on the HumanEval base dataset surpasses that of GPT-4 Turbo (April 2024). (**90.9% vs 90.2%**).\n\nAdditionally, compared to previous open-source models, AutoCoder offers a new feature: it can **automatically install the required packages** and attempt to run the code until it deems there are no issues, **whenever the user wishes to execute the code**.\n\n* Difference between the code interpreter of AutoCoder and the GPT-4 Turbo:\n\nBelow are the video demos for the code interpreter comparison between GPT-4 Turbo and AutoCoder: \n\nGPT-4o can not access the external library.\n\n[GPT-4o](https://github.com/bin123apple/AutoCoder/assets/99925255/be47b449-4e8a-4b77-981b-ec79b15970cc)\n\nAutoCoder can automatically install the required packages. This feature expands the scope of code interpreter's application.\n\n[AutoCoder](https://github.com/bin123apple/AutoCoder/assets/99925255/1893f904-c1f2-4f59-9ec5-45b69efcc26a)\n\n* Difference between the code interpreter of AutoCoder and the current open-source code interpreter [OpenCodeInterpreter](https://opencodeinterpreter.github.io/):\n\nThe code interpreter of AutoCoder, like GPT-4 Turbo, is only called when the user has a need to verify the code, while OpenCodeInterpreter runs all generated python code.\n\n## Model :gift:\nThe Model is avaliable on Huggingface:\n \n[AutoCoder (33B)](https://huggingface.co/Bin12345/AutoCoder)\n[AutoCoder-S (6.7B)](https://huggingface.co/Bin12345/AutoCoder_S_6.7B)\n\nThe base models of AutoCoder (33B) and AutoCoder-S (6.7B) are deepseeker-coder.\n\n[AutoCoder_QW_7B](https://huggingface.co/Bin12345/AutoCoder_QW_7B)\n\nThe base model of AutoCoder_QW_7B is CodeQwen1.5-7b.\n\n## Quick Start :rocket:\n1. Create the conda env\n\n```\nconda create -n AutoCoder python=3.11\nconda activate AutoCoder\npip install -r requirements.txt\n```\n\n2. Test on HumanEval **90.9% on base, 78.0% on base + extra**. (Skip to Step 5, if you don't want to test its performance on benchmarks)\n\n```\ncd Evaluation\npython test_humaneval.py\n```\nYou will receive a file named AutoCoder_HumanEval+.jsonl, which follows the EvalPlus format, after this step.\n\nThen follow the testing framework of the [EvalPlus GitHub](https://github.com/evalplus/evalplus). You will see the results. \n\n**NOTE**: \n* Don't forget to use evalplus's `evalplus.sanitize` to post-process the code. \n* If you don't use the greedy method (for example set the `do_sample=True`) for the code generation. You will probably see the different results.\n\n3. Test on MBPP **82.5% on base, 70.6% on base + extra**. (Skip to Step 5, if you don't want to test its performance on benchmarks)\n\n```\npython test_humaneval.py\n```\n\nPost-process to delete the nature language for testing\n```\npython postprocess_mbpp.py\n```\nYour will get a AutoCoder_Mbpp+-sanitized.jsonl file after this step, it extracted all the code blocks. \nThen, directly test it by using [EvalPlus GitHub](https://github.com/evalplus/evalplus) (You don't need to use to use evalplus's `evalplus.sanitize` to post-process the code this time).\n\n4. Test on DS-1000. (Skip to Step 5, if you don't want to test its performance on benchmarks)\n\n```\npython test_ds1000.py\n```\n\nYour will get a jsonl file after this step, it extracted all the code blocks. \nThen, directly test it by using [DS-1000 GitHub](https://github.com/xlang-ai/DS-1000).\n\n5. Web demo (Include code interpreter)\n\nInstall gradio and Run:\n\n```\npip install gradio==3.48.0\ncd /Web_demo\npython chatbot.py\n```\n\n## **NOTE** :warning:\n* We suggest to set `do_sample = True` (default setting here) while using the code interpreter.\n\n* It would be preferable to use Linux for deploying everything.\n\n## Contact :email:\nIf you have any inquiries, please feel free to raise an issue or reach out to leib2765@gmail.com.\n\n## Citation :book:\n```\n@misc{lei2024autocoder,\n      title={AutoCoder: Enhancing Code Large Language Model with \\textsc{AIEV-Instruct}}, \n      author={Bin Lei and Yuchen Li and Qiuwu Chen},\n      year={2024},\n      eprint={2405.14906},\n      archivePrefix={arXiv},\n      primaryClass={cs.SE}\n}\n```\n\n## Acknowledgments :pray:\nThanks to Tianyu Zheng, the first author of the [OpenCodeInterpreter](https://opencodeinterpreter.github.io/), for guidance on some technical details.\n\n","funding_links":[],"categories":["Python","📦 Legacy \u0026 Inactive Projects","Language Models \u0026 Engines"],"sub_categories":["Other IDEs"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbin123apple%2FAutoCoder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbin123apple%2FAutoCoder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbin123apple%2FAutoCoder/lists"}