{"id":5932,"url":"https://github.com/huybery/Awesome-Code-LLM","name":"Awesome-Code-LLM","description":"๐Ÿ‘จโ€๐Ÿ’ป An awesome and curated list of best code-LLM for research.","projects_count":98,"last_synced_at":"2026-07-04T07:00:25.576Z","repository":{"id":178859023,"uuid":"662443651","full_name":"huybery/Awesome-Code-LLM","owner":"huybery","description":"๐Ÿ‘จโ€๐Ÿ’ป An awesome and curated list of best code-LLM for research.","archived":false,"fork":false,"pushed_at":"2024-12-10T08:10:54.000Z","size":325,"stargazers_count":1284,"open_issues_count":3,"forks_count":74,"subscribers_count":33,"default_branch":"main","last_synced_at":"2026-06-16T15:04:17.137Z","etag":null,"topics":["awesome","code-generation","large-language-models"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huybery.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-07-05T06:42:09.000Z","updated_at":"2026-06-11T12:55:22.000Z","dependencies_parsed_at":"2024-04-09T02:02:53.439Z","dependency_job_id":"5e7d1a0e-a6b5-4a86-907e-8453b4df6fc3","html_url":"https://github.com/huybery/Awesome-Code-LLM","commit_stats":null,"previous_names":["huybery/awesome-code-llm"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/huybery/Awesome-Code-LLM","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huybery%2FAwesome-Code-LLM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huybery%2FAwesome-Code-LLM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huybery%2FAwesome-Code-LLM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huybery%2FAwesome-Code-LLM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huybery","download_url":"https://codeload.github.com/huybery/Awesome-Code-LLM/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huybery%2FAwesome-Code-LLM/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35112708,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-04T02:00:05.987Z","response_time":113,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2024-01-07T00:40:29.562Z","updated_at":"2026-07-04T07:00:25.576Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":["๐Ÿ“š Awesome Code LLMs Papers","๐Ÿš€ Leaderboard","๐Ÿ“š Paper","๐Ÿš€ Top Code LLMs","๐Ÿš€ Awesome Code LLMs Leaderboard","Star History","Acknowledgement","๐Ÿ’ก Evaluation Toolkit:","News"],"sub_categories":["๐ŸŒŠ Awesome Code Pre-Training Papers","โ–ถ๏ธ Pre-Training","๐Ÿฌ Awesome Code Alignment Papers","โ–ถ๏ธ Instruction Tuning","โ–ถ๏ธ Alignment with Feedback","๐Ÿ‹ Awesome Code Prompting Papers","โ–ถ๏ธ Prompting","โ–ถ๏ธ Evaluation \u0026 Benchmark","โ–ถ๏ธ Using LLMs while coding","๐Ÿ™ Awesome Code Benchmark \u0026 Evaluation Papers","๐Ÿณ Awesome Code Instruction-Tuning Papers"],"readme":"\u003cdiv align=\"center\"\u003e\n \u003ch1\u003e๐Ÿ‘จโ€๐Ÿ’ป Awesome Code LLM\u003c/h1\u003e\n \u003ca href=\"https://awesome.re\"\u003e\n \u003cimg src=\"https://awesome.re/badge.svg\" alt=\"Awesome\"\u003e\n \u003c/a\u003e\n \u003ca href=\"https://img.shields.io/badge/PRs-Welcome-red\"\u003e\n \u003cimg src=\"https://img.shields.io/badge/PRs-Welcome-red\" alt=\"PRs Welcome\"\u003e\n \u003c/a\u003e\n \u003ca href=\"https://img.shields.io/github/last-commit/huybery/Awesome-Code-LLM?color=green\"\u003e\n \u003cimg src=\"https://img.shields.io/github/last-commit/huybery/Awesome-Code-LLM?color=green\" alt=\"Last Commit\"\u003e\n \u003c/a\u003e\n\u003c/div\u003e\n\n![](assets/banner.png)\n\n\u0026nbsp;\n\n## ๐Ÿ”† How to Contribute\n\nContributions are welcome!\nIf you have any resources, tools, papers, or insights related to Code LLMs, feel free to submit a pull request.\nLet's work together to make this project better!\n\n\u0026nbsp;\n\n## News\n\n- ๐Ÿ”ฅ๐Ÿ”ฅ๐Ÿ”ฅ **[2024-11-12]** [**Qwen2.5-Coder series**](https://huggingface.co/collections/Qwen/qwen25-66e81a666513e518adb90d9e) are released, offering six model sizes (0.5B, 1.5B, 3B, 7B, 14B, 32B), with Qwen2.5-Coder-32B-Instruct now the most powerful open-source code model.\n- ๐Ÿ”ฅ๐Ÿ”ฅ **[2024-11-08]** [OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models](https://arxiv.org/abs/2411.04905) is released.\n\n\u0026nbsp;\n\n## ๐Ÿงต Table of Contents\n\n- [๐Ÿงต Table of Contents](#-table-of-contents)\n- [๐Ÿš€ Top Code LLMs](#-top-code-llms)\n- [๐Ÿ’ก Evaluation Toolkit](#-evaluation-toolkit)\n- [๐Ÿš€ Awesome Code LLMs Leaderboard](#-awesome-code-llms-leaderboard)\n- [๐Ÿ“š Awesome Code LLMs Papers](#-awesome-code-llms-papers)\n - [๐ŸŒŠ Awesome Code Pre-Training Papers](#-awesome-code-pre-training-papers)\n - [๐Ÿณ Awesome Code Instruction-Tuning Papers](#-awesome-code-instruction-tuning-papers)\n - [๐Ÿฌ Awesome Code Alignment Papers](#-awesome-code-alignment-papers)\n - [๐Ÿ‹ Awesome Code Prompting Papers](#-awesome-code-prompting-papers)\n - [๐Ÿ™ Awesome Code Benchmark \\\u0026 Evaluation Papers](#-awesome-code-benchmark--evaluation-papers)\n- [๐Ÿ™Œ Contributors](#-contributors)\n- [Cite as](#cite-as)\n- [Acknowledgement](#acknowledgement)\n- [Star History](#star-history)\n\n\u0026nbsp;\n\n## ๐Ÿš€ Top Code LLMs\n###### Sort by HumanEval Pass@1\n\n| Rank | Model | Params | HumanEval | MBPP | Source |\n|------|-------------------------------------------------------------------------------------------------|---------|-----------|------|------------------------------------------------------------|\n| 1 | o1-mini-2024-09-12 | - | 97.6 | 93.9 | [paper](https://arxiv.org/abs/2409.12186) |\n| 2 | o1-preview-2024-09-12 | - | 95.1 | 93.4 | [paper](https://arxiv.org/abs/2409.12186) |\n| 3 | [Qwen2.5-Coder-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct) | 32B | 92.7 | 90.2 | [github](https://github.com/QwenLM/Qwen2.5-Coder) |\n| 4 | Claude-3.5-Sonnet-20241022 | - | 92.1 | 91.0 | [paper](https://arxiv.org/abs/2409.12186) |\n| 5 | GPT-4o-2024-08-06 | - | 92.1 | 86.8 | [paper](https://arxiv.org/abs/2409.12186) |\n| 6 | [Qwen2.5-Coder-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct) | 14B | 89.6 | 86.2 | [github](https://github.com/QwenLM/Qwen2.5-Coder) |\n| 7 | Claude-3.5-Sonnet-20240620 | - | 89.0 | 87.6 | [paper](https://arxiv.org/abs/2409.12186) |\n| 8 | GPT-4o-mini-2024-07-18 | - | 87.8 | 86.0 | [paper](https://arxiv.org/abs/2409.12186) |\n| 9 | [Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct) | 7B | 88.4 | 83.5 | [github](https://github.com/QwenLM/Qwen2.5-Coder) |\n| 10 | [DS-Coder-V2-Instruct](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct) | 21/236B | 85.4 | 89.4 | [github](https://github.com/deepseek-ai/DeepSeek-Coder-V2) |\n| 11 | [Qwen2.5-Coder-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct) | 3B | 84.1 | 73.6 | [github](https://github.com/QwenLM/Qwen2.5-Coder) |\n| 12 | [DS-Coder-V2-Lite-Instruct](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct) | 2.4/16B | 81.1 | 82.8 | [github](https://github.com/deepseek-ai/DeepSeek-Coder-V2) |\n| 13 | [CodeQwen1.5-7B-Chat](https://huggingface.co/Qwen/CodeQwen1.5-7B-Chat) | 7B | 83.5 | 70.6 | [github](https://github.com/QwenLM/CodeQwen1.5) |\n| 14 | [DeepSeek-Coder-33B-Instruct](https://hf.co/deepseek-ai/deepseek-coder-33b-instruct) | 33B | 79.3 | 70.0 | [github](https://github.com/deepseek-ai/DeepSeek-Coder) |\n| 15 | [DeepSeek-Coder-6.7B-Instruct](https://hf.co/deepseek-ai/deepseek-coder-6.7b-instruct) | 6.7B | 78.6 | 65.4 | [github](https://github.com/deepseek-ai/DeepSeek-Coder) |\n| 16 | GPT-3.5-Turbo | - | 76.2 | 70.8 | [github](https://github.com/deepseek-ai/DeepSeek-Coder) |\n| 17 | [CodeLlama-70B-Instruct](https://huggingface.co/meta-llama/CodeLlama-70b-Instruct-hf) | 70B | 72.0 | 77.8 | [paper](https://arxiv.org/abs/2308.12950) |\n| 18 | [Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) | 1.5B | 70.7 | 69.2 | [github](https://github.com/QwenLM/Qwen2.5-Coder) |\n| 19 | [StarCoder2-15B-Instruct-v0.1](https://huggingface.co/bigcode/starcoder2-15b-instruct-v0.1) | 15B | 67.7 | 78.0 | [paper](https://arxiv.org/abs/2305.06161) |\n| 20 | [Qwen2.5-Coder-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct) | 0.5B | 61.6 | 52.4 | [github](https://github.com/QwenLM/Qwen2.5-Coder) |\n| 21 | Pangu-Coder2 | 15B | 61.6 | - | [paper](https://arxiv.org/abs/2307.14936) |\n| 22 | [WizardCoder-15B](https://hf.co/WizardLM/WizardCoder-15B-V1.0) | 15B | 57.3 | 51.8 | [paper](https://arxiv.org/abs/2306.08568) |\n| 23 | [CodeQwen1.5-7B](https://huggingface.co/Qwen/CodeQwen1.5-7B) | 7B | 51.8 | 61.8 | [github](https://github.com/QwenLM/CodeQwen1.5) |\n| 24 | [CodeLlama-34B-Instruct](https://huggingface.co/meta-llama/CodeLlama-34b-Instruct-hf) | 34B | 48.2 | 61.1 | [paper](https://arxiv.org/abs/2308.12950) |\n| 25 | Code-Davinci-002 | - | 47.0 | - | [paper](https://arxiv.org/abs/2107.03374) |\n\n\u0026nbsp;\n\n## ๐Ÿ’ก Evaluation Toolkit:\n\n- [bigcode-evaluation-harness](https://github.com/bigcode-project/bigcode-evaluation-harness): A framework for the evaluation of autoregressive code generation language models.\n- [code-eval](https://github.com/abacaj/code-eval): A framework for the evaluation of autoregressive code generation language models on HumanEval.\n- [SandboxFusion](https://bytedance.github.io/SandboxFusion): A secure sandbox for running and judging code generated by LLMs.\n\n\u0026nbsp;\n\n## ๐Ÿš€ Awesome Code LLMs Leaderboard\n| Leaderboard | Description |\n|:--------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| [Evalperf Leaderboard](https://evalplus.github.io/evalperf.html) | Evaluating LLMs for Efficient Code Generation. |\n| [Aider Code Editing Leaderboard](https://aider.chat/docs/leaderboards/) | Measuring the LLMโ€™s coding ability, and whether it can write new code that integrates into existing code. |\n| [BigCodeBench Leaderboard](https://bigcode-bench.github.io) | BigCodeBench evaluates LLMs with practical and challenging programming tasks. |\n| [LiveCodeBench Leaderboard](https://livecodebench.github.io/leaderboard.html) | Holistic and Contamination Free Evaluation of Large Language Models for Code. |\n| [Big Code Models Leaderboard](https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard) | Compare performance of base multilingual code generation models on HumanEval benchmark and MultiPL-E. |\n| [BIRD Leaderboard](https://bird-bench.github.io) | BIRD contains over 12,751 unique question-SQL pairs, 95 big databases with a total size of 33.4 GB. It also covers more than 37 professional domains, such as blockchain, hockey, healthcare and education, etc. |\n| [CanAiCode Leaderboard](https://huggingface.co/spaces/mike-ravkine/can-ai-code-results) | CanAiCode Leaderboard |\n| [Coding LLMs Leaderboard](https://leaderboard.tabbyml.com) | Coding LLMs Leaderboard |\n| [CRUXEval Leaderboard](https://crux-eval.github.io/leaderboard.html) | CRUXEval is a benchmark complementary to HumanEval and MBPP measuring code reasoning, understanding, and execution capabilities! |\n| [EvalPlus Leaderboard](https://evalplus.github.io/leaderboard.html) | EvalPlus evaluates AI Coders with rigorous tests. |\n| [InfiBench Leaderboard](https://infi-coder.github.io/infibench/) | InfiBench is a comprehensive benchmark for code large language models evaluating model ability on answering freeform real-world questions in the code domain. |\n| [InterCode Leaderboard](https://intercode-benchmark.github.io) | InterCode is a benchmark for evaluating language models on the interactive coding task. Given a natural language request, an agent is asked to interact with a software system (e.g., database, terminal) with code to resolve the issue. |\n| [Program Synthesis Models Leaderboard](https://accubits.com/open-source-program-synthesis-models-leaderboard) | They created this leaderboard to help researchers easily identify the best open-source model with an intuitive leadership quadrant graph. They evaluate the performance of open-source code models to rank them based on their capabilities and market adoption. |\n| [Spider Leaderboard](https://yale-lily.github.io/spider) | Spider is a large-scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 Yale students. The goal of the Spider challenge is to develop natural language interfaces to cross-domain databases. |\n\n\u0026nbsp;\n\n\n## ๐Ÿ“š Awesome Code LLMs Papers\n\n### ๐ŸŒŠ Awesome Code Pre-Training Papers\n| Title | Venue | Date | Code | Resources |\n|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|-----------|------------------------------------------------------------|-----------------------------------------------------------------------------------|\n| ![Star](https://img.shields.io/github/stars/OpenCoder-llm/OpenCoder-llm.svg?style=social\u0026label=Star) \u003cbr\u003e [**OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models**](https://arxiv.org/abs/2411.04905) \u003cbr\u003e | `Preprint` | `2024.11` | [Github](https://github.com/OpenCoder-llm/OpenCoder-llm) | [HF](https://huggingface.co/infly/OpenCoder-8B-Instruct) |\n| ![Star](https://img.shields.io/github/stars/QwenLM/Qwen2.5-Coder.svg?style=social\u0026label=Star) \u003cbr\u003e [**Qwen2.5-Coder Technical Report**](https://arxiv.org/abs/2409.12186) \u003cbr\u003e | `Preprint` | `2024.09` | [Github](https://github.com/QwenLM/Qwen2.5-Coder) | [HF](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct) |\n| ![Star](https://img.shields.io/github/stars/deepseek-ai/DeepSeek-Coder-V2.svg?style=social\u0026label=Star) \u003cbr\u003e [**DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence**](https://arxiv.org/abs/2406.11931) \u003cbr\u003e | `Preprint` | `2024.06` | [Github](https://github.com/deepseek-ai/DeepSeek-Coder-V2) | [HF](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct) |\n| ![Star](https://img.shields.io/github/stars/bigcode-project/starcoder2.svg?style=social\u0026label=Star) \u003cbr\u003e [**StarCoder 2 and The Stack v2: The Next Generation**](https://arxiv.org/abs/2402.19173) \u003cbr\u003e | `Preprint` | `2024.02` | [Github](https://github.com/bigcode-project/starcoder2) | [HF](https://huggingface.co/bigcode) |\n| ![Star](https://img.shields.io/github/stars/deepseek-ai/DeepSeek-Coder.svg?style=social\u0026label=Star) \u003cbr\u003e [**DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence**](https://arxiv.org/abs/2401.14196) \u003cbr\u003e | `Preprint` | `2024.01` | [Github](https://github.com/deepseek-ai/DeepSeek-Coder) | [HF](https://huggingface.co/deepseek-ai/deepseek-coder-33b-instruct) |\n| ![Star](https://img.shields.io/github/stars/meta-llama/codellama.svg?style=social\u0026label=Star) \u003cbr\u003e [**Code Llama: Open Foundation Models for Code**](https://arxiv.org/abs/2308.12950) \u003cbr\u003e | `Preprint` | `2023.08` | [Github](https://github.com/meta-llama/codellama) | [HF](https://huggingface.co/meta-llama/CodeLlama-7b-hf) |\n| [**Textbooks Are All You Need**](https://arxiv.org/abs/2306.11644) \u003cbr\u003e | `Preprint` | `2023.06` | - | [HF](https://huggingface.co/microsoft/phi-1) |\n| ![Star](https://img.shields.io/github/stars/salesforce/CodeT5.svg?style=social\u0026label=Star) \u003cbr\u003e [**CodeT5+: Open Code Large Language Models for Code Understanding and Generation**](https://arxiv.org/abs/2305.07922) \u003cbr\u003e | `Preprint` | `2023.05` | [Github](https://github.com/salesforce/CodeT5) | [HF](https://huggingface.co/Salesforce/codet5p-16b) |\n| ![Star](https://img.shields.io/github/stars/bigcode-project/starcoder.svg?style=social\u0026label=Star) \u003cbr\u003e [**StarCoder: may the source be with you!**](https://arxiv.org/abs/2305.06161) \u003cbr\u003e | `Preprint` | `2023.05` | [Github](https://github.com/bigcode-project/starcoder) | [HF](https://huggingface.co/bigcode/starcoder) |\n| ![Star](https://img.shields.io/github/stars/salesforce/CodeGen.svg?style=social\u0026label=Star) \u003cbr\u003e [**CodeGen2: Lessons for Training LLMs on Programming and Natural Languages**](https://arxiv.org/abs/2305.02309) \u003cbr\u003e | `ICLR23` | `2023.05` | [Github](https://github.com/salesforce/CodeGen) | [HF](https://huggingface.co/Salesforce/codegen25-7b-multi_P) |\n| ![Star](https://img.shields.io/github/stars/THUDM/CodeGeeX.svg?style=social\u0026label=Star) \u003cbr\u003e [**CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X**](https://arxiv.org/abs/2303.17568) \u003cbr\u003e | `Preprint` | `2023.03` | [Github](https://github.com/THUDM/CodeGeeX) | [HF](https://huggingface.co/collections/THUDM/codegeex4-6694e777e98246f00632fcf1) |\n| [**SantaCoder: don't reach for the stars!**](https://arxiv.org/abs/2301.03988) \u003cbr\u003e | `Preprint` | `2023.01` | - | [HF](https://huggingface.co/bigcode/santacoder) |\n| ![Star](https://img.shields.io/github/stars/salesforce/CodeGen.svg?style=social\u0026label=Star) \u003cbr\u003e [**CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis**](https://arxiv.org/abs/2203.13474) \u003cbr\u003e | `ICLR'23` | `2022.03` | [Github](https://github.com/salesforce/CodeGen) | [HF](https://huggingface.co/Salesforce/codegen25-7b-multi_P) |\n| ![Star](https://img.shields.io/github/stars/openai/human-eval.svg?style=social\u0026label=Star) \u003cbr\u003e [**Evaluating Large Language Models Trained on Code**](https://arxiv.org/abs/2107.03374) \u003cbr\u003e | `Preprint` | `2021.07` | [Github](https://github.com/openai/human-eval) | - |\n\n\u0026nbsp;\n\n### ๐Ÿณ Awesome Code Instruction-Tuning Papers\n| Title | Venue | Date | Code | Resources |\n|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|-----------|-------------------------------------------------------|----------------------------------------------------------------|\n| ![Star](https://img.shields.io/github/stars/ise-uiuc/magicoder.svg?style=social\u0026label=Star) \u003cbr\u003e [**Magicoder: Source Code Is All You Need**](https://arxiv.org/abs/2312.02120) \u003cbr\u003e | `ICML'24` | `2023.12` | [Github](https://github.com/ise-uiuc/magicoder) | [HF](https://huggingface.co/ise-uiuc/Magicoder-DS-6.7B) |\n| ![Star](https://img.shields.io/github/stars/bigcode-project/octopack.svg?style=social\u0026label=Star) \u003cbr\u003e [**OctoPack: Instruction Tuning Code Large Language Models**](https://arxiv.org/abs/2308.07124) \u003cbr\u003e | `ICLR'24` | `2023.08` | [Github](https://github.com/bigcode-project/octopack) | [HF](https://huggingface.co/bigcode/octocoder) |\n| ![Star](https://img.shields.io/github/stars/nlpxucan/WizardLM.svg?style=social\u0026label=Star) \u003cbr\u003e [**WizardCoder: Empowering Code Large Language Models with Evol-Instruct**](https://arxiv.org/abs/2306.08568) \u003cbr\u003e | `Preprint` | `2023.07` | [Github](https://github.com/nlpxucan/WizardLM) | [HF](https://huggingface.co/WizardLMTeam/WizardCoder-15B-V1.0) |\n| ![Star](https://img.shields.io/github/stars/sahil280114/codealpaca.svg?style=social\u0026label=Star) \u003cbr\u003e [**Code Alpaca: An Instruction-following LLaMA Model trained on code generation instructions**](https://github.com/sahil280114/codealpaca) \u003cbr\u003e | `Preprint` | `2023.xx` | [Github](https://github.com/sahil280114/codealpaca) | [HF](https://huggingface.co/datasets/sahil2801/CodeAlpaca-20k) |\n\n\u0026nbsp;\n\n\n### ๐Ÿฌ Awesome Code Alignment Papers\n| Title | Venue | Date | Code | Resources |\n|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|-----------|---------------------------------------------------------------|-----------|\n| [**ProSec: Fortifying Code LLMs with Proactive Security Alignment**](https://arxiv.org/abs/2411.12882) \u003cbr\u003e | `Preprint` | `2024.11` | - | - |\n| [**PLUM: Preference Learning Plus Test Cases Yields Better Code Language Models**](https://arxiv.org/abs/2406.06887) \u003cbr\u003e | `Preprint` | `2024.06` | - | - |\n| [**PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback**](https://arxiv.org/abs/2307.14936) \u003cbr\u003e | `Preprint` | `2023.07` | - | - |\n| ![Star](https://img.shields.io/github/stars/Zyq-scut/RLTF.svg?style=social\u0026label=Star) \u003cbr\u003e [**RLTF: Reinforcement Learning from Unit Test Feedback**](https://arxiv.org/abs/2307.04349) \u003cbr\u003e | `Preprint` | `2023.07` | [Github](https://github.com/Zyq-scut/RLTF) | - |\n| ![Star](https://img.shields.io/github/stars/reddy-lab-code-research/PPOCoder.svg?style=social\u0026label=Star) \u003cbr\u003e [**Execution-based Code Generation using Deep Reinforcement Learning**](https://arxiv.org/abs/2301.13816) \u003cbr\u003e | `TMLR'23` | `2023.01` | [Github](https://github.com/reddy-lab-code-research/PPOCoder) | - |\n| ![Star](https://img.shields.io/github/stars/salesforce/CodeRL.svg?style=social\u0026label=Star) \u003cbr\u003e [**CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning**](https://arxiv.org/abs/2207.01780) \u003cbr\u003e | `NeurIPS'22` | `2022.07` | [Github](https://github.com/salesforce/CodeRL) | - |\n \n\u0026nbsp;\n\n### ๐Ÿ‹ Awesome Code Prompting Papers\n| Title | Venue | Date | Code | Resources |\n|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|-----------|------------------------------------------------------------------------|-----------|\n| ![Star](https://img.shields.io/github/stars/YerbaPage/MGDebugger.svg?style=social\u0026label=Star) \u003cbr\u003e [**From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging**](https://arxiv.org/abs/2410.01215) \u003cbr\u003e | `Preprint` | `2024.10` | [Github](https://github.com/YerbaPage/MGDebugger) | - |\n| ![Star](https://img.shields.io/github/stars/Hambaobao/HCP-Coder.svg?style=social\u0026label=Star) \u003cbr\u003e [**Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMs**](https://arxiv.org/abs/2406.18294) \u003cbr\u003e | `AAAI'25` | `2024.06` | [Github](https://github.com/Hambaobao/HCP-Coder) | - |\n| ![Star](https://img.shields.io/github/stars/FloridSleeves/LLMDebugger.svg?style=social\u0026label=Star) \u003cbr\u003e [**Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step**](https://arxiv.org/abs/2402.16906) \u003cbr\u003e | `ACL'24` | `2024.02` | [Github](https://github.com/FloridSleeves/LLMDebugger) | - |\n| [**SelfEvolve: A Code Evolution Framework via Large Language Models**](https://arxiv.org/abs/2306.02907) \u003cbr\u003e | `Preprint` | `2023.06` | - | - |\n| ![Star](https://img.shields.io/github/stars/theoxo/self-repair.svg?style=social\u0026label=Star) \u003cbr\u003e [**Demystifying GPT Self-Repair for Code Generation**](https://arxiv.org/abs/2306.09896) \u003cbr\u003e | `ICLR'24` | `2023.06` | [Github](https://github.com/theoxo/self-repair) | - |\n| [**Teaching Large Language Models to Self-Debug**](https://arxiv.org/abs/2304.05128) \u003cbr\u003e | `ICLR'24` | `2023.06` | - | - |\n| ![Star](https://img.shields.io/github/stars/niansong1996/lever.svg?style=social\u0026label=Star) \u003cbr\u003e [**LEVER: Learning to Verify Language-to-Code Generation with Execution**](https://arxiv.org/abs/2302.08468) \u003cbr\u003e | `ICML'23` | `2023.02` | [Github](https://github.com/niansong1996/lever) | - |\n| ![Star](https://img.shields.io/github/stars/facebookresearch/coder_reviewer_reranking.svg?style=social\u0026label=Star) \u003cbr\u003e [**Coder Reviewer Reranking for Code Generation**](https://arxiv.org/abs/2211.16490) \u003cbr\u003e | `ICML'23` | `2022.11` | [Github](https://github.com/facebookresearch/coder_reviewer_reranking) | - |\n| ![Star](https://img.shields.io/github/stars/microsoft/CodeT.svg?style=social\u0026label=Star) \u003cbr\u003e [**CodeT: Code Generation with Generated Tests**](https://arxiv.org/abs/2207.10397) \u003cbr\u003e | `ICLR'23` | `2022.07` | [Github](https://github.com/microsoft/CodeT) | - |\n\n\u0026nbsp;\n\n### ๐Ÿ™ Awesome Code Benchmark \u0026 Evaluation Papers\n| Dataset | Title | Venue | Date | Code | Resources |\n|:------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|-----------|-----------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|\n| `CodeArena` | ![Star](https://img.shields.io/github/stars/QwenLM/Qwen2.5-Coder.svg?style=social\u0026label=Star) \u003cbr\u003e [**Evaluating and Aligning CodeLLMs on Human Preference**](https://arxiv.org/abs/2412.05210) \u003cbr\u003e | `Preprint` | `2024.12` | [Github](https://github.com/QwenLM/Qwen2.5-Coder/tree/main/qwencoder-eval/instruct/CodeArena) | [HF](https://huggingface.co/datasets/CSJianYang/CodeArena) |\n| `FullStack Bench` | ![Star](https://img.shields.io/github/stars/bytedance/FullStackBench.svg?style=social\u0026label=Star) \u003cbr\u003e [**FullStack Bench: Evaluating LLMs as Full Stack Coders**](https://arxiv.org/abs/2412.00535) \u003cbr\u003e | `Preprint` | `2024.12` | [Github](https://github.com/bytedance/FullStackBench) | [HF](https://huggingface.co/datasets/ByteDance/FullStackBench) [Github](https://github.com/bytedance/SandboxFusion) |\n| `GitChameleon` | ![Star](https://img.shields.io/github/stars/NizarIslah/GitChameleon.svg?style=social\u0026label=Star) \u003cbr\u003e [**GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models**](https://arxiv.org/abs/2411.05830) \u003cbr\u003e | `Preprint` | `2024.11` | [Github](https://github.com/NizarIslah/GitChameleon) | - |\n| `Evalperf` | ![Star](https://img.shields.io/github/stars/evalplus/evalplus.svg?style=social\u0026label=Star) \u003cbr\u003e [**Evaluating Language Models for Efficient Code Generation**](https://arxiv.org/abs/2408.06450) \u003cbr\u003e | `COLM'24` | `2024.08` | [Github](https://github.com/evalplus/evalplus) | [HF](https://huggingface.co/evalplus) |\n| `LiveCodeBench` | ![Star](https://img.shields.io/github/stars/LiveCodeBench/LiveCodeBench.svg?style=social\u0026label=Star) \u003cbr\u003e [**LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code**](https://arxiv.org/abs/2403.07974) \u003cbr\u003e | `Preprint` | `2024.03` | [Github](https://github.com/LiveCodeBench/LiveCodeBench) | [HF](https://huggingface.co/datasets/livecodebench/code_generation_lite) |\n| `DevBench` | ![Star](https://img.shields.io/github/stars/open-compass/DevBench.svg?style=social\u0026label=Star) \u003cbr\u003e [**DevBench: A Comprehensive Benchmark for Software Development**](https://arxiv.org/abs/2403.08604) \u003cbr\u003e | `Preprint` | `2024.03` | [Github](https://github.com/open-compass/DevBench) | - |\n| `SWE-bench` | ![Star](https://img.shields.io/github/stars/princeton-nlp/SWE-bench.svg?style=social\u0026label=Star) \u003cbr\u003e [**SWE-bench: Can Language Models Resolve Real-World GitHub Issues?**](https://arxiv.org/abs/2310.06770) \u003cbr\u003e | `ICLR'24` | `2024.03` | [Github](https://github.com/princeton-nlp/SWE-bench) | [HF](https://huggingface.co/datasets/princeton-nlp/SWE-bench) |\n| `CrossCodeEval` | ![Star](https://img.shields.io/github/stars/amazon-science/cceval.svg?style=social\u0026label=Star) \u003cbr\u003e [**CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion**](https://arxiv.org/abs/2306.03091) \u003cbr\u003e | `NeurIPS'23` | `2023.11` | [Github](https://github.com/amazon-science/cceval) | - |\n| `RepoCoder` | ![Star](https://img.shields.io/github/stars/microsoft/CodeT.svg?style=social\u0026label=Star) \u003cbr\u003e [**Repository-Level Code Completion Through Iterative Retrieval and Generation**](https://arxiv.org/abs/2306.03091) \u003cbr\u003e | `EMNLP'23` | `2023.10` | [Github](https://github.com/microsoft/CodeT/tree/main/RepoCoder) | - |\n| `LongCoder` | ![Star](https://img.shields.io/github/stars/microsoft/CodeBERT.svg?style=social\u0026label=Star) \u003cbr\u003e [**LongCoder: A Long-Range Pre-trained Language Model for Code Completion**](https://arxiv.org/abs/2306.14893) \u003cbr\u003e | `ICML'23` | `2023.10` | [Github](https://github.com/microsoft/CodeBERT) | - |\n| - | [**Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation**](https://arxiv.org/abs/2308.10335) \u003cbr\u003e | `Preprint` | `2023.08` | - | - |\n| `BioCoder` | ![Star](https://img.shields.io/github/stars/gersteinlab/BioCoder.svg?style=social\u0026label=Star) \u003cbr\u003e [**BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models**](https://arxiv.org/abs/2308.16458) \u003cbr\u003e | `ISMB'24` | `2023.08` | [Github](https://github.com/gersteinlab/BioCoder) | - |\n| `RepoBench` | ![Star](https://img.shields.io/github/stars/Leolty/repobench.svg?style=social\u0026label=Star) \u003cbr\u003e [**RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems**](https://arxiv.org/abs/2306.03091) \u003cbr\u003e | `ICLR'24` | `2023.06` | [Github](https://github.com/Leolty/repobench) | [HF](https://huggingface.co/datasets/tianyang/repobench_python_v1.1) |\n| `Evalplus` | ![Star](https://img.shields.io/github/stars/evalplus/evalplus.svg?style=social\u0026label=Star) \u003cbr\u003e [**Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation**](https://arxiv.org/abs/2305.01210) \u003cbr\u003e | `NeurIPS'23` | `2023.05` | [Github](https://github.com/evalplus/evalplus) | [HF](https://huggingface.co/evalplus) |\n| `Coeditor` | ![Star](https://img.shields.io/github/stars/MrVPlusOne/Coeditor.svg?style=social\u0026label=Star) \u003cbr\u003e [**Coeditor: Leveraging Contextual Changes for Multi-round Code Auto-editing**](https://arxiv.org/abs/2305.18584) \u003cbr\u003e | `ICLR'24` | `2023.05` | [Github](https://github.com/MrVPlusOne/Coeditor) | - |\n| `DS-1000` | ![Star](https://img.shields.io/github/stars/xlang-ai/DS-1000.svg?style=social\u0026label=Star) \u003cbr\u003e [**DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation**](https://arxiv.org/abs/2211.11501) \u003cbr\u003e | `ICML'23` | `2022.11` | [Github](https://github.com/xlang-ai/DS-1000) | [HF](https://huggingface.co/datasets/xlangai/DS-1000) |\n| `MultiPL-E` | ![Star](https://img.shields.io/github/stars/nuprl/MultiPL-E.svg?style=social\u0026label=Star) \u003cbr\u003e [**MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation**](https://arxiv.org/abs/2208.08227) \u003cbr\u003e | `Preprint` | `2022.08` | [Github](https://github.com/nuprl/MultiPL-E) | [HF](https://huggingface.co/datasets/xlangai/DS-1000) |\n| `MBPP` | ![Star](https://img.shields.io/github/stars/google-research/google-research.svg?style=social\u0026label=Star) \u003cbr\u003e [**Program Synthesis with Large Language Models**](https://arxiv.org/abs/2108.07732) \u003cbr\u003e | `Preprint` | `2021.08` | [Github](https://github.com/google-research/google-research/blob/master/mbpp/README.md) | [HF](https://huggingface.co/datasets/nuprl/MultiPL-E) |\n| `APPS` | ![Star](https://img.shields.io/github/stars/hendrycks/apps.svg?style=social\u0026label=Star) \u003cbr\u003e [**Measuring Coding Challenge Competence With APPS**](https://arxiv.org/abs/2105.09938) \u003cbr\u003e | `NeurIPS'21` | `2021.05` | [Github](https://github.com/hendrycks/apps) | [HF](https://huggingface.co/datasets/codeparrot/apps) |\n\n\u0026nbsp;\n\n## ๐Ÿ™Œ Contributors\n\n\u003ca href=\"https://github.com/huybery\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/13436140?v=4\" width=\"50\" /\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/Yangjiaxi\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/6203054?v=4\" width=\"50\" /\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/GanjinZero\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/19466330?v=4\" width=\"50\" /\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/TyDunn\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/13314504?v=4\" width=\"50\" /\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/Hambaobao\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/48345096?v=4\" width=\"50\" /\u003e\u003c/a\u003e\n\nThis is an active repository and your contributions are always welcome! If you have any question about this opinionated list, do not hesitate to contact me `huybery@gmail.com`.\n\n\u0026nbsp;\n\n## Cite as\n\n```\n@software{awesome-code-llm,\n author = {Binyuan Hui, Lei Zhang},\n title = {An awesome and curated list of best code-LLM for research},\n howpublished = {\\url{https://github.com/huybery/Awesome-Code-LLM}},\n year = 2023,\n}\n```\n\n\u0026nbsp;\n\n## Acknowledgement\n\nThis project is inspired by [Awesome-LLM](https://github.com/Hannibal046/Awesome-LLM).\n\n\u0026nbsp;\n\n## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=huybery/Awesome-Code-LLM\u0026type=Date)](https://star-history.com/#huybery/Awesome-Code-LLM\u0026Date)\n\n\n**[โฌ† Back to ToC](#-table-of-contents)**\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/huybery%2Fawesome-code-llm/projects"}