Awesome-Code-LLM

👨‍💻 An awesome and curated list of best code-LLM for research.
https://github.com/huybery/Awesome-Code-LLM

Last synced: 15 days ago
JSON representation

Acknowledgement
- 🐙 Awesome Code Benchmark & Evaluation Papers
  - Awesome-LLM
🚀 Awesome Code LLMs Leaderboard
- Big Code Models Leaderboard - E. |
- CanAiCode Leaderboard
- EvalPlus Leaderboard
- CRUXEval Leaderboard
- Spider Leaderboard - scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 Yale students. The goal of the Spider challenge is to develop natural language interfaces to cross-domain databases. |
- Evalperf Leaderboard
- Aider Code Editing Leaderboard
- LiveCodeBench Leaderboard
- InfiBench Leaderboard - world questions in the code domain. |
- Program Synthesis Models Leaderboard - source model with an intuitive leadership quadrant graph. They evaluate the performance of open-source code models to rank them based on their capabilities and market adoption. |
- BigCodeBench Leaderboard
📚 Awesome Code LLMs Papers
- 🐬 Awesome Code Alignment Papers
 - **PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback** - | - |
 - **PLUM: Preference Learning Plus Test Cases Yields Better Code Language Models** - | - |
 - Star - scut/RLTF) | - |
 - Star - based Code Generation using Deep Reinforcement Learning**](https://arxiv.org/abs/2301.13816) | `TMLR'23` | `2023.01` | [Github](https://github.com/reddy-lab-code-research/PPOCoder) | - |
 - Star - |
 - **ProSec: Fortifying Code LLMs with Proactive Security Alignment** - | - |
 - Star - scut/RLTF) | - |
 - Star - based Code Generation using Deep Reinforcement Learning**](https://arxiv.org/abs/2301.13816) | `TMLR'23` | `2023.01` | [Github](https://github.com/reddy-lab-code-research/PPOCoder) | - |
 - Star - |
- 🐙 Awesome Code Benchmark & Evaluation Papers
 - **Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation** - | - |
 - Star - Level Code Completion Through Iterative Retrieval and Generation**](https://arxiv.org/abs/2306.03091) | `EMNLP'23` | `2023.10` | [Github](https://github.com/microsoft/CodeT/tree/main/RepoCoder) | - |
 - Star - Switching Capabilities of Code Generation Models**](https://arxiv.org/abs/2411.05830) | `Preprint` | `2024.11` | [Github](https://github.com/NizarIslah/GitChameleon) | - |
 - Star
 - Star
 - Star - compass/DevBench) | - |
 - Star - bench: Can Language Models Resolve Real-World GitHub Issues?**](https://arxiv.org/abs/2310.06770) | `ICLR'24` | `2024.03` | [Github](https://github.com/princeton-nlp/SWE-bench) | [HF](https://huggingface.co/datasets/princeton-nlp/SWE-bench) |
 - Star - File Code Completion**](https://arxiv.org/abs/2306.03091) | `NeurIPS'23` | `2023.11` | [Github](https://github.com/amazon-science/cceval) | - |
 - Star - Range Pre-trained Language Model for Code Completion**](https://arxiv.org/abs/2306.14893) | `ICML'23` | `2023.10` | [Github](https://github.com/microsoft/CodeBERT) | - |
 - Star - |
 - Star - Level Code Auto-Completion Systems**](https://arxiv.org/abs/2306.03091) | `ICLR'24` | `2023.06` | [Github](https://github.com/Leolty/repobench) | [HF](https://huggingface.co/datasets/tianyang/repobench_python_v1.1) |
 - Star - round Code Auto-editing**](https://arxiv.org/abs/2305.18584) | `ICLR'24` | `2023.05` | [Github](https://github.com/MrVPlusOne/Coeditor) | - |
 - Star - 1000: A Natural and Reliable Benchmark for Data Science Code Generation**](https://arxiv.org/abs/2211.11501) | `ICML'23` | `2022.11` | [Github](https://github.com/xlang-ai/DS-1000) | [HF](https://huggingface.co/datasets/xlangai/DS-1000) |
 - Star - E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation**](https://arxiv.org/abs/2208.08227) | `Preprint` | `2022.08` | [Github](https://github.com/nuprl/MultiPL-E) | [HF](https://huggingface.co/datasets/xlangai/DS-1000) |
 - Star
 - Star
 - Star
 - Star - Switching Capabilities of Code Generation Models**](https://arxiv.org/abs/2411.05830) | `Preprint` | `2024.11` | [Github](https://github.com/NizarIslah/GitChameleon) | - |
 - Star
 - Star
 - Star - compass/DevBench) | - |
 - Star - bench: Can Language Models Resolve Real-World GitHub Issues?**](https://arxiv.org/abs/2310.06770) | `ICLR'24` | `2024.03` | [Github](https://github.com/princeton-nlp/SWE-bench) | [HF](https://huggingface.co/datasets/princeton-nlp/SWE-bench) |
 - Star - File Code Completion**](https://arxiv.org/abs/2306.03091) | `NeurIPS'23` | `2023.11` | [Github](https://github.com/amazon-science/cceval) | - |
 - Star - Range Pre-trained Language Model for Code Completion**](https://arxiv.org/abs/2306.14893) | `ICML'23` | `2023.10` | [Github](https://github.com/microsoft/CodeBERT) | - |
 - Star - |
 - Star - Level Code Auto-Completion Systems**](https://arxiv.org/abs/2306.03091) | `ICLR'24` | `2023.06` | [Github](https://github.com/Leolty/repobench) | [HF](https://huggingface.co/datasets/tianyang/repobench_python_v1.1) |
 - Star - round Code Auto-editing**](https://arxiv.org/abs/2305.18584) | `ICLR'24` | `2023.05` | [Github](https://github.com/MrVPlusOne/Coeditor) | - |
 - Star - 1000: A Natural and Reliable Benchmark for Data Science Code Generation**](https://arxiv.org/abs/2211.11501) | `ICML'23` | `2022.11` | [Github](https://github.com/xlang-ai/DS-1000) | [HF](https://huggingface.co/datasets/xlangai/DS-1000) |
 - Star - E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation**](https://arxiv.org/abs/2208.08227) | `Preprint` | `2022.08` | [Github](https://github.com/nuprl/MultiPL-E) | [HF](https://huggingface.co/datasets/xlangai/DS-1000) |
 - Star
- 🐳 Awesome Code Instruction-Tuning Papers
 - Star - uiuc/magicoder) | [HF](https://huggingface.co/ise-uiuc/Magicoder-DS-6.7B) |
 - Star - project/octopack) | [HF](https://huggingface.co/bigcode/octocoder) |
 - Star - Instruct**](https://arxiv.org/abs/2306.08568) | `Preprint` | `2023.07` | [Github](https://github.com/nlpxucan/WizardLM) | [HF](https://huggingface.co/WizardLMTeam/WizardCoder-15B-V1.0) |
 - Star - following LLaMA Model trained on code generation instructions**](https://github.com/sahil280114/codealpaca) | `Preprint` | `2023.xx` | [Github](https://github.com/sahil280114/codealpaca) | [HF](https://huggingface.co/datasets/sahil2801/CodeAlpaca-20k) |
 - Star - uiuc/magicoder) | [HF](https://huggingface.co/ise-uiuc/Magicoder-DS-6.7B) |
 - Star - project/octopack) | [HF](https://huggingface.co/bigcode/octocoder) |
 - Star - Instruct**](https://arxiv.org/abs/2306.08568) | `Preprint` | `2023.07` | [Github](https://github.com/nlpxucan/WizardLM) | [HF](https://huggingface.co/WizardLMTeam/WizardCoder-15B-V1.0) |
 - Star - following LLaMA Model trained on code generation instructions**](https://github.com/sahil280114/codealpaca) | `Preprint` | `2023.xx` | [Github](https://github.com/sahil280114/codealpaca) | [HF](https://huggingface.co/datasets/sahil2801/CodeAlpaca-20k) |
- 🌊 Awesome Code Pre-Training Papers
 - **Textbooks Are All You Need** - | [HF](https://huggingface.co/microsoft/phi-1) |
 - **SantaCoder: don't reach for the stars!** - | [HF](https://huggingface.co/bigcode/santacoder) |
 - Star - Tier Code Large Language Models**](https://arxiv.org/abs/2411.04905) | `Preprint` | `2024.11` | [Github](https://github.com/OpenCoder-llm/OpenCoder-llm) | [HF](https://huggingface.co/infly/OpenCoder-8B-Instruct) |
 - Star - Coder Technical Report**](https://arxiv.org/abs/2409.12186) | `Preprint` | `2024.09` | [Github](https://github.com/QwenLM/Qwen2.5-Coder) | [HF](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct) |
 - Star - Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence**](https://arxiv.org/abs/2406.11931) | `Preprint` | `2024.06` | [Github](https://github.com/deepseek-ai/DeepSeek-Coder-V2) | [HF](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct) |
 - Star - project/starcoder2) | [HF](https://huggingface.co/bigcode) |
 - Star - Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence**](https://arxiv.org/abs/2401.14196) | `Preprint` | `2024.01` | [Github](https://github.com/deepseek-ai/DeepSeek-Coder) | [HF](https://huggingface.co/deepseek-ai/deepseek-coder-33b-instruct) |
 - Star - llama/codellama) | [HF](https://huggingface.co/meta-llama/CodeLlama-7b-hf) |
 - Star - 16b) |
 - Star - project/starcoder) | [HF](https://huggingface.co/bigcode/starcoder) |
 - Star - Turn Program Synthesis**](https://arxiv.org/abs/2203.13474) | `ICLR'23` | `2022.03` | [Github](https://github.com/salesforce/CodeGen) | [HF](https://huggingface.co/Salesforce/codegen25-7b-multi_P) |
 - Star - Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X**](https://arxiv.org/abs/2303.17568) | `Preprint` | `2023.03` | [Github](https://github.com/THUDM/CodeGeeX) | [HF](https://huggingface.co/collections/THUDM/codegeex4-6694e777e98246f00632fcf1) |
 - Star - eval) | - |
 - Star - Tier Code Large Language Models**](https://arxiv.org/abs/2411.04905) | `Preprint` | `2024.11` | [Github](https://github.com/OpenCoder-llm/OpenCoder-llm) | [HF](https://huggingface.co/infly/OpenCoder-8B-Instruct) |
 - Star - Coder Technical Report**](https://arxiv.org/abs/2409.12186) | `Preprint` | `2024.09` | [Github](https://github.com/QwenLM/Qwen2.5-Coder) | [HF](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct) |
 - Star - Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence**](https://arxiv.org/abs/2406.11931) | `Preprint` | `2024.06` | [Github](https://github.com/deepseek-ai/DeepSeek-Coder-V2) | [HF](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct) |
 - Star - project/starcoder2) | [HF](https://huggingface.co/bigcode) |
 - Star - Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence**](https://arxiv.org/abs/2401.14196) | `Preprint` | `2024.01` | [Github](https://github.com/deepseek-ai/DeepSeek-Coder) | [HF](https://huggingface.co/deepseek-ai/deepseek-coder-33b-instruct) |
 - Star - llama/codellama) | [HF](https://huggingface.co/meta-llama/CodeLlama-7b-hf) |
 - Star - 16b) |
 - Star - project/starcoder) | [HF](https://huggingface.co/bigcode/starcoder) |
 - Star - 7b-multi_P) |
 - Star - Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X**](https://arxiv.org/abs/2303.17568) | `Preprint` | `2023.03` | [Github](https://github.com/THUDM/CodeGeeX) | [HF](https://huggingface.co/collections/THUDM/codegeex4-6694e777e98246f00632fcf1) |
 - Star - eval) | - |
- 🐋 Awesome Code Prompting Papers
 - **Teaching Large Language Models to Self-Debug** - | - |
 - **SelfEvolve: A Code Evolution Framework via Large Language Models** - | - |
 - Star - |
 - Star - by-step**](https://arxiv.org/abs/2402.16906) | `ACL'24` | `2024.02` | [Github](https://github.com/FloridSleeves/LLMDebugger) | - |
 - Star - Repair for Code Generation**](https://arxiv.org/abs/2306.09896) | `ICLR'24` | `2023.06` | [Github](https://github.com/theoxo/self-repair) | - |
 - Star - to-Code Generation with Execution**](https://arxiv.org/abs/2302.08468) | `ICML'23` | `2023.02` | [Github](https://github.com/niansong1996/lever) | - |
 - Star - |
 - Star - World Code Completion with Repository-Level Pretrained Code LLMs**](https://arxiv.org/abs/2406.18294) | `AAAI'25` | `2024.06` | [Github](https://github.com/Hambaobao/HCP-Coder) | - |
 - Star - |
 - Star - World Code Completion with Repository-Level Pretrained Code LLMs**](https://arxiv.org/abs/2406.18294) | `AAAI'25` | `2024.06` | [Github](https://github.com/Hambaobao/HCP-Coder) | - |
 - Star - by-step**](https://arxiv.org/abs/2402.16906) | `ACL'24` | `2024.02` | [Github](https://github.com/FloridSleeves/LLMDebugger) | - |
 - Star - Repair for Code Generation**](https://arxiv.org/abs/2306.09896) | `ICLR'24` | `2023.06` | [Github](https://github.com/theoxo/self-repair) | - |
 - Star - to-Code Generation with Execution**](https://arxiv.org/abs/2302.08468) | `ICML'23` | `2023.02` | [Github](https://github.com/niansong1996/lever) | - |
 - Star - |
 - Star - |
💡 Evaluation Toolkit:
🚀 Leaderboard
- paper
- ckpt - ai/DeepSeek-Coder) |
- ckpt
- ckpt
- paper
- ckpt
- paper
- paper
- [Source
- [Source
- [Source
- [Source
- [Source
- [Source
News
- 2024-11-12 - Coder-32B-Instruct now the most powerful open-source code model.
- 2024-11-08
📚 Paper
- ▶️ Alignment with Feedback
  - [Paper
  - [Paper
  - [Paper
- ▶️ Evaluation & Benchmark
  - [Paper
  - [Paper - tau Yih, Daniel Fried, Sida Wang, Tao Yu.* 2022.11
  - [Paper
  - [Paper - Guang Lou, Weizhu Chen.* 2023.10
  - [Paper
  - [Paper
  - [Paper - compass/DevBench)] *Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng, Kai Chen* 2024.3
  - [Paper
  - [Paper
  - [Paper
  - [Paper
- ▶️ Instruction Tuning
  - [Paper - project/octopack)] *Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, Shayne Longpre.* 2023.08
  - [Paper
  - [Paper - uiuc/magicoder)] *Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, Lingming Zhang* 2023.12
  - [Repo
- ▶️ Pre-Training
  - [Paper
  - [Paper
  - [Paper
  - [Paper
  - [Paper
  - [Paper
  - [Paper - code)] *Yekun Chai, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, and Hua Wu.* 2022.12
  - [Paper
  - [Paper - Poirier, Nouamane Tazi, Ao Tang et al.* 2024.02
- ▶️ Prompting
  - [Paper - Lezama.* 2023.06
  - [Paper - Guang Lou, Weizhu Chen.* 2022.07
  - [Paper - tau Yih, Daniel Fried, Sida I Wang.* 2022.11
  - [Paper - tau Yih, Sida I. Wang, Xi Victoria Lin.* 2023.02
  - [Paper
- ▶️ Using LLMs while coding
  - [Repo
Star History
- 🐙 Awesome Code Benchmark & Evaluation Papers
  - ![Star History Chart - history.com/#huybery/Awesome-Code-LLM&Date)
- ▶️ Using LLMs while coding
  - ![Star History Chart - history.com/#huybery/Awesome-Code-LLM&Date)
🚀 Top Code LLMs
- paper
- CodeQwen1.5-7B-Chat
- CodeQwen1.5-7B
- github
- paper
- Qwen2.5-Coder-32B-Instruct - Coder) |
- Qwen2.5-Coder-14B-Instruct - Coder) |
- Qwen2.5-Coder-7B-Instruct - Coder) |
- DS-Coder-V2-Instruct - ai/DeepSeek-Coder-V2) |
- Qwen2.5-Coder-3B-Instruct - Coder) |
- DS-Coder-V2-Lite-Instruct - ai/DeepSeek-Coder-V2) |
- DeepSeek-Coder-6.7B-Instruct - ai/DeepSeek-Coder) |
- CodeLlama-70B-Instruct
- Qwen2.5-Coder-1.5B-Instruct - Coder) |
- StarCoder2-15B-Instruct-v0.1
- Qwen2.5-Coder-0.5B-Instruct - Coder) |
- CodeLlama-34B-Instruct
- DeepSeek-Coder-33B-Instruct - ai/DeepSeek-Coder) |
- DeepSeek-Coder-6.7B-Instruct - ai/DeepSeek-Coder) |
- WizardCoder-15B

Programming Languages

Python 5 JavaScript 2 Julia 1

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

Awesome-Code-LLM

Acknowledgement

🐙 Awesome Code Benchmark & Evaluation Papers

🚀 Awesome Code LLMs Leaderboard

📚 Awesome Code LLMs Papers

🐬 Awesome Code Alignment Papers

🐙 Awesome Code Benchmark & Evaluation Papers

🐳 Awesome Code Instruction-Tuning Papers

🌊 Awesome Code Pre-Training Papers

🐋 Awesome Code Prompting Papers

💡 Evaluation Toolkit:

🚀 Leaderboard

News

📚 Paper

▶️ Alignment with Feedback

▶️ Evaluation & Benchmark

▶️ Instruction Tuning

▶️ Pre-Training

▶️ Prompting

▶️ Using LLMs while coding

Star History

🐙 Awesome Code Benchmark & Evaluation Papers

▶️ Using LLMs while coding

🚀 Top Code LLMs