awesome-code-benchmark

A comprehensive code domain benchmark review of LLM researches.
https://github.com/tongye98/awesome-code-benchmark

Last synced: 10 days ago
JSON representation

🚀 Benchmark Categories
- Code Generation & Completion
  - SR-Eval: Evaluating LLMs on Code Generation under Stepwise Requirement Refinement - `task: Code generation / completion` - `granularity: Function / Repository` - `interaction: Single-turn / Multi-turn` - `evaluation: Unit Tests / Execution`
- Code Understanding, Search & Review
  - SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback - `task: Pull request review` - `granularity: Repository / Workflow` - `interaction: Single-turn / Agentic` - `evaluation: LLM-as-Judge / Human`
  - Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice - `task: Fine-grained code review` - `granularity: File / Repository` - `interaction: Single-turn / Agentic` - `evaluation: LLM-as-Judge / Human`
  - Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA - `task: Repository-level code QA` - `granularity: Repository` - `interaction: Agentic` - `evaluation: LLM-as-Judge / Human`
  - CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding - `task: Repository-aware code retrieval` - `granularity: Function / Repository` - `interaction: Single-turn / Agentic` - `evaluation: Retrieval Metrics`
  - Beyond Retrieval: A Multitask Benchmark and Model for Code Search - `task: Code retrieval and reranking` - `granularity: Function / Repository` - `interaction: Single-turn` - `evaluation: nDCG / Retrieval Metrics`
  - CLARC: C/C++ Benchmark for Robust Code Search - `task: Robust code search` - `granularity: Function / Repository` - `interaction: Single-turn` - `evaluation: Retrieval Metrics` - [🤗Dataset](https://huggingface.co/datasets/ClarcTeam/CLARC)
  - Evaluating LLMs Code Reasoning Under Real-World Context - `task: Real-world code reasoning` - `granularity: Function / Repository` - `interaction: Single-turn` - `evaluation: Execution / Accuracy`
  - Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks - `task: Code reasoning quality evaluation` - `granularity: Function / File` - `interaction: Single-turn` - `evaluation: Human / LLM-as-Judge` - [Github](https://github.com/MrLYG/CodeRQ-Bench)
  - CodeGlance: Understanding Code Reasoning Challenges in LLMs through Multi-Dimensional Feature Analysis - `task: Code behavior reasoning` - `granularity: Function / API` - `interaction: Single-turn` - `evaluation: Accuracy / Feature Analysis`
  - Imandra CodeLogician: Neuro-Symbolic Reasoning for Precise Analysis of Software Logic - `task: Formal software logic reasoning` - `granularity: Function / Program` - `interaction: Agentic` - `evaluation: Formal Verification / Accuracy`
- Performance Optimization
  - CodegenBench: Can LLMs Write Efficient Code Across Architectures? - `task: Performance optimization` - `granularity: Function / Repository` - `interaction: Single-turn / Agentic` - `evaluation: Execution / Performance` - [Code](https://anonymous.4open.science/r/CodegenBench-EDE1/) [Dataset](https://anonymous.4open.science/r/CodegenBenchDataset-2551/)
- Program Repair, Testing & Debugging
  - Planning to Explore: Curiosity-Driven Planning for LLM Test Generation - `task: Iterative test generation` - `granularity: Repository` - `interaction: Agentic` - `evaluation: Unit Tests / Execution`
  - BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software - `task: Build and compilation repair` - `granularity: Repository / Workflow` - `interaction: Agentic` - `evaluation: Execution`
  - CI-Repair-Bench: A Repository-Aware Benchmark for Automated Patch Validation via CI Workflows - `task: CI failure diagnosis and repair` - `granularity: Repository / Workflow` - `interaction: Agentic` - `evaluation: CI / Execution`
  - ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS - `task: ArkTS automated program repair` - `granularity: Project / Repository` - `interaction: Single-turn / Agentic` - `evaluation: Unit Tests / Execution`
  - HEJ-Robust: A Robustness Benchmark for LLM-Based Automated Program Repair - `task: Robust automated program repair` - `granularity: Function` - `interaction: Single-turn` - `evaluation: Unit Tests / Execution`
  - Fixturize: Bridging the Fixture Gap in Test Generation - `task: Test fixture classification and generation` - `granularity: Function / File` - `interaction: Agentic` - `evaluation: Unit Tests / Execution`
  - RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements - `task: Requirement-based REST API test generation` - `granularity: API / Workflow` - `interaction: Single-turn / Multi-turn` - `evaluation: Mutation Testing / Execution`
  - QBugLM: An Agentic Benchmarking Framework for LLM-based Quantum Software Debugging - `task: Quantum software debugging and repair` - `granularity: File / Project` - `interaction: Multi-agent` - `evaluation: Simulation / Execution`
  - A11YRepair: Bridging Web Accessibility Barriers via Knowledge-Enhanced Divide-and-Conquer Repair - `task: Web accessibility repair` - `granularity: Project / Repository` - `interaction: Agentic` - `evaluation: Browser Tests / Execution`
  - SelfHeal: Empirical Fix Pattern Analysis and Bug Repair in LLM Agents - `task: LLM-agent bug repair` - `granularity: File / Project` - `interaction: Agentic` - `evaluation: Unit Tests / Execution`
- Repository & Agentic Software Engineering
  - ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development - `task: End-to-end project development` - `granularity: Project / Repository` - `interaction: Agentic` - `evaluation: Unit Tests / LLM-as-Judge` - [Github](https://github.com/zsworld6/projdevbench)
  - HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks - `task: Hardware repository bug repair` - `granularity: Repository` - `interaction: Agentic` - `evaluation: Execution`
  - IDE-Bench: Evaluating Large Language Models as IDE Agents on Real-World Software Engineering Tasks - `task: IDE-native software engineering` - `granularity: Repository / Workflow` - `interaction: Agentic` - `evaluation: Unit Tests / Execution`
  - RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository - `task: End-to-end microservice repository generation` - `granularity: Project / Repository` - `interaction: Agentic` - `evaluation: Unit Tests / Execution / Deployment` - [Github](https://github.com/pzy2000/RepoGenesis)
  - SWE-Refactor: A Repository-Level Benchmark for Real-World LLM-Based Code Refactoring - `task: Repository-level refactoring` - `granularity: Repository` - `interaction: Single-turn / Agentic` - `evaluation: Compilation / Unit Tests / Static Analysis`
  - SWE Context Bench: A Benchmark for Context Learning in Coding - `task: Context reuse across related coding tasks` - `granularity: Repository / Workflow` - `interaction: Multi-turn / Agentic` - `evaluation: Unit Tests / Execution / Cost`
  - SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications? - `task: Industrial mobile application development` - `granularity: Repository / Workflow` - `interaction: Multimodal / Agentic` - `evaluation: Unit Tests / Execution` - [🌐Website](https://swebenchmobile.com)
  - RepoMod-Bench: A Benchmark for Code Repository Modernization via Implementation-Agnostic Testing - `task: Repository modernization / migration` - `granularity: Project / Repository` - `interaction: Agentic` - `evaluation: Black-box Tests / Execution` - [Github](https://github.com/Modelcode-ai/mcode-benchmark)
  - SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding - `task: Agentic repository-level code understanding` - `granularity: Repository` - `interaction: Agentic` - `evaluation: QA Accuracy / Execution`
  - Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution - `task: Sequential software evolution` - `granularity: Repository / Workflow` - `interaction: Multi-turn / Agentic` - `evaluation: Unit Tests / Execution / Static Analysis`
  - RepoZero: Can LLMs Generate a Code Repository from Scratch? - `task: Repository generation from scratch` - `granularity: Project / Repository` - `interaction: Agentic` - `evaluation: Black-box Tests / Execution`
  - VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents - `task: Visual spec-to-web-app development` - `granularity: Project / Workflow` - `interaction: Multimodal / Agentic` - `evaluation: Browser Tests / Visual Similarity / DOM Matching` - [Github](https://github.com/kaboider/VIS_APP_Code) [🌐Website](https://kaboider.github.io/VIS_APP/) [🤗Dataset](https://huggingface.co/datasets/JunJiaGuo/VIS-APP-Bench)
  - Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems - `task: Production runtime assessment` - `granularity: Project / Workflow` - `interaction: Agentic` - `evaluation: Execution / Runtime / Cost` - [🌐Website](https://ramp.nexa-lang.com/)
  - Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks - `task: Agent harness evaluation for issue resolving` - `granularity: Repository` - `interaction: Agentic` - `evaluation: Unit Tests / Execution / Cost` - [Github](https://github.com/opensquilla/claw-swe-bench) [🤗Dataset](https://huggingface.co/datasets/TokenRhythm/Claw-SWE-Bench)
  - Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents - `task: Dialogue-driven issue resolving` - `granularity: Repository / Workflow` - `interaction: Multi-turn / Agentic` - `evaluation: Unit Tests / Dialogue Quality`
  - SWE-Future: Forecast-Conditioned Data Synthesis for Future-Oriented Software Engineering Agents - `task: Future-oriented repository task synthesis` - `granularity: Repository` - `interaction: Agentic` - `evaluation: Semantic Matching / Execution`
  - SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? - `task: Freelance software engineering` - `granularity: Repository / Workflow` - `interaction: Agentic` - `evaluation: Execution / Human / Economic` - [Github](https://github.com/openai/SWELancer-Benchmark)
  - Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving - `task: Multilingual issue resolving` - `granularity: Repository` - `interaction: Agentic` - `evaluation: Unit Tests / Execution`
  - SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? - `task: Long-horizon issue resolving` - `granularity: Repository / Workflow` - `interaction: Agentic` - `evaluation: Unit Tests / Execution`
  - Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation - `task: PRD-driven project development` - `granularity: Project / Repository` - `interaction: Agentic` - `evaluation: Execution / LLM-as-Judge`
  - SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories - `task: Repository-level bug fixing and feature requests` - `granularity: Repository` - `interaction: Agentic` - `evaluation: Unit Tests / Execution`
  - AInsteinBench: Benchmarking Coding Agents on Scientific Repositories - `task: Scientific repository development` - `granularity: Repository / Workflow` - `interaction: Agentic` - `evaluation: Execution`
  - SWE-Dev: Evaluating and Training Autonomous Feature-Driven Software Development - `task: Feature-driven software development` - `granularity: Repository / Workflow` - `interaction: Agentic` - `evaluation: Unit Tests / Execution` - [Github](https://github.com/DorothyDUUU/SWE-Dev)
  - SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner - `task: Test-driven incremental development` - `granularity: Repository / Workflow` - `interaction: Agentic` - `evaluation: Unit Tests / Execution` - [Github](https://github.com/Hambaobao/SWE-Flow)
  - LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering - `task: Long-context software engineering workflows` - `granularity: Repository / Workflow` - `interaction: Multi-turn / Agentic` - `evaluation: Execution / Tool-use Metrics / LLM-as-Judge`
  - Measuring Reward Hacking in Long-Horizon Coding Agents - `task: Reward hacking / spec compliance` - `granularity: Systems-level task (JSON parser → OS kernel)` - `interaction: Agentic` - `evaluation: Held-out vs visible tests` - [Github](https://github.com/WecoAI/SpecBench)
- Security, Reliability & Robustness
  - RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories - `task: Secure code generation` - `granularity: Repository` - `interaction: Single-turn / Agentic` - `evaluation: SecurePass / SAST / Execution`
  - HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generation - `task: Hardware and firmware secure code generation` - `granularity: Function / Project` - `interaction: Single-turn / Agentic` - `evaluation: Unit Tests / Security Exploit / Execution`
  - TOSSS: a CVE-based Software Security Benchmark for Large Language Models - `task: Secure-vs-vulnerable code selection` - `granularity: Function / Snippet` - `interaction: Single-turn` - `evaluation: Security Score`
  - Does Teaming-Up LLMs Improve Secure Code Generation? A Comprehensive Evaluation with Multi-LLMSecCodeEval - `task: Secure code generation and remediation pipeline evaluation` - `granularity: Function / Workflow` - `interaction: Multi-agent` - `evaluation: SAST / Security Exploit / Execution`
  - RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code - `task: Vulnerability detection in real-world code` - `granularity: Repository` - `interaction: Single-turn / Agentic` - `evaluation: Vulnerability Detection / F-score` - [🌐Website](https://realvuln.kolega.dev/)
  - Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks - `task: Code hallucination detection` - `granularity: Function / File` - `interaction: Single-turn` - `evaluation: Compilation / Execution / Human` - [Github](https://github.com/microsoft/delulu)
  - SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks? - `task: Long-horizon software security bug hunting` - `granularity: Repository / Workflow` - `interaction: Agentic` - `evaluation: PoC / Oracle / Execution`
  - RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents - `task: Agent evaluation-integrity robustness` - `granularity: Project / Workflow` - `interaction: Agentic` - `evaluation: Integrity / Runtime / Execution`
  - Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware - `task: Reliability / hallucination / robustness` - `granularity: Function / API` - `interaction: Single-turn` - `evaluation: Execution` - [Github](https://github.com/yujiachen99/APIMitigation)[![Stars](https://img.shields.io/github/stars/yujiachen99/APIMitigation?style=social&label=Stars)](https://github.com/yujiachen99/APIMitigation)
News
Survey
🚀 Top Code Benchmark
- Code Completion & Code Generation
  - Evaluating Large Language Models Trained on Code - eval) [![Stars](https://img.shields.io/github/stars/openai/human-eval?style=social&label=Stars)](https://github.com/openai/human-eval) | [🤗Dataset](https://huggingface.co/datasets/openai/openai_humaneval) |
  - Program Synthesis with Large Language Models - research/google-research/tree/master/mbpp) [![Stars](https://img.shields.io/github/stars/google-research/google-research?style=social&label=Stars)](https://github.com/google-research/google-research) | [🤗Dataset](https://huggingface.co/datasets/google-research-datasets/mbpp) |
  - Is Your Code Generated by Chat{GPT} Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
  - MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation - E) [![Stars](https://img.shields.io/github/stars/nuprl/MultiPL-E?style=social&label=Stars)](https://github.com/nuprl/MultiPL-E) | [🤗Dataset](https://huggingface.co/datasets/nuprl/MultiPL-E) |
  - Python Code Generation by Asking Clarification Questions - b-L10vNpk7Onyft9BXK8GlMIGl52q/view?usp=sharing) |
  - BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions - project/bigcodebench) [![Stars](https://img.shields.io/github/stars/bigcode-project/bigcodebench?style=social&label=Stars)](https://github.com/bigcode-project/bigcodebench)| [🤗Dataset](https://huggingface.co/collections/bigcode/bigcodebench-666ed21a5039c618e608ab06) [📊LeaderBoard](https://huggingface.co/spaces/bigcode/bigcodebench-leaderboard) |
  - DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories
  - Measuring Coding Challenge Competence With APPS
  - DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation
  - MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages - conala) [![Stars](https://img.shields.io/github/stars/zorazrw/multilingual-conala?style=social&label=Stars)](https://github.com/zorazrw/multilingual-conala) | [🤗Dataset](https://huggingface.co/datasets/neulab/mconala) |
  - LongCoder: A Long-Range Pre-trained Language Model for Code Completion
  - RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems
  - LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
  - Execution-Based Evaluation for Open-Domain Code Generation
  - BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models
  - CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion - science/cceval) [![Stars](https://img.shields.io/github/stars/amazon-science/cceval?style=social&label=Stars)](https://github.com/amazon-science/cceval) | [Dataset](https://github.com/amazon-science/cceval/tree/main/data) |
  - MT-Bench: How Good are LLMs at Multi-turn Question Answering - bench-101) | |
  - ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code - bench) [![Stars](https://img.shields.io/github/stars/gersteinlab/ML-bench?style=social&label=Stars)](https://github.com/gersteinlab/ML-bench) | [🤗Dataset](https://huggingface.co/datasets/super-dainiu/ml-bench) [🌐Website](https://ml-bench.github.io/)|
  - PLPilot: Benchmark an Automated Programming Language Design Framework Enabled by LLMs
  - CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
  - OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models - eval) [![Stars](https://img.shields.io/github/stars/alphadl/OOP-eval?style=social&label=Stars)](https://github.com/alphadl/OOP-eval) | [🤗Dataset](https://huggingface.co/datasets/codeai-dteam/oop) |
  - A Static Evaluation of Code Completion by Large Language Models
  - L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models
  - ICE-Score: Instructing Large Language Models to Evaluate Code - score) | |
  - Exploring Language Model's Code Generation Ability with Auxiliary Functions
  - R2E: Turning Any GitHub Repository into a Programming Agent Test Environment - project/r2e) | |
  - EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories
  - A Performance Study of LLM-Generated Code on Leetcode
  - Coeditor: Leveraging Contextual Changes for Multi-round Code Auto-editing
  - Competition-Level Code Generation with AlphaCode - deepmind/code_contests) [![Stars](https://img.shields.io/github/stars/google-deepmind/code_contests?style=social&label=Stars)](https://github.com/google-deepmind/code_contests) | [Dataset](https://github.com/google-deepmind/code_contests)|
  - LLM4Decompile: Decompiling Binary Code with Large Language Models - ghidra-100k)|
  - Enhancing Repository-Level Code Generation with Integrated Contextual Information
  - AICoderEval: Improving AI Domain Code Generation of Large Language Models
  - CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges
  - Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks
  - ConCodeEval: Evaluating Large Language Models for Code Constraints in Domain-Specific Languages
  - CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
  - StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code - EASEL-lab/StudentEval)[![Stars](https://img.shields.io/github/stars/Wellesley-EASEL-lab/StudentEval?style=social&label=Stars)](https://github.com/Wellesley-EASEL-lab/StudentEval) | [🤗Dataset](https://huggingface.co/datasets/wellesley-easel/StudentEval) |
  - LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs
  - A Large-scale Class-level Benchmark Dataset for Code Generation with LLMs
  - CodeMixBench: Evaluating Large Language Models on Code Generation with Code-Mixed Prompts
  - IFEvalCode: Controlled Code Generation
  - HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation - Pro/CodeEval-Pro)[![Stars](https://img.shields.io/github/stars/CodeEval-Pro/CodeEval-Pro?style=social&label=Stars)](https://github.com/CodeEval-Pro/CodeEval-Pro) | [🤗Dataset](https://huggingface.co/CodeEval-Pro) [🌐Website](https://answers111.github.io/evalpro.github.io/index.html) [📊LeaderBoard]([CodeEval-Pro](https://answers111.github.io/evalpro.github.io/leaderboard.html/)) |
- Code Efficiency
  - Evaluating Language Models for Efficient Code Generation
  - EffiBench: Benchmarking the Efficiency of Automatically Generated Code
  - Mercury: A Code Efficiency Benchmark for Code Large Language Models
  - ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?
  - Learning Performance-Improving Code Edits
  - How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark - rz/enamel) [![Stars](https://img.shields.io/github/stars/q-rz/enamel?style=social&label=Stars)](https://github.com/q-rz/enamel) | [🤗Dataset](https://huggingface.co/datasets/q-rz/enamel) |
  - Improving Assembly Code Performance with Large Language Models via Reinforcement Learning
- CodeFix & Bug-Fix
  - OctoPack: Instruction Tuning Code Large Language Models - project/octopack)[![Stars](https://img.shields.io/github/stars/bigcode-project/octopack?style=social&label=Stars)](https://github.com/bigcode-project/octopack) | [🤗Dataset](https://huggingface.co/datasets/bigcode/humanevalpack) |
  - SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents - star-ai/SWT-Bench) [![Stars](https://img.shields.io/github/stars/logic-star-ai/SWT-Bench?style=social&label=Stars)](https://github.com/logic-star-ai/SWT-Bench) | [🌐Website](https://swtbench.com) |
  - SWE-bench: Can Language Models Resolve Real-World GitHub Issues? - bench/SWE-bench) [![Stars](https://img.shields.io/github/stars/swe-bench/SWE-bench?style=social&label=Stars)](https://github.com/swe-bench/SWE-bench) | [🌐Website](https://www.swebench.com) |
  - SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? - bench/SWE-bench) [![Stars](https://img.shields.io/github/stars/swe-bench/SWE-bench?style=social&label=Stars)](https://github.com/swe-bench/SWE-bench) | [🤗Dataset](https://www.swebench.com/multimodal) [🌐Website](https://www.swebench.com/multimodal) |
  - LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
  - Large Language Models of Code Fail at Completing Code with Potential Bugs - science/buggy-code-completion)[![Stars](https://img.shields.io/github/stars/amazon-science/buggy-code-completion?style=social&label=Stars)](https://github.com/amazon-science/buggy-code-completion) | [Dataset](https://github.com/amazon-science/buggy-code-completion) |
  - GitBug-Java: A Reproducible Benchmark of Recent Java Bugs - java)[![Stars](https://img.shields.io/github/stars/gitbugactions/gitbug-java?style=social&label=Stars)](https://github.com/gitbugactions/gitbug-java) | [🤗Dataset](https://huggingface.co/datasets/gitbugactions/gitbug-java) [🌐Website](https://nuno.saavedra.pt/gitbug-java#!/) |
  - GitBug-Actions: Building Reproducible Bug-Fix Benchmarks with GitHub Actions
  - When Large Language Models Confront Repository-Level Automatic Program Repair: How Well They Done?
  - RepoFixEval: A Repository-Level Program Repair Benchmark From Issue Discovering to Bug Fixing
  - DebugBench: Evaluating Debugging Capability of Large Language Models
  - Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debugging
  - Socratic Questioning of Novice Debuggers: A Benchmark Dataset and Preliminary Evaluations - debugging-benchmark) | |
  - Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code - Gym/Project-Coffee-Gym) |
  - INTERVENOR: Prompt the Coding Ability of Large Language Models with the Interactive Chain of Repairing
  - Towards Low-Resource Automatic Program Repair with Meta-Learning and Pretrained Language Models - weishi/Meta-APR) | |
  - ZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets using LLMs
  - FeedbackEval A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks
  - COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis
  - CVE-Bench:Benchmarking LLM-based Software Engineering Agent’s Ability to Repair Real-World CVE Vulnerabilities
  - CVE-Bench:Benchmarking LLM-based Software Engineering Agent’s Ability to Repair Real-World CVE Vulnerabilities
- Code Hallucination

Programming Languages

Categories

🚀 Top Code Benchmark 202 News 66 🚀 Benchmark Categories 57 Survey 3

Sub Categories

Code Completion & Code Generation 43 Repository & Agentic Software Engineering 26 MultiModal Code Tasks 25 CodeFix & Bug-Fix 21 Text2SQL 19 Industry Code Generation 15 Multi & Other Dimension 15 Code Translation 11 Code Security & Robustness 11 Program Repair, Testing & Debugging 10 Code Understanding, Search & Review 10 Code Reasoning & Understanding 10 Security, Reliability & Robustness 9 Data science 8 Code Version 8 Code Efficiency 7 Code Hallucination 4 MultiModal Code Generation 3 Security Code Generation & Test Generation 2 Code Generation & Completion 1 Performance Optimization 1