🚀 Benchmark Categories

Code Generation & Completion

• SR-Eval: Evaluating LLMs on Code Generation under Stepwise Requirement Refinement - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution

Code Understanding, Search & Review

• SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback - task: Pull request review - granularity: Repository / Workflow - interaction: Single-turn / Agentic - evaluation: LLM-as-Judge / Human
• Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice - task: Fine-grained code review - granularity: File / Repository - interaction: Single-turn / Agentic - evaluation: LLM-as-Judge / Human
• Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA - task: Repository-level code QA - granularity: Repository - interaction: Agentic - evaluation: LLM-as-Judge / Human
• CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding - task: Repository-aware code retrieval - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Retrieval Metrics
• Beyond Retrieval: A Multitask Benchmark and Model for Code Search - task: Code retrieval and reranking - granularity: Function / Repository - interaction: Single-turn - evaluation: nDCG / Retrieval Metrics
• CLARC: C/C++ Benchmark for Robust Code Search - task: Robust code search - granularity: Function / Repository - interaction: Single-turn - evaluation: Retrieval Metrics - 🤗Dataset
• Evaluating LLMs Code Reasoning Under Real-World Context - task: Real-world code reasoning - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / Accuracy
• Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks - task: Code reasoning quality evaluation - granularity: Function / File - interaction: Single-turn - evaluation: Human / LLM-as-Judge - Github
• CodeGlance: Understanding Code Reasoning Challenges in LLMs through Multi-Dimensional Feature Analysis - task: Code behavior reasoning - granularity: Function / API - interaction: Single-turn - evaluation: Accuracy / Feature Analysis
• Imandra CodeLogician: Neuro-Symbolic Reasoning for Precise Analysis of Software Logic - task: Formal software logic reasoning - granularity: Function / Program - interaction: Agentic - evaluation: Formal Verification / Accuracy

Frontend, UI & Visual-Interactive Development

• StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs - task: Structured output generation / format conversion - granularity: File / UI - interaction: Single-turn - evaluation: Execution / Structural Validation / VQA - GitHub 🌐Website 🤗Dataset.

Performance Optimization

• CodegenBench: Can LLMs Write Efficient Code Across Architectures? - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Code Dataset

Program Repair, Testing & Debugging

• Planning to Explore: Curiosity-Driven Planning for LLM Test Generation - task: Iterative test generation - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
• BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software - task: Build and compilation repair - granularity: Repository / Workflow - interaction: Agentic - evaluation: Execution
• CI-Repair-Bench: A Repository-Aware Benchmark for Automated Patch Validation via CI Workflows - task: CI failure diagnosis and repair - granularity: Repository / Workflow - interaction: Agentic - evaluation: CI / Execution
• ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS - task: ArkTS automated program repair - granularity: Project / Repository - interaction: Single-turn / Agentic - evaluation: Unit Tests / Execution
• HEJ-Robust: A Robustness Benchmark for LLM-Based Automated Program Repair - task: Robust automated program repair - granularity: Function - interaction: Single-turn - evaluation: Unit Tests / Execution
• Fixturize: Bridging the Fixture Gap in Test Generation - task: Test fixture classification and generation - granularity: Function / File - interaction: Agentic - evaluation: Unit Tests / Execution
• RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements - task: Requirement-based REST API test generation - granularity: API / Workflow - interaction: Single-turn / Multi-turn - evaluation: Mutation Testing / Execution
• QBugLM: An Agentic Benchmarking Framework for LLM-based Quantum Software Debugging - task: Quantum software debugging and repair - granularity: File / Project - interaction: Multi-agent - evaluation: Simulation / Execution
• A11YRepair: Bridging Web Accessibility Barriers via Knowledge-Enhanced Divide-and-Conquer Repair - task: Web accessibility repair - granularity: Project / Repository - interaction: Agentic - evaluation: Browser Tests / Execution
• SelfHeal: Empirical Fix Pattern Analysis and Bug Repair in LLM Agents - task: LLM-agent bug repair - granularity: File / Project - interaction: Agentic - evaluation: Unit Tests / Execution

Repository & Agentic Software Engineering

• ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development - task: End-to-end project development - granularity: Project / Repository - interaction: Agentic - evaluation: Unit Tests / LLM-as-Judge - Github
• HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks - task: Hardware repository bug repair - granularity: Repository - interaction: Agentic - evaluation: Execution
• IDE-Bench: Evaluating Large Language Models as IDE Agents on Real-World Software Engineering Tasks - task: IDE-native software engineering - granularity: Repository / Workflow - interaction: Agentic - evaluation: Unit Tests / Execution
• RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository - task: End-to-end microservice repository generation - granularity: Project / Repository - interaction: Agentic - evaluation: Unit Tests / Execution / Deployment - Github
• SWE-Refactor: A Repository-Level Benchmark for Real-World LLM-Based Code Refactoring - task: Repository-level refactoring - granularity: Repository - interaction: Single-turn / Agentic - evaluation: Compilation / Unit Tests / Static Analysis
• SWE Context Bench: A Benchmark for Context Learning in Coding - task: Context reuse across related coding tasks - granularity: Repository / Workflow - interaction: Multi-turn / Agentic - evaluation: Unit Tests / Execution / Cost
• SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications? - task: Industrial mobile application development - granularity: Repository / Workflow - interaction: Multimodal / Agentic - evaluation: Unit Tests / Execution - 🌐Website
• RepoMod-Bench: A Benchmark for Code Repository Modernization via Implementation-Agnostic Testing - task: Repository modernization / migration - granularity: Project / Repository - interaction: Agentic - evaluation: Black-box Tests / Execution - Github
• SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding - task: Agentic repository-level code understanding - granularity: Repository - interaction: Agentic - evaluation: QA Accuracy / Execution
• Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution - task: Sequential software evolution - granularity: Repository / Workflow - interaction: Multi-turn / Agentic - evaluation: Unit Tests / Execution / Static Analysis
• RepoZero: Can LLMs Generate a Code Repository from Scratch? - task: Repository generation from scratch - granularity: Project / Repository - interaction: Agentic - evaluation: Black-box Tests / Execution
• VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents - task: Visual spec-to-web-app development - granularity: Project / Workflow - interaction: Multimodal / Agentic - evaluation: Browser Tests / Visual Similarity / DOM Matching - Github 🌐Website 🤗Dataset
• Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems - task: Production runtime assessment - granularity: Project / Workflow - interaction: Agentic - evaluation: Execution / Runtime / Cost - 🌐Website
• Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks - task: Agent harness evaluation for issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution / Cost - Github 🤗Dataset
• Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents - task: Dialogue-driven issue resolving - granularity: Repository / Workflow - interaction: Multi-turn / Agentic - evaluation: Unit Tests / Dialogue Quality
• SWE-Future: Forecast-Conditioned Data Synthesis for Future-Oriented Software Engineering Agents - task: Future-oriented repository task synthesis - granularity: Repository - interaction: Agentic - evaluation: Semantic Matching / Execution
• SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? - task: Freelance software engineering - granularity: Repository / Workflow - interaction: Agentic - evaluation: Execution / Human / Economic - Github
• Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving - task: Multilingual issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
• SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? - task: Long-horizon issue resolving - granularity: Repository / Workflow - interaction: Agentic - evaluation: Unit Tests / Execution
• Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation - task: PRD-driven project development - granularity: Project / Repository - interaction: Agentic - evaluation: Execution / LLM-as-Judge
• SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories - task: Repository-level bug fixing and feature requests - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
• AInsteinBench: Benchmarking Coding Agents on Scientific Repositories - task: Scientific repository development - granularity: Repository / Workflow - interaction: Agentic - evaluation: Execution
• SWE-Dev: Evaluating and Training Autonomous Feature-Driven Software Development - task: Feature-driven software development - granularity: Repository / Workflow - interaction: Agentic - evaluation: Unit Tests / Execution - Github
• SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner - task: Test-driven incremental development - granularity: Repository / Workflow - interaction: Agentic - evaluation: Unit Tests / Execution - Github
• LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering - task: Long-context software engineering workflows - granularity: Repository / Workflow - interaction: Multi-turn / Agentic - evaluation: Execution / Tool-use Metrics / LLM-as-Judge
• Measuring Reward Hacking in Long-Horizon Coding Agents - task: Reward hacking / spec compliance - granularity: Systems-level task (JSON parser → OS kernel) - interaction: Agentic - evaluation: Held-out vs visible tests - Github

Security, Reliability & Robustness

• RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories - task: Secure code generation - granularity: Repository - interaction: Single-turn / Agentic - evaluation: SecurePass / SAST / Execution
• HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generation - task: Hardware and firmware secure code generation - granularity: Function / Project - interaction: Single-turn / Agentic - evaluation: Unit Tests / Security Exploit / Execution
• TOSSS: a CVE-based Software Security Benchmark for Large Language Models - task: Secure-vs-vulnerable code selection - granularity: Function / Snippet - interaction: Single-turn - evaluation: Security Score
• Does Teaming-Up LLMs Improve Secure Code Generation? A Comprehensive Evaluation with Multi-LLMSecCodeEval - task: Secure code generation and remediation pipeline evaluation - granularity: Function / Workflow - interaction: Multi-agent - evaluation: SAST / Security Exploit / Execution
• RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code - task: Vulnerability detection in real-world code - granularity: Repository - interaction: Single-turn / Agentic - evaluation: Vulnerability Detection / F-score - 🌐Website
• Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks - task: Code hallucination detection - granularity: Function / File - interaction: Single-turn - evaluation: Compilation / Execution / Human - Github
• SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks? - task: Long-horizon software security bug hunting - granularity: Repository / Workflow - interaction: Agentic - evaluation: PoC / Oracle / Execution
• RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents - task: Agent evaluation-integrity robustness - granularity: Project / Workflow - interaction: Agentic - evaluation: Integrity / Runtime / Execution

News

• SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents
• aiXamine: Simplified LLM Safety and Security
• CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilation
• Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning
• CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation - ->
• Draw with Thought: Unleashing Multimodal Reasoning for Scientific Diagram Generation
• ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programming
• OSS-Bench: Benchmark Generator for Coding LLMs
• VERINA: Benchmarking Verifiable Code Generation
• ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code
• EFFIBENCH-X:A Multi-Language Benchmark fo rMeasuring Effciency ofLLM.Generated Code
• Synthesizing Performance Constraints for Evaluating and Improving Code Efficiency
• Breakpoint: Scalable evaluation of system-level reasoning in LLM code agents
• LongCodeBench: Evaluating Coding LLMs at 1M Context Windows
• Success is in the Details: Evaluate and Enhance Details Sensitivity of Code
• CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning
• Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability
• CODEMENV: Benchmarking Large Language Models on Code Migration
• DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios - ->
• VADER: A Human-Evaluated Benchmark for Vulnerability Assessment, Detection, Explanation, and Remediation
• CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming
• DS-Bench: A Realistic Benchmark for Data Science Code Generation
• Rethinking Repetition Problems of LLMs in Code Generation
• WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch
• OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue Resolution - sen University
• CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation
• Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware
• OIBench: Benchmarking Strong Reasoning Models with Olympiad in Informatics - Eval
• FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluation
• CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval
• CodeMirage: A Multi-Lingual Benchmark for Detecting AI-Generated and Paraphrased Source Code from Production-Level LLMs
• SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks - Champaign
• RAS-Eval: A Comprehensive Benchmark for Security Evaluation of LLM Agents in Real-World Environments
• MLDebugging: Towards Benchmarking Code Debugging Across Multi-Library Scenarios
• OJBench: A Competition Level Code Benchmark For Large Language Models
• TeXpert: A Multi-Level Benchmark for Evaluating LaTeX Code Generation by LLMs - ->
• DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
• PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Models
• CORE: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks
• ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation
• CoreCodeBench: A Configurable Multi-Scenario Repository-Level Benchmark
• Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs
• Model Editing for LLMs4Code: How Far are We?
• VeriBench: Benchmarking Large Language Models for Verilog Code Generation and Design Synthesis
• ResBench: Benchmarking LLM-Generated FPGA Designs with Resource Awareness
• Natural language is not enough: Benchmarking multi-modal generative AI for Verilog generation
• JsDeObsBench: Measuring and Benchmarking LLMs for JavaScript Deobfuscation
• From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking
• Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs - ->
• AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators
• Dynamic Benchmark Construction for Evaluating Large Language Models on Real-World Codes
• STEPWISE-CODEX-Bench: Evaluating Complex Multi-Function Comprehension and Fine-Grained Execution Reasoning
• SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?
• CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
• Multilingual Multimodal Software Developer for Code Generation
• CodeAssistBench (CAB): Dataset & Benchmarking for Multi-turn Chat-Based Code Assistance
• SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks
• Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security
• MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts? - Champaign
• Turning the Tide: Repository-based Code Reflection
• TRACY: Benchmarking Execution Efficiency of LLM-Based Code Translation
• BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models
• A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code
• GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging
• LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineering
• CodeFuse-CR-Bench: A Comprehensiveness-aware Benchmark for End-to-End Code Review Evaluation in Python Projects

Survey

• Software Development Life Cycle Perspective A Survey of Benchmarks for Code Large Language Models and Agents
• Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks
• A Survey on Large Language Model Benchmarks

🚀 Top Code Benchmark

Code Completion & Code Generation

• Evaluating Large Language Models Trained on Code - eval) | 🤗Dataset |
• Program Synthesis with Large Language Models - research/google-research/tree/master/mbpp) | 🤗Dataset |
• Is Your Code Generated by Chat{GPT} Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
• MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation - E) | 🤗Dataset |
• Python Code Generation by Asking Clarification Questions - b-L10vNpk7Onyft9BXK8GlMIGl52q/view?usp=sharing) |
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions - project/bigcodebench) | 🤗Dataset 📊LeaderBoard |
• DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories
• Measuring Coding Challenge Competence With APPS
• DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation
• MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages - conala) | 🤗Dataset |
• LongCoder: A Long-Range Pre-trained Language Model for Code Completion
• RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
• Execution-Based Evaluation for Open-Domain Code Generation
• BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models
• CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion - science/cceval) | Dataset |
• MT-Bench: How Good are LLMs at Multi-turn Question Answering - bench-101) | |
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code - bench) | 🤗Dataset 🌐Website|
• PLPilot: Benchmark an Automated Programming Language Design Framework Enabled by LLMs
• CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
• OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models - eval) | 🤗Dataset |
• A Static Evaluation of Code Completion by Large Language Models
• L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models
• ICE-Score: Instructing Large Language Models to Evaluate Code - score) | |
• Exploring Language Model's Code Generation Ability with Auxiliary Functions
• R2E: Turning Any GitHub Repository into a Programming Agent Test Environment - project/r2e) | |
• EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories
• A Performance Study of LLM-Generated Code on Leetcode
• Coeditor: Leveraging Contextual Changes for Multi-round Code Auto-editing
• Competition-Level Code Generation with AlphaCode - deepmind/code_contests) | Dataset|
• LLM4Decompile: Decompiling Binary Code with Large Language Models - ghidra-100k)|
• Enhancing Repository-Level Code Generation with Integrated Contextual Information
• AICoderEval: Improving AI Domain Code Generation of Large Language Models
• CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges
• Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks
• ConCodeEval: Evaluating Large Language Models for Code Constraints in Domain-Specific Languages
• CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
• StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code - EASEL-lab/StudentEval) | 🤗Dataset |
• LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs
• A Large-scale Class-level Benchmark Dataset for Code Generation with LLMs
• CodeMixBench: Evaluating Large Language Models on Code Generation with Code-Mixed Prompts
• IFEvalCode: Controlled Code Generation
• HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation - Pro/CodeEval-Pro) | 🤗Dataset 🌐Website 📊LeaderBoard |

Code Efficiency

• Evaluating Language Models for Efficient Code Generation
• EffiBench: Benchmarking the Efficiency of Automatically Generated Code
• Mercury: A Code Efficiency Benchmark for Code Large Language Models
• ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?
• Learning Performance-Improving Code Edits
• How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark - rz/enamel) | 🤗Dataset |
• Improving Assembly Code Performance with Large Language Models via Reinforcement Learning

CodeFix & Bug-Fix

• OctoPack: Instruction Tuning Code Large Language Models - project/octopack) | 🤗Dataset |
• SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents - star-ai/SWT-Bench) | 🌐Website |
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? - bench/SWE-bench) | 🌐Website |
• SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? - bench/SWE-bench) | 🤗Dataset 🌐Website |
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• Large Language Models of Code Fail at Completing Code with Potential Bugs - science/buggy-code-completion) | Dataset |
• GitBug-Java: A Reproducible Benchmark of Recent Java Bugs - java) | 🤗Dataset 🌐Website |
• GitBug-Actions: Building Reproducible Bug-Fix Benchmarks with GitHub Actions
• When Large Language Models Confront Repository-Level Automatic Program Repair: How Well They Done?
• RepoFixEval: A Repository-Level Program Repair Benchmark From Issue Discovering to Bug Fixing
• DebugBench: Evaluating Debugging Capability of Large Language Models
• Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debugging
• Socratic Questioning of Novice Debuggers: A Benchmark Dataset and Preliminary Evaluations - debugging-benchmark) | |
• Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code - Gym/Project-Coffee-Gym) |
• INTERVENOR: Prompt the Coding Ability of Large Language Models with the Interactive Chain of Repairing
• Towards Low-Resource Automatic Program Repair with Meta-Learning and Pretrained Language Models - weishi/Meta-APR) | |
• ZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets using LLMs
• FeedbackEval A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks
• COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis
• CVE-Bench:Benchmarking LLM-based Software Engineering Agent’s Ability to Repair Real-World CVE Vulnerabilities
• CVE-Bench:Benchmarking LLM-based Software Engineering Agent’s Ability to Repair Real-World CVE Vulnerabilities

Code Hallucination

• Exploring and Evaluating Hallucinations in LLM-Powered Code Generation
• CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification
• Collu-Bench: A Benchmark for Predicting Language Model Hallucinations in Code - asset/collu-bench)|