https://github.com/open-compass/saga
https://github.com/open-compass/saga
Last synced: 6 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/open-compass/saga
- Owner: open-compass
- Created: 2025-07-04T08:07:46.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-07-10T02:03:44.000Z (about 1 year ago)
- Last Synced: 2025-07-10T11:17:30.905Z (about 1 year ago)
- Size: 3.19 MB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
# SAGA: Strategic Adversarial & Constraint-differential Generative workflow for Test Case Generation
[](https://arxiv.org/abs/2507.06920)
[](https://huggingface.co/datasets/opencompass/CodeCompass)
[](https://huggingface.co/datasets/opencompass/CodeForce_SAGA)
This repository is the official implementation and supplementary material for the research paper, **"Rethinking Verification for LLM Code Generation: From Generation to Testing."** This work is a proud contribution from the [OpenCompass](https://github.com/open-compass/opencompass) team to advance the robust evaluation of large language models.
## 🚀 Introduction to SAGA
Current benchmarks for evaluating Large Language Model code generation often rely on a limited or homogeneous set of test cases. This can lead to inflated performance metrics and hinder the development of truly robust and reliable models.
To address these critical shortcomings, we introduce **SAGA (Strategic Adversarial & Constraint-differential Generative workflow)**, a novel human-LLM collaborative framework for advanced Test Case Generation (TCG). SAGA systematically integrates deep human programming expertise with the reasoning capabilities of LLMs to produce high-quality, diverse, and discriminative test suites. It achieves this through a dual-pronged analytical approach:
* **Multidimensional Analysis:** Leveraging profound insights from correct human solutions to engineer challenging test scenarios.
* **Differential Analysis:** Identifying subtle error patterns by analyzing incorrect human submissions against their corrected versions.
The core mission of SAGA is to significantly enhance the verification of LLM-generated code by maximizing both individual test case potency and overall test suite diversity.
## 🌟 Key Assets
This project provides several key assets to support full reproducibility and further research.
### 📦 Datasets on Hugging Face
We release two major datasets built with the SAGA framework:
#### 🔹 CodeCompass Benchmark
- **URL:** [https://huggingface.co/datasets/opencompass/CodeCompass](https://huggingface.co/datasets/opencompass/CodeCompass)
- A high-quality **evaluation benchmark** containing rigorous and diverse verifiers for all problems in TCGBench-Lite, designed for evaluating LLM code generation systems.
#### 🔸 CodeForce_SAGA Training Set
- **URL:** [https://huggingface.co/datasets/opencompass/CodeForce_SAGA](https://huggingface.co/datasets/opencompass/CodeForce_SAGA)
- A large-scale **training dataset** constructed using the SAGA framework. It is built from competitive programming problems and enriched via SAGA to produce fine-grained test cases for training code intelligence models.
### 📁 Local Assets in this Repository
* **TCGBench-Lite Problem Set:**
- Problem descriptions for the 270 problems used in our experiments.
- Located at: `data/tcgbenc_lite_problems.jsonl` (demo included)
* **SAGA Prompt Templates:**
- Prompt templates for both Multidimensional and Differential Analysis.
- Located at: `prompts/`
* **SAGA-Generated Test Case Demo:**
- A demo Python script showcasing a generated test case.
- Located at: `demos/parse.py`
## 🛠️ Integration with OpenCompass (Coming Soon)
The **CodeCompass Benchmark** will soon be integrated into the [**OpenCompass**](https://github.com/open-compass/opencompass) evaluation ecosystem, enabling plug-and-play benchmarking of LLMs in code generation tasks.
## 📚 Citation
If you find our work useful, please consider citing:
```bibtex
@inproceedings{
ma2025rethinking,
title={Rethinking Verification for {LLM} Code Generation: From Generation to Testing},
author={Zihan Ma and Taolin Zhang and Maosongcao and Junnan Liu and Wenwei Zhang and Minnan Luo and Songyang Zhang and Kai Chen},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=Gp2vgxWROE}
}
```