{"id":50632277,"url":"https://github.com/R2E-Gym/R2E-Gym","last_synced_at":"2026-06-23T19:00:40.985Z","repository":{"id":286796790,"uuid":"961591619","full_name":"R2E-Gym/R2E-Gym","owner":"R2E-Gym","description":"Official repository for R2E-Gym: Procedural Environment Generation and Hybrid Verifiers for Scaling Open-Weights SWE Agents","archived":false,"fork":false,"pushed_at":"2025-07-13T04:53:01.000Z","size":2199,"stargazers_count":113,"open_issues_count":0,"forks_count":9,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-13T06:34:12.115Z","etag":null,"topics":["agents","coding-agents","llm","llm-code","swe"],"latest_commit_sha":null,"homepage":"https://r2e-gym.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/R2E-Gym.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-06T20:28:55.000Z","updated_at":"2025-07-13T05:09:51.000Z","dependencies_parsed_at":"2025-07-13T06:33:59.152Z","dependency_job_id":null,"html_url":"https://github.com/R2E-Gym/R2E-Gym","commit_stats":null,"previous_names":["r2e-gym/r2e-gym"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/R2E-Gym/R2E-Gym","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/R2E-Gym%2FR2E-Gym","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/R2E-Gym%2FR2E-Gym/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/R2E-Gym%2FR2E-Gym/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/R2E-Gym%2FR2E-Gym/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/R2E-Gym","download_url":"https://codeload.github.com/R2E-Gym/R2E-Gym/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/R2E-Gym%2FR2E-Gym/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34702919,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-23T02:00:07.161Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agents","coding-agents","llm","llm-code","swe"],"created_at":"2026-06-06T23:00:22.904Z","updated_at":"2026-06-23T19:00:40.979Z","avatar_url":"https://github.com/R2E-Gym.png","language":"Python","funding_links":[],"categories":["Coding \u0026 Repository Environments"],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e R2E-Gym: Procedural Environment Generation and Hybrid Verifiers for Scaling Open-Weights SWE Agents \u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://naman-ntc.github.io/\" style=\"text-decoration: none;\"\u003eNaman Jain\u003csup\u003e*,1\u003c/sup\u003e\u003c/a\u003e, \n  \u003ca href=\"https://1jsingh.github.io/\" style=\"text-decoration: none;\"\u003eJaskirat Singh\u003csup\u003e*,2\u003c/sup\u003e\u003c/a\u003e,\n  \u003ca href=\"https://manishs.org/\" style=\"text-decoration: none;\"\u003eManish Shetty\u003csup\u003e1\u003c/sup\u003e\u003c/a\u003e,\n  \u003ca href=\"https://scholar.google.com/citations?user=vNHqr3oAAAAJ\u0026hl=en\" style=\"text-decoration: none;\"\u003eLiang Zheng\u003csup\u003e2\u003c/sup\u003e\u003c/a\u003e,\n  \u003ca href=\"https://scholar.google.com/citations?user=Vn3L_ioAAAAJ\u0026hl=en\" style=\"text-decoration: none;\"\u003eKoushik Sen\u003csup\u003e1\u003c/sup\u003e\u003c/a\u003e,\n  \u003ca href=\"https://scholar.google.com/citations?user=vN-is70AAAAJ\u0026hl=en\" style=\"text-decoration: none;\"\u003eIon Stoica\u003csup\u003e1\u003c/sup\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003csup\u003e1\u003c/sup\u003eUC Berkeley, \u003csup\u003e2\u003c/sup\u003eANU \u003c/br\u003e\n  \u003csub\u003e\u003csup\u003e*\u003c/sup\u003eEqual contribution, \u003csup\u003e^\u003c/sup\u003eEqual supervision\u003c/sub\u003e\n\u003c/p\u003e\n\n\u003c!-- paper . data and models . project page --\u003e\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://arxiv.org/abs/2504.07164\"\u003e📃 Paper\u003c/a\u003e\n•\n\u003ca href=\"https://huggingface.co/R2E-Gym\" \u003e🤗 Data \u0026 Models\u003c/a\u003e\n•\n\u003c!-- project page --\u003e\n\u003ca href=\"https://r2e-gym.github.io/\" \u003e🌐 Project Page\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n## 🚨 UPDATES\n\n🔥 **NEW: DeepSWE Models Available!** We've released **DeepSWE**, our latest state-of-the-art SWE agent models that achieve exceptional performance on SWE-Bench trained with [**rLLM**](https://github.com/agentica-project/rLLM). \n\n- 🤗 **Model**: [agentica-org/DeepSWE-Preview](https://huggingface.co/agentica-org/DeepSWE-Preview) \n- 📋 **Reproduction Guides**: Check out our detailed reproduction guides in the [`reproduction/`](./reproduction/) folder:\n  - [`DEEPSWE_REPRODUCTION.MD`](./reproduction/DEEPSWE_REPRODUCTION.MD) - Complete guide for reproducing DeepSWE results\n  - [`DEEPSWE_TTS_REPRODUCTION.MD`](./reproduction/DEEPSWE_TTS_REPRODUCTION.MD) - Test-time scaling reproduction guide\n\n---\n\n\n\nWe present **R2E-Gym**, the largest procedurally curated environment for training real-world SWE-Agents.\nWe show that R2E-Gym enables more scalable train and test-time scaling, achieving **51% on the SWE-Bench Verified benchmark**, reflecting a new state-of-the-art for open-weight SWE-Agents and for first time being competitive with proprietary models such as o1 and sonnet-3.5-v2 with tools.\n\n![!R2E-Gym Environment](./assets/docs-teaser-v1.png)\n\u003cp align=\"left\"\u003e\n    \u003csmall\u003eR2E-Gym is powered by two main contributions: (a) SWE-GEN: a synthetic data curation recipe for curating executable training environments w/o relying on human tests and issues. (b) Hybrid Inference Time Scaling: showing that while both execution-based and execution-free verifiers elicit inference-time gains; significantly better performance can be achieved by leveraging the strengths of both. (c) Overall, the final approach reflects SOTA performance for open-weight SWE-Agents, while also being competitive with some proprietary model baselines.\u003c/small\u003e\n\u003c/p\u003e\n\n---\n\u003c!-- \u003e[!] --\u003e\n\u003e  While LLM-based SWE-Agents have demonstrated remarkable improvements, state-of-the-art performance is largely driven by proprietary models — with open-models lagging behind. Closing this performance gap requires addressing two core challenges: First, we need scalable methods to curate diverse, high-quality execution environments for training. Second, we need efficient strategies for scaling test-time compute. R2EGym presents a joint framework for address both these challenges. \n\n## R2E-Gym Environment\n\nWe create R2E-Gym, the largest procedurally curated gym environment for training real-world SWE-Agents,  — consisting of more than **8.1K problems across 13 repos**, with executable gym environments, unit tests, and natural-language task descriptions.\n\n\u003c!-- add env stats --\u003e\n\u003c!-- ![!R2E-Gym Environment Statistics](./assets/docs-env-stats-v1.png) --\u003e\n\u003c!-- \u003cimg src=\"./assets/docs-env-stats-v1.png\" alt=\"R2E-Gym Environment Statistics\" width=\"80%\" align=\"center\"\u003e --\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./assets/docs-env-stats-v1.png\" alt=\"R2E-Gym Environment Statistics\" width=\"80%\"\u003e\n\u003c/div\u003e\n\n## Synthetic Data Enables Scalable Agent Training\n\nR2E-Gym is powered by **SWE-GEN — a novel synthetic data curation recipe** that enables collection of a large number of executable training environments without reliance on human-written pull requests (PRs) or unit tests. We show that instead of using human-written PRs, good-quality execution environments can directly be curated from **commits**. Compared to PR-based data collection, we find that this approach enables more scalable data curation and agent-training, resulting in a SOTA pass@1 performance of 34.4% on the challenging SWE-Bench Verified benchmark.\n\n\u003c!-- ![!Synthetic Data Enables Scalable Training](./assets/docs-training-v1.png) --\u003e\n\u003c!-- \u003cimg src=\"./assets/docs-training-v1.png\" alt=\"Synthetic Data Enables Scalable Training\" width=\"80%\" align=\"center\"\u003e --\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./assets/docs-training-v1.png\" alt=\"Synthetic Data Enables Scalable Training\" width=\"80%\"\u003e\n\u003c/div\u003e\n\n## Hybrid Test-time Scaling\n\nFinally, we introduce **Hybrid Test-time Scaling**, a novel paradigm for scaling test-time compute. We show that while both execution-based and execution-free verifiers elicit inference-time gains; they exchit complementary strengths and weakness. Leveraging the strengths of each approach allows significantly better performance when scaling test-time compute - resulting in a **51% pass@1 performance on the SWE-Bench Verified benchmark**, reflecting a new **state-of-the-art for open-weight SWE-Agents**.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./assets/bestk_plot_agent_nopass.png\" alt=\"Hybrid Test-time Scaling\" width=\"60%\"\u003e\n\u003c/div\u003e\n\n---\n\n## 🔧 Setup\n\u003e [!IMPORTANT]\n\u003e Installation is required!\n\n```bash\n## Install uv\ncurl -LsSf https://astral.sh/uv/install.sh | sh\nsource $HOME/.local/bin/env\n\n# activate venv\nuv venv\nsource .venv/bin/activate\nuv sync \u0026\u0026 uv pip install -e .\n```\n\n## 🚀 Quickstart\n* **Usage**: R2E-Gym environment can be simply used as:\n```python\nfrom r2egym.agenthub.environment.env import EnvArgs, RepoEnv\nfrom r2egym.agenthub.agent.agent import AgentArgs, Agent\nfrom pathlib import Path\nfrom datasets import load_dataset\n\n# load gym dataset [R2E-Gym/R2E-Gym-Subset, R2E-Gym/R2E-Gym-Full, R2E-Gym/SWE-Bench-Verified, R2E-Gym/SWE-Bench-Lite]\nds = load_dataset(\"R2E-Gym/R2E-Gym-Lite\")\nsplit = 'train' # split of the dataset [train, test]\n\n# load gym environment\nenv_index = 100 # index of the environment [0, len(ds)]\nenv_args = EnvArgs(ds = ds[split][env_index])\nenv = RepoEnv(env_args)\n\n# load agent\nagent_args = AgentArgs.from_yaml(Path('./src/r2egym/agenthub/config/edit_fn_calling.yaml'))\n# define llm: ['claude-3-5-sonnet-20241022', 'gpt-4o', 'vllm/R2E-Gym/R2EGym-32B-Agent']\nagent_args.llm_name = 'claude-3-5-sonnet-20241022'\nagent = Agent(name=\"EditingAgent\", args=agent_args)\n\n# run the agent (note: disable fn_calling for R2E-Gym agents)\noutput = agent.run(env, max_steps=40, use_fn_calling=True)\n```\n\n\u003e [!NOTE]\n\u003e The output of the agent is a `Trajectory` object, which contains detailed stats including full agent trajectory, problem statement, max execution time, exit-reason, and output patch. Please refer `src/r2egym/agenthub/agent/agent.py` and `src/r2egym/agenthub/trajectory/trajectory.py` for more details.\n\n* **Reward Calculation:** All R2E-Gym environments support automated reward calculation using unit tests.\n```python\n# calculate reward\nout = env.runtime._calculate_reward()\n```\n\n* **Gym Environment Stats**: The detailed stats for each environment (including natural language task description, repo name, ground truth patch) can be easily accessed as,\n```python\n# get the environment stats\nenv_stats_dict = env.get_stats()\n```\n\n\u003e [!TIP]\n\u003e R2EGym environments also offer a range of other convenient functions, such as `apply_patch`, `get_gt_commit`, `reverse_patch` etc. Please refer `src/r2egym/agenthub/environment/env.py` and `src/r2egym/agenthub/runtime/runtime.py` for more details.\n\n\n* **Trajectory Visualization**: We also provide flask app `apps/app.py` for visualizing the generated trajectories.\n```python\n# run the flask app\nuv run apps/app.py\n```\n## 🔥 Training Open-Weight SWE-Agents\n\n### Data Collection and Inference\n\n* **Trajectory Collection**: R2E-Gym supports parallelized inference for training trajectory collection or evaluation. For instance to collect SFT trajectories on first 2000 R2E-Gym-Lite environments, run the following command:\n```python\nuv run python src/r2egym/agenthub/run/edit.py runagent_multiple \\\n  --traj_dir \"./traj\" \\\n  --max_workers 54 \\\n  --start_idx 0 \\\n  --k 2000 \\\n  --dataset \"R2E-Gym/R2E-Gym-Lite\" \\\n  --split \"train\" \\\n  --llm_name 'gpt-4o' \\\n  --use_fn_calling True \\\n  --exp_name r2egym-training-trajectories \\\n  --temperature 0.2 \\\n  --max_steps 40\n```\n\u003e [!IMPORTANT]\n\u003e Please adjust the number of gym environments to be collected using `--k` argument. The above command will collect 2000 trajectories in parallel using 54 workers. Each executable gym instance has a docker image ~300MB-500Mb, so please ensure you have enough disk space.\n\nSimilarly, evaluation on SWE-Bench-Verified can be done using:\n```bash\nuv run python src/r2egym/agenthub/run/edit.py runagent_multiple \\\n  --traj_dir \"./traj\" \\\n  --max_workers 54 \\\n  --start_idx 0 \\\n  --k 500 \\\n  --dataset \"R2E-Gym/SWE-Bench-Verified\" \\\n  --split \"test\" \\\n  --llm_name 'vllm/R2E-Gym/R2EGym-32B-Agent' \\\n  --use_fn_calling False \\\n  --exp_name r2egym-32B-editingagent-swebv-eval \\\n  --temperature 0 \\\n  --max_steps 40\n```\n\n\u003e [!NOTE]\n\u003e The above evaluation command will only generate the output trajectories and patches. Please use the official [SWE-Bench evaluation harness](https://github.com/SWE-bench/SWE-bench) script for final evaluation scores.\n\n### 💻 Training\n\nFor ease of use, we provide precollected SFT trajectories using **claude-3-5-sonnet-20241022** for training different SWE-Agents and Verifiers, including:\n1. **Code Editing Agent**: a general-purpose SWE-Agent for SWE tasks\n2. **Execution-based Testing Agent**: a specialized testing agent for generating targeted unit tests.\n3. **Execution-free Verifier Agent**: for reranking the generated patches in a training-free manner.\n\n\u003c!-- table below for two hf datasets one fro editing agent and other for testing agent --\u003e\n| Dataset                                                                                                                        | Description                                        |\n| ------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------- |\n| 🤗 [R2E-Gym/R2EGym-SFT-Trajectories](https://huggingface.co/datasets/R2E-Gym/R2EGym-SFT-Trajectories)                           | SFT Trajectories for Editing Agent                 |\n| 🤗 [R2E-Gym/R2EGym-TestingAgent-SFT-Trajectories](https://huggingface.co/datasets/R2E-Gym/R2EGym-TestingAgent-SFT-Trajectories) | SFT Trajectories for Execution-Based Testing Agent |\n| 🤗 [R2E-Gym/R2EGym-Verifier-Trajectories](https://huggingface.co/datasets/R2E-Gym/R2EGym-Verifier-Trajectories)                 | SFT Trajectories for Execution-Free Verifier Agent |\n\n\u003e [!TIP]\n\u003e **Agent Training**: We provide easy to use config files for training your own agents using [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) on R2E-Gym trajectories.\n\nFirst clone and install [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory)\n```bash\ngit clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git\ncd LLaMA-Factory\npip install -e \".[torch,metrics]\"\n```\n\n\u003e [!TIP]\n\u003e Optional dependencies for faster training: flashattention2 (fa2), deepspeed, liger-kernel, unsloth\n\n\n\u003e [!NOTE]\n\u003e Please update `data/dataset_info.json` to use your custom dataset. For instance, we provide a reference `train/dataset_info.json` for different R2EGym datasets.\n\n\nTrain your own agents using provided config files in `train` folder. For instance, to train the general-purpose prompting SWE agent,\n```bash\nllamafactory-cli train train/train_r2egym_32B_agent.yaml\n```\n\nSimilarly to train a specialized execution-based testing agent, run:\n```bash\nllamafactory-cli train train/train_r2egym_32B_testing_agent.yaml\n```\n\n## 🧪 Executable SWE Environment Generation\nPlease refer [ENV_GENERATION.md](docs/ENV_GENERATION.md) for details on SWE-GEN based environment generation.\n\n\n## 📚 Citation\n\n```bibtex\n@article{jain2025r2e,\n  title={R2e-gym: Procedural environments and hybrid verifiers for scaling open-weights swe agents},\n  author={Jain, Naman and Singh, Jaskirat and Shetty, Manish and Zheng, Liang and Sen, Koushik and Stoica, Ion},\n  journal={arXiv preprint arXiv:2504.07164},\n  year={2025}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FR2E-Gym%2FR2E-Gym","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FR2E-Gym%2FR2E-Gym","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FR2E-Gym%2FR2E-Gym/lists"}