{"id":26045538,"url":"https://github.com/SWE-bench/SWE-bench","last_synced_at":"2025-03-07T20:01:10.357Z","repository":{"id":199490667,"uuid":"700117650","full_name":"SWE-bench/SWE-bench","owner":"SWE-bench","description":"SWE-bench [Multimodal]: Can Language Models Resolve Real-world Github Issues?","archived":false,"fork":false,"pushed_at":"2025-03-02T23:58:26.000Z","size":10862,"stargazers_count":2582,"open_issues_count":28,"forks_count":433,"subscribers_count":31,"default_branch":"main","last_synced_at":"2025-03-04T20:38:33.285Z","etag":null,"topics":["benchmark","language-model","software-engineering"],"latest_commit_sha":null,"homepage":"https://www.swebench.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SWE-bench.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-04T01:22:46.000Z","updated_at":"2025-03-04T10:53:43.000Z","dependencies_parsed_at":"2023-10-21T08:28:27.388Z","dependency_job_id":"a821599b-e3e2-4263-97e5-ee43cc47584c","html_url":"https://github.com/SWE-bench/SWE-bench","commit_stats":null,"previous_names":["princeton-nlp/swe-bench","swe-bench/swe-bench"],"tags_count":31,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SWE-bench%2FSWE-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SWE-bench%2FSWE-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SWE-bench%2FSWE-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SWE-bench%2FSWE-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SWE-bench","download_url":"https://codeload.github.com/SWE-bench/SWE-bench/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242454561,"owners_count":20130807,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","language-model","software-engineering"],"created_at":"2025-03-07T20:00:46.333Z","updated_at":"2025-03-07T20:01:10.350Z","avatar_url":"https://github.com/SWE-bench.png","language":"Python","funding_links":[],"categories":["Related","Python","Benchmark","Evaluation Benchmarks","Agent Harnessing and Evaluation","9. Evaluation, Benchmarks \u0026 Datasets","Coding \u0026 Repository Environments","3）参考实现与开源工具（GitHub）"],"sub_categories":["Benchmarks \u0026 Evaluation","Code","Benchmark Reality Check (real-world tool use)","评测框架与 Agent Benchmarks"],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"http://swe-bench.github.io\"\u003e\n    \u003cimg src=\"assets/figures/swellama_banner.svg\" style=\"height: 10em\" alt=\"Kawi the SWE-Llama\" /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n | [日本語](docs/README_JP.md) | [English](https://github.com/swe-bench/SWE-bench) | [中文简体](docs/README_CN.md) | [中文繁體](docs/README_TW.md) |\n\n\u003c/div\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://www.python.org/\"\u003e\n        \u003cimg alt=\"Build\" src=\"https://img.shields.io/badge/Python-3.8+-1f425f.svg?color=purple\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://copyright.princeton.edu/policy\"\u003e\n        \u003cimg alt=\"License\" src=\"https://img.shields.io/badge/License-MIT-blue\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://badge.fury.io/py/swebench\"\u003e\n        \u003cimg src=\"https://badge.fury.io/py/swebench.svg\"\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n---\n\nCode and data for the following works:\n* [ICLR 2025] \u003ca href=\"https://arxiv.org/abs/2410.03859\"\u003eSWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?\u003c/a\u003e\n* [ICLR 2024 Oral] \u003ca href=\"https://arxiv.org/abs/2310.06770\"\u003eSWE-bench: Can Language Models Resolve Real-World GitHub Issues?\u003c/a\u003e\n\n## 📰 News\n* **[Jan. 13, 2025]**: We've integrated [SWE-bench Multimodal](https://swebench.github.io/multimodal) ([paper](https://arxiv.org/abs/2410.03859), [dataset](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Multimodal)) into this repository! Unlike SWE-bench, we've kept evaluation for the test split *private*. Submit to the leaderboard using [sb-cli](https://github.com/swe-bench/sb-cli/tree/main), our new cloud-based evaluation tool.\n* **[Jan. 11, 2025]**: Thanks to [Modal](https://modal.com/), you can now run evaluations entirely on the cloud! See [here](https://github.com/swe-bench/SWE-bench/blob/main/assets/evaluation.md#%EF%B8%8F-evaluation-with-modal) for more details.\n* **[Aug. 13, 2024]**: Introducing *SWE-bench Verified*! Part 2 of our collaboration with [OpenAI Preparedness](https://openai.com/preparedness/). A subset of 500 problems that real software engineers have confirmed are solvable. Check out more in the [report](https://openai.com/index/introducing-swe-bench-verified/)!\n* **[Jun. 27, 2024]**: We have an exciting update for SWE-bench - with support from [OpenAI's Preparedness](https://openai.com/preparedness/) team: We're moving to a fully containerized evaluation harness using Docker for more reproducible evaluations! Read more in our [report](https://github.com/swe-bench/SWE-bench/blob/main/docs/20240627_docker/README.md).\n* **[Apr. 2, 2024]**: We have released [SWE-agent](https://github.com/SWE-agent/SWE-agent), which sets the state-of-the-art on the full SWE-bench test set! ([Tweet 🔗](https://twitter.com/jyangballin/status/1775114444370051582))\n* **[Jan. 16, 2024]**: SWE-bench has been accepted to ICLR 2024 as an oral presentation! ([OpenReview 🔗](https://openreview.net/forum?id=VTF8yNQM66))\n\n## 👋 Overview\nSWE-bench is a benchmark for evaluating large language models on real world software issues collected from GitHub.\nGiven a *codebase* and an *issue*, a language model is tasked with generating a *patch* that resolves the described problem.\n\n\u003cimg src=\"assets/figures/teaser.png\"\u003e\n\nTo access SWE-bench, copy and run the following code:\n```python\nfrom datasets import load_dataset\nswebench = load_dataset('princeton-nlp/SWE-bench', split='test')\n```\n\n## 🚀 Set Up\nSWE-bench uses Docker for reproducible evaluations.\nFollow the instructions in the [Docker setup guide](https://docs.docker.com/engine/install/) to install Docker on your machine.\nIf you're setting up on Linux, we recommend seeing the [post-installation steps](https://docs.docker.com/engine/install/linux-postinstall/) as well.\n\nFinally, to build SWE-bench from source, follow these steps:\n```bash\ngit clone git@github.com:princeton-nlp/SWE-bench.git\ncd SWE-bench\npip install -e .\n```\n\nTest your installation by running:\n```bash\npython -m swebench.harness.run_evaluation \\\n    --predictions_path gold \\\n    --max_workers 1 \\\n    --instance_ids sympy__sympy-20590 \\\n    --run_id validate-gold\n```\n\n## 💽 Usage\nEvaluate patch predictions on SWE-bench Lite with the following command:\n```bash\npython -m swebench.harness.run_evaluation \\\n    --dataset_name princeton-nlp/SWE-bench_Lite \\\n    --predictions_path \u003cpath_to_predictions\u003e \\\n    --max_workers \u003cnum_workers\u003e \\\n    --run_id \u003crun_id\u003e\n    # use --predictions_path 'gold' to verify the gold patches\n    # use --run_id to name the evaluation run\n```\n\nThis command will generate docker build logs (`logs/build_images`) and evaluation logs (`logs/run_evaluation`) in the current directory.\n\nThe final evaluation results will be stored in the `evaluation_results` directory.\n\n\u003e [!WARNING]\n\u003e SWE-bench evaluation can be resource intensive\n\u003e We recommend running on an `x86_64` machine with at least 120GB of free storage, 16GB of RAM, and 8 CPU cores.\n\u003e We recommend using fewer than `min(0.75 * os.cpu_count(), 24)` for `--max_workers`.\n\u003e\n\u003e If running with Docker desktop, make sure to increase your virtual disk space to ~120 free GB. Set max_workers to be consistent with the above for the CPUs available to Docker.\n\u003e\n\u003e Support for `arm64` machines is experimental.\n\nTo see the full list of arguments for the evaluation harness, run:\n```bash\npython -m swebench.harness.run_evaluation --help\n```\n\nSee the [evaluation tutorial](assets/evaluation.md) for the full rundown on datasets you can evaluate.\nIf you're looking for non-local, cloud based evaluations, check out...\n* [sb-cli](https://github.com/swe-bench/sb-cli), our tool for running evaluations automatically on AWS, or...\n* Running SWE-bench evaluation on [Modal](https://modal.com/). Details [here](https://github.com/swe-bench/SWE-bench/blob/main/assets/evaluation.md#%EF%B8%8F-evaluation-with-modal)\n\nAdditionally, you can also:\n* [Train](https://github.com/swe-bench/SWE-bench/tree/main/swebench/inference/make_datasets) your own models on our pre-processed datasets.\n* Run [inference](https://github.com/swe-bench/SWE-bench/blob/main/swebench/inference/README.md) on existing models (both local and API models). The inference step is where you give the model a repo + issue and have it generate a fix.\n*  Run SWE-bench's [data collection procedure](https://github.com/swe-bench/SWE-bench/blob/main/swebench/collect/) ([tutorial](assets/evaluation.md)) on your own repositories, to make new SWE-Bench tasks.\n\n## ⬇️ Downloads\n| Datasets | Models | RAG |\n| - | - | - |\n| [💿 SWE-bench](https://huggingface.co/datasets/princeton-nlp/SWE-bench) | [🦙 SWE-Llama 13b](https://huggingface.co/princeton-nlp/SWE-Llama-13b) | [🤗 \"Oracle\" Retrieval](https://huggingface.co/datasets/princeton-nlp/SWE-bench_oracle) |\n| [💿 SWE-bench Lite](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Lite) | [🦙 SWE-Llama 13b (PEFT)](https://huggingface.co/princeton-nlp/SWE-Llama-13b-peft) | [🤗 BM25 Retrieval 13K](https://huggingface.co/datasets/princeton-nlp/SWE-bench_bm25_13K) |\n| [💿 SWE-bench Verified](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified) | [🦙 SWE-Llama 7b](https://huggingface.co/princeton-nlp/SWE-Llama-7b) | [🤗 BM25 Retrieval 27K](https://huggingface.co/datasets/princeton-nlp/SWE-bench_bm25_27K) |\n| [💿 SWE-bench Multimodal](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Multimodal) | [🦙 SWE-Llama 7b (PEFT)](https://huggingface.co/princeton-nlp/SWE-Llama-7b-peft) | [🤗 BM25 Retrieval 40K](https://huggingface.co/datasets/princeton-nlp/SWE-bench_bm25_40K) |\n| | | [🤗 BM25 Retrieval 50K (Llama tokens)](https://huggingface.co/datasets/princeton-nlp/SWE-bench_bm25_50k_llama) |\n\n## 💫 Contributions\nWe would love to hear from the broader NLP, Machine Learning, and Software Engineering research communities, and we welcome any contributions, pull requests, or issues!\nTo do so, please either file a new pull request or issue and fill in the corresponding templates accordingly. We'll be sure to follow up shortly!\n\nContact person: [Carlos E. Jimenez](http://www.carlosejimenez.com/) and [John Yang](https://john-b-yang.github.io/) (Email: carlosej@princeton.edu, johnby@stanford.edu).\n\n## ✍️ Citation\nIf you find our work helpful, please use the following citations.\n\n```\n@inproceedings{\n    jimenez2024swebench,\n    title={{SWE}-bench: Can Language Models Resolve Real-world Github Issues?},\n    author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},\n    booktitle={The Twelfth International Conference on Learning Representations},\n    year={2024},\n    url={https://openreview.net/forum?id=VTF8yNQM66}\n}\n\n@inproceedings{\n    yang2024swebenchmultimodal,\n    title={{SWE}-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?},\n    author={John Yang and Carlos E. Jimenez and Alex L. Zhang and Kilian Lieret and Joyce Yang and Xindi Wu and Ori Press and Niklas Muennighoff and Gabriel Synnaeve and Karthik R. Narasimhan and Diyi Yang and Sida I. Wang and Ofir Press},\n    booktitle={The Thirteenth International Conference on Learning Representations},\n    year={2025},\n    url={https://openreview.net/forum?id=riTiq3i21b}\n}\n```\n\n## 🪪 License\nMIT. Check `LICENSE.md`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSWE-bench%2FSWE-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FSWE-bench%2FSWE-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSWE-bench%2FSWE-bench/lists"}