{"id":28366919,"url":"https://github.com/aider-ai/aider-swe-bench","last_synced_at":"2025-08-19T00:10:31.501Z","repository":{"id":241820575,"uuid":"807716966","full_name":"Aider-AI/aider-swe-bench","owner":"Aider-AI","description":"Harness used to benchmark aider against SWE Bench benchmarks","archived":false,"fork":false,"pushed_at":"2024-06-27T22:44:36.000Z","size":4097,"stargazers_count":72,"open_issues_count":9,"forks_count":23,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-05-29T00:13:36.627Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Aider-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-29T16:32:03.000Z","updated_at":"2025-05-22T17:55:16.000Z","dependencies_parsed_at":"2024-10-05T19:12:31.685Z","dependency_job_id":null,"html_url":"https://github.com/Aider-AI/aider-swe-bench","commit_stats":null,"previous_names":["paul-gauthier/aider-swe-bench","aider-ai/aider-swe-bench"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Aider-AI/aider-swe-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aider-AI%2Faider-swe-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aider-AI%2Faider-swe-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aider-AI%2Faider-swe-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aider-AI%2Faider-swe-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Aider-AI","download_url":"https://codeload.github.com/Aider-AI/aider-swe-bench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aider-AI%2Faider-swe-bench/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261012852,"owners_count":23096924,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-29T00:13:30.764Z","updated_at":"2025-07-11T04:35:21.527Z","avatar_url":"https://github.com/Aider-AI.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Aider SWE Bench harness\n\n[Aider recently scored 26.3%](https://github.com/swe-bench/experiments/pull/7)\non the\n[SWE Bench Lite benchmark](https://www.swebench.com),\nachieving a state-of-the-art result. \nThis repo contains the benchmarking harness that was used to\nobtain that result.\n\n## Methodology\n\nFor the benchmark, \naider was launched in each problem's git repository\nwith the problem statement\nsubmitted as the opening chat message from \"the user.\"\nAfter that aider runs as normal, with the following modifications:\n\n- Aider's suggestions were always accepted without user approval.\n- A simple harness was used to retry the SWE Bench problem if aider produced code that wasn't *plausibly correct*.\nPlausibly correct means that aider reported that it had successfully edited the repo\nwithout causing syntax errors or breaking any *pre-existing* tests.\n- If the solution isn't plausible, the harness launches aider to try again from scratch,\nalternating between using aider with GPT-4o and Opus.\n- If no plausible solution is found after six tries, the harness picks the solution\nwith the fewest edit/lint/test problems.\n\nIt's important to be clear that\n*aider and the benchmark harness\nonly had access to the pre-existing tests in each problem's repo*.\nThe held out \"acceptance tests\" were *only* used\nafter benchmarking to compute statistics on which problems aider\ncorrectly resolved.\n\nSee the\n[article on Aider's SWE Bench Lite result](https://aider.chat/2024/05/22/swe-bench-lite.html)\nfor more details on the methodology.\n\n## The \"aider agent\"\n\nThe \"aider agent\" is dead simple.\nIt simply invokes aider on a fresh copy the problem's git repo\nover and over,\niterating through the models it's been told to use.\nAider is invoked repeatedly until aider reports that it\nsuccessfully edited the repo without any outstanding edit, lint or test errors.\nThis is a plausible solution, so the agent is done.\n\nAider is configured\nwith a test command to run all the pre-existing tests in the problem's repo.\nAider is also configured\nto proceed with all its suggestioned actions\nwithout any user approval.\n\nIn pseudo-code:\n\n```python\ndef aider_agent(swe_bench_problem):\n    num_tries = 3\n    models = [\"gpt-4o\", \"opus\"]\n    \n    for attempt in range(num_tries):\n        for model in models:\n            repo_tmp_dirname = git_checkout_the_problems_repo(swe_bench_problem)\n\n            aider_result = aider(\n                model=model,\n                repo_dirname=repo_tmp_dirname,\n                user_input_message=swe_bench_problem.problem_statement,\n                test_cmd=swe_bench_problem.test_cmd_for_preexisting_tests,\n                accept_all_suggestions=True,\n                )\n            \n            if aider_result.edit_outcome and \\\n               aider_result.lint_outcome and \\\n               aider_result.test_outcome:\n                   # We found a plausible solution!\n                   return aider_result.diffs\n```\n\nThe \n[actual function for this](https://github.com/paul-gauthier/aider-swe-bench/blob/main/harness.py#L198)\nis a bit more verbose because it's keeping\ntrack of various data for statistics, etc.\nIt also handles the case where no plausible solution is ever found,\nby picking the least bad candidate solution.\n\n## Installation\n\n```\n# Clone this repo\ngit clone https://github.com/paul-gauthier/aider-swe-bench\n\n# Clone the SWE Bench docker repo into a subdir of this repo\ncd aider-swe-bench\ngit clone https://github.com/aorwall/SWE-bench-docker\n\n# Install pip requirements\npip install -r requirements.txt\n\n# You may want to install the latest main branch of aider\npython -m pip install --upgrade git+https://github.com/paul-gauthier/aider.git\n```\n\nSee the\n[SWE Bench Docker docs](https://github.com/aorwall/SWE-bench-docker)\nto ensure you have built or pulled all the SWE Bench testbed\ndocker images you'll need.\n\n## Running the benchmark and computing results\n\nThe workflow for working with SWE Bench in general is 2 steps:\n\n1. Run your agent on the problems to produce predictions, which are a series of json records that get bundled up into a jsonl file.\n2. Evaluate the predictions jsonl file using the acceptance tests. This produces `.eval.log` files with logs of the testing procedure.\n\nThis repo is for running and evaluating aider on SWE Bench. As described in the README, it consists of 2 scripts:\n\n1. The `harness.py` script will run aider on all the problems and produce predictions. It does not do any *acceptance* testing. It does run any pre-existing tests that were part of the problem's repo, but never runs any acceptance tests. This script produces a bunch of predictions as individual json files in `predictions/\u003cDIRNAME\u003e/\u003cinstance_id\u003e.json`.\n\n2. The `report.py` script consumes all those predictions and turns them into `predictions/\u003cDIRNAME\u003e/all_preds.jsonl`. It then feeds that jsonl file through the SWE Bench evaluation and reporting scripts to produce `logs/\u003cDIRNAME\u003e/\u003cinstance_id\u003e...eval.log` files as well as a summary report in `predictions/\u003cDIRNAME\u003e/results.json`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faider-ai%2Faider-swe-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faider-ai%2Faider-swe-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faider-ai%2Faider-swe-bench/lists"}