{"id":51598986,"url":"https://github.com/e2b-dev/swe-bench","last_synced_at":"2026-07-11T20:31:01.453Z","repository":{"id":370596527,"uuid":"1262581556","full_name":"e2b-dev/swe-bench","owner":"e2b-dev","description":"A harness that evaluates SWE-bench instances inside E2B sandboxes","archived":false,"fork":false,"pushed_at":"2026-07-09T17:37:18.000Z","size":30,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-09T19:15:20.694Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/e2b-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-08T05:57:07.000Z","updated_at":"2026-07-09T17:38:12.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/e2b-dev/swe-bench","commit_stats":null,"previous_names":["e2b-dev/swe-bench"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/e2b-dev/swe-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/e2b-dev%2Fswe-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/e2b-dev%2Fswe-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/e2b-dev%2Fswe-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/e2b-dev%2Fswe-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/e2b-dev","download_url":"https://codeload.github.com/e2b-dev/swe-bench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/e2b-dev%2Fswe-bench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35375152,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-11T02:00:05.354Z","response_time":104,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-11T20:30:56.169Z","updated_at":"2026-07-11T20:31:01.447Z","avatar_url":"https://github.com/e2b-dev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SWE-bench on E2B\n\nRun the [SWE-bench](https://github.com/swe-bench/SWE-bench) benchmark on\n[E2B](https://e2b.dev/?utm_source=github\u0026utm_medium=referral\u0026utm_campaign=readme\u0026utm_content=swe-bench) sandboxes.\n\n---\n\n\nA harness that evaluates SWE-bench instances inside **E2B sandboxes** instead of\nlocal Docker. SWE-bench's grading logic is pure Python (no Docker dependency), so\nthis project reuses all of it and only swaps the *execution environment*: where\nthe official harness runs each task in a Docker container, this runs it in an\nE2B sandbox.\n\nPer instance it: places the repo at the right commit, applies a candidate patch\n(a model's fix — or the **gold** fix, for validation), runs the held-out tests,\nand computes `resolved` with the canonical `swebench` grader.\n\nIt ships:\n- a one-shot **smoke test**,\n- a **batched, resumable** build-and-verify pipeline for the whole dataset, backed by a persistent ledger,\n- plain **build** and **eval** scripts,\n- an importable Python package (`e2b_swebench`).\n\n---\n\n## 2. Architecture\n\n**One E2B template per instance.**\n\nSWE-bench publishes a prebuilt Docker image *per instance* on Docker Hub:\n`swebench/sweb.eval.x86_64.\u003cinstance_id\u003e` (with `__` → `_1776_`). Each image\nalready has the repo checked out at `base_commit` in `/testbed` and a conda env\n`testbed` with the project installed.\n\nWe build **one E2B template per instance, `FROM` that image**. A sandbox then\nspawns ready-to-eval, with everything on the sandbox's fast **local disk** — no\nper-run clone or install. Builds run server-side on E2B (**no local Docker**).\n\n```\nSWE-bench image (Docker Hub)        E2B template (per instance)      E2B sandbox (per run)\nsweb.eval.x86_64.\u003cid\u003e        ──►     swebench-\u003cid\u003e            ──►      /testbed @ base_commit\n  /testbed @ base_commit             (FROM the image)                 conda env \"testbed\" ready\n  conda env \"testbed\"\n```\n\nThe per-run flow maps the official Docker harness 1:1 onto E2B:\n\n| official harness (Docker)     | this project (E2B)                              |\n| ----------------------------- | ----------------------------------------------- |\n| build_container               | `Sandbox.create(template)`                      |\n| copy patch into container     | `sandbox.files.write(...)`                      |\n| exec `git apply`              | `sandbox.commands.run(\"git apply ...\")`         |\n| exec `/eval.sh`               | `sandbox.commands.run(\"/bin/bash /eval.sh ...\")`|\n| parse logs → resolved         | `swebench.harness.grading.get_eval_report` (unchanged) |\n\n**Grading is canonical.** Combined stdout+stderr is captured and fed to\n`get_eval_report`, which selects the correct per-repo log parser and applies the\nrule: **`resolved` iff every `FAIL_TO_PASS` test passes AND every `PASS_TO_PASS`\ntest still passes.**\n\n**Three things worth knowing:**\n\n- The SWE-bench image adds a `\"SWE-bench\"` commit on top of `base_commit` (it\n  tweaks a build file), so `/testbed` HEAD ≠ `base_commit` — `base_commit` is its\n  *ancestor*. A real agent should capture its work as `git -C /testbed diff`\n  (against HEAD), not reset to `base_commit`.\n- Some instances run the *entire* repo test-suite, whose order depends on\n  filesystem `scandir` order. On a non-Docker filesystem this can differ from\n  Docker and cause a few benign `PASS_TO_PASS` failures (test pollution) — **not**\n  a build defect. The pipeline flags these as **`ordering_artifact`** (gold fix\n  applies, all `FAIL_TO_PASS` pass, only `PASS_TO_PASS` regress) so they're\n  distinguishable from real failures.\n- The published **`:latest` images drift** — they now ship newer base packages (e.g.\n  **setuptools 68**, newer **pytest**) than these strict old test suites tolerate, and\n  astropy's *programmatic* warnings-as-errors promotes the resulting deprecation\n  warnings to failures. Two manifestations: setuptools' distutils `DeprecationWarning`\n  breaks **collection** (the module won't import → `collection_error`, e.g.\n  astropy-8872), and pytest's nose-`setup()` deprecation breaks tests at **run time**\n  (→ `warning_error`, e.g. astropy-8707). This hits the **official Docker harness too**\n  (it's the image's env, not the runtime) and **can't be cleanly suppressed** by env/CLI\n  warning filters. So we **don't silently hack around it** — the pipeline classifies\n  these distinctly and reports them, kept separate from genuine failures. (Recovering\n  them means pinning older image deps, which diverges from the published image; out of scope.)\n\n---\n\n## 3. Setup\n\n```bash\npython3 -m venv .venv \u0026\u0026 source .venv/bin/activate\npip install -r requirements.txt\npip install -e .                 # makes the `e2b_swebench` package importable from scripts\nexport E2B_API_KEY=...           # required (or put it in a .env file)\n# export HF_TOKEN=...            # optional: faster dataset downloads\n```\n\nValidated with `e2b 2.26`, `swebench 4.1`, `datasets 5.0`. Dataset defaults to\n**SWE-bench_Verified** (500 instances, full x86_64 image coverage).\n\n### Choosing a dataset\n\nPick the dataset with `SWEBENCH_DATASET` (and `SWEBENCH_SPLIT`, default `test`):\n\n```bash\nSWEBENCH_DATASET=princeton-nlp/SWE-bench_Lite \\\n  python scripts/build_and_verify.py --all --batch-size 25 --stop-on-fail\n```\n\n| variant | HuggingFace id | test size | when to use | works with this project? |\n| --- | --- | --- | --- | --- |\n| **Verified** *(default)* | `princeton-nlp/SWE-bench_Verified` | 500 | human-validated, non-flaky — the standard leaderboard set | ✅ 100% prebuilt x86_64 images |\n| **Lite** | `princeton-nlp/SWE-bench_Lite` | 300 | cheap/fast iteration; a curated easier subset | ✅ covered (subset of Full) |\n| **Full** | `princeton-nlp/SWE-bench` | 2,294 | the complete set — comprehensive but slower \u0026 noisier | ✅ ~99.8% image coverage (a few unbuilt) |\n| **Multimodal** | `SWE-bench/SWE-bench_Multimodal` | 500 | JS/visual issues with screenshots | ❌ cloud-only (`sb-cli`); private test split, no public images — different harness |\n| **Multilingual** | `SWE-bench/SWE-bench_Multilingual` | 300 | 9 programming languages / 42 repos | ⚠️ newer \u0026 non-Python; `swebench` can grade it, but **verify prebuilt-image coverage** before a full run |\n\n**Recommendation:** start with **Verified** (the default). Drop to **Lite** for a\ncheaper smoke; use **Full** when you need everything. **Multimodal** requires the\ncloud `sb-cli` path (out of scope for this per-instance-template project), and\n**Multilingual** is newer/experimental here.\n\nNotes:\n- Both HuggingFace orgs work for the core sets — `princeton-nlp/…` (legacy, the\n  default here) and `SWE-bench/…` (the newer canonical org, which also hosts the\n  newest variants like Multilingual).\n- All variants evaluate on the `test` split; some also ship a smaller `dev` split.\n- *\"SWE-bench Pro\"* is a separate third-party benchmark — not part of this family\n  and not loadable through this project.\n\n---\n\n## 4. How to run\n\nRun from the repo root with the venv active and `E2B_API_KEY` set. Use\n`python -u` for live progress on long background runs.\n\n### a) Smoke test — start here (1 instance, ~3 min)\n\nBuilds one template, checks the environment, runs the **gold** patch (must\nresolve) and an **empty** patch (must not).\n\n```bash\npython scripts/smoke_test.py                      # default: astropy__astropy-12907\npython scripts/smoke_test.py sympy__sympy-20438   # a specific instance\n```\n\n### b) Build + verify the whole dataset, in resumable batches — the main path\n\nBuilds templates in batches and gold-verifies each batch, recording every\noutcome in a ledger you can **stop and resume across sessions**.\n\n```bash\n# everything, batches of 25 — halt on any non-clean batch so you can investigate\npython scripts/build_and_verify.py --all --batch-size 25 --stop-on-fail\n\n# only a few batches now (e.g. tonight), continue later\npython scripts/build_and_verify.py --all --batch-size 25 --max-batches 4\n\n# check progress any time (reads the ledger, doesn't touch a running job)\npython scripts/build_and_verify.py --status\n\n# a quick cross-repo sample instead of the full set\npython scripts/build_and_verify.py --per-repo 2 --batch-size 12\n```\n\n| option | default | meaning |\n| --- | --- | --- |\n| `--all` / `--per-repo N` / `--limit N` / `--instances a,b` | — | which instances (choose one) |\n| `--batch-size` | 25 | instances per batch (build + verify, then ledger save) |\n| `--workers` | 8 | parallel template builds (processes) |\n| `--verify-concurrency` | 20 | concurrent verify sandboxes (E2B free-tier cap; raise on paid tiers) |\n| `--max-batches` | — | stop after N batches this session |\n| `--cpu` / `--memory-mb` | 8 / 16384 | template resources |\n| `--stop-on-fail` | — | halt after any batch that isn't all `pass`/`ordering_artifact`, to investigate before continuing |\n| `--ledger` | `results/ledger.json` | progress file |\n| `--status` | — | print ledger summary and exit |\n\nLedger verify categories: **`pass`** (resolved), **`ordering_artifact`** (template\ngood; benign test-ordering), **`collection_error`** / **`warning_error`** (template\ngood; upstream image drift at import- / test-time — see §2), **`fail`** (genuine\nfailure to investigate), **`error`** (transient → retried on resume). With\n`--stop-on-fail` the run halts **only on a genuine `fail`**; pass / ordering_artifact /\ncollection_error / warning_error are tallied and skipped on resume.\n\n### c) Build templates only\n\n```bash\npython scripts/build_templates.py --per-repo 1          # one per repo (~12)\npython scripts/build_templates.py --limit 20\npython scripts/build_templates.py --instances astropy__astropy-12907,sympy__sympy-20438\npython scripts/build_templates.py --all --workers 8     # all 500\n```\n\nOptions: `--instances` / `--per-repo N` / `--limit N` / `--all`, `--workers`\n(default 4), `--cpu`, `--memory-mb`, `--force` (rebuild even if it exists).\n\n### d) Evaluate — gold sanity, or a real model's predictions\n\n```bash\n# gold across a cross-repo sample, building any missing templates first\npython scripts/run_eval.py --gold --per-repo 1 --build\n\n# gold across all 500 (--concurrency defaults to 20 = E2B free-tier cap; raise on paid tiers)\npython scripts/run_eval.py --gold\n\n# score a model's predictions, resuming if interrupted\npython scripts/run_eval.py --predictions preds.jsonl --concurrency 20 --build --resume\n```\n\n| option | default | meaning |\n| --- | --- | --- |\n| `--gold` \\| `--predictions PATH` | — | **required**: gold patches, or a predictions file |\n| `--instances` / `--limit N` / `--per-repo N` | — | restrict the set |\n| `--build` | off | lazily build missing templates first |\n| `--build-workers` | 8 | parallel builds when `--build` is set |\n| `--concurrency` | 20 | concurrent eval sandboxes (E2B free-tier cap; raise on paid tiers) |\n| `--out` | `results` | output directory |\n| `--resume` | off | skip instances already in `\u003cout\u003e/verdicts.jsonl` |\n\nA prediction is one JSON object per line:\n`{\"instance_id\", \"model_name_or_path\", \"model_patch\"}` — the format the official\nharness consumes. For a real agent: hand it a sandbox at HEAD + the\n`problem_statement`, let it edit `/testbed`, and capture `git -C /testbed diff`.\n\n---\n\n## 5. Configuration\n\nAll settings live in `e2b_swebench/config.py` and are overridable via env vars.\nThe per-script CLI flags (§4) take precedence over these defaults.\n\n| env var | default | controls |\n| --- | --- | --- |\n| `SWEBENCH_DATASET` | `princeton-nlp/SWE-bench_Verified` | which dataset to load |\n| `SWEBENCH_SPLIT` | `test` | dataset split |\n| `SWEBENCH_NAMESPACE` | `swebench` | image source — `swebench` pulls the prebuilt images from Docker Hub |\n| `SWEBENCH_CPU` | `8` | vCPUs per template (also `--cpu`) |\n| `SWEBENCH_MEMORY_MB` | `16384` | RAM per template, MiB, must be even (also `--memory-mb`) |\n| `SWEBENCH_SANDBOX_TIMEOUT` | `2400` | sandbox lifetime, seconds |\n| `SWEBENCH_CMD_TIMEOUT` | `1800` | per-command (`eval.sh`) timeout, seconds |\n| `SWEBENCH_CONCURRENCY` | `20` | **max concurrent sandboxes = E2B free-tier cap**; raise on paid tiers (also `--concurrency` / `--verify-concurrency`) |\n\nNotes:\n- Arch is fixed to **x86_64** (E2B is amd64; SWE-bench's arm64 images are incomplete).\n- **Template-build parallelism** is a separate knob — `--workers` (default 4 in\n  `build_templates.py`, 8 in `build_and_verify.py`); it stays below the concurrency cap.\n\n---\n\n## 6. Outputs\n\n| file | written by | purpose |\n| --- | --- | --- |\n| `results/ledger.json` | build_and_verify | per-instance build+verify state; **resume source of truth** |\n| `results/verdicts.jsonl` | run_eval | one verdict per instance as it completes; resume source |\n| `results/report.json` | run_eval | summary + all verdicts |\n| `results/predictions.jsonl` | run_eval | the predictions that were scored |\n\n---\n\n## 7. Layout\n\n```\ne2b_swebench/\n  config.py      dataset, image namespace, resources, timeouts, concurrency\n  dataset.py     load instances, parse tests, gold/empty predictions, select_per_repo\n  templates.py   template_name, instance_image, ensure_template, build_many (process pool)\n  driver.py      run_instance / run_instance_async: write patch → eval.sh → grade\n  runner.py      run_many (concurrent, resumable) + summarize\n  ledger.py      Ledger + categorize_verdict (pass / ordering_artifact / fail / error)\n  logs.py        quiet_logs\nscripts/\n  smoke_test.py        one-instance sanity check\n  build_and_verify.py  batched, resumable build + gold-verify with a ledger\n  build_templates.py   build templates only\n  run_eval.py          evaluate gold patches or a predictions file\n```\n\n### Notes\n\n- **Timeouts in seconds**; sandbox max continuous life is 1 h (Base) / 24 h (Pro).\n- Combined **stdout+stderr** is captured to a file, so Django-style stderr results\n  and non-zero exits are handled.\n- **Fresh sandbox per run**; grading always goes through `get_eval_report` (the\n  per-repo parser), never a hand-rolled regex.\n- Parallel builds use **processes** — the E2B SDK shares one HTTP/2 connection, so\n  thread-based parallelism collides (`invalid_new_stream_id`).\n  ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fe2b-dev%2Fswe-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fe2b-dev%2Fswe-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fe2b-dev%2Fswe-bench/lists"}