{"id":50527758,"url":"https://github.com/repowise-dev/repowise-bench","last_synced_at":"2026-06-03T09:31:02.890Z","repository":{"id":358910415,"uuid":"1202499139","full_name":"repowise-dev/repowise-bench","owner":"repowise-dev","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-29T15:25:48.000Z","size":42455,"stargazers_count":6,"open_issues_count":1,"forks_count":3,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-05-29T16:11:45.167Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/repowise-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-06T04:53:08.000Z","updated_at":"2026-05-29T15:27:01.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/repowise-dev/repowise-bench","commit_stats":null,"previous_names":["repowise-dev/repowise-bench"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/repowise-dev/repowise-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/repowise-dev%2Frepowise-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/repowise-dev%2Frepowise-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/repowise-dev%2Frepowise-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/repowise-dev%2Frepowise-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/repowise-dev","download_url":"https://codeload.github.com/repowise-dev/repowise-bench/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/repowise-dev%2Frepowise-bench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33858569,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-03T02:00:06.370Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-03T09:31:01.476Z","updated_at":"2026-06-03T09:31:02.880Z","avatar_url":"https://github.com/repowise-dev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# repowise-bench — Benchmark Suite\n\n\u003e **[Repowise](https://github.com/repowise-dev/repowise)** is the codebase\n\u003e intelligence layer for AI coding agents. It indexes repositories into five\n\u003e intelligence layers — dependency graphs, git analytics, auto-generated docs,\n\u003e architectural decisions, and code health scores — and exposes them through\n\u003e nine MCP tools. The result: fewer tool calls, fewer file reads, lower LLM\n\u003e costs, and health scores that predict real-world defects.\n\u003e\n\u003e **This repo proves those claims with reproducible benchmarks on public\n\u003e codebases.**\n\n[![GitHub stars](https://img.shields.io/github/stars/repowise-dev/repowise?style=flat)](https://github.com/repowise-dev/repowise)\n[![License](https://img.shields.io/github/license/repowise-dev/repowise)](https://github.com/repowise-dev/repowise/blob/main/LICENSE)\n[![Latest Release](https://img.shields.io/github/v/release/repowise-dev/repowise)](https://github.com/repowise-dev/repowise/releases)\n\n---\n\n## Benchmarks\n\n| Benchmark | Status | Headline | Report |\n|-----------|--------|----------|--------|\n| [**SWE-QA**](#swe-qa-coding-agent-efficiency) | Complete | -36-70% tool calls, -29-36% cost, quality at parity | [flask48](BENCHMARK_REPORT_FLASK48.md) · [sklearn48](BENCHMARK_REPORT_SKLEARN48.md) |\n| [**health-defect**](#health-defect-code-health-vs-defect-prediction) | Complete | 10-75x defect ratio, ROC AUC 0.70-0.74 | [README](health-defect/README.md) · [full report](health-defect/BENCHMARK_REPORT.md) |\n\n---\n\n## SWE-QA — Coding Agent Efficiency\n\nA paired benchmark comparing two coding-agent configurations on\n[SWE-QA](https://arxiv.org/abs/2401.00000) tasks drawn from\n[`pallets/flask`](https://github.com/pallets/flask) and\n[`scikit-learn/scikit-learn`](https://github.com/scikit-learn/scikit-learn).\n\n**What is compared:**\n\n| Configuration | Tools available to the agent |\n|---------------|------------------------------|\n| **C0_bare** | `Read`, `Grep`, `Glob`, `Bash`, `Agent` (built-in coding-agent toolkit) |\n| **C2_full** | All of the above **plus** four MCP tools (`get_answer`, `get_symbol`, `get_context`, `search_codebase`) backed by a precomputed documentation index of the repository |\n\nBoth configurations use the same model (`claude-sonnet-4-6`), the same SWE-QA\nprompt scaffolding, the same per-task budget cap, and the same LLM judge. The\nonly variable is the tool surface presented to the agent.\n\n### flask48 — `pallets/flask` (48 paired tasks)\n\n| Metric | C0 (baseline) | C2 (doc-augmented) | Δ |\n|---|---:|---:|---:|\n| Cost / task (mean) | $0.1396 | $0.0890 | **-36.2 %** |\n| Wall / task (mean) | 41.7 s | 33.9 s | **-18.6 %** |\n| Tool calls (mean) | 7.4 | 3.8 | **-49.2 %** |\n| Files read (mean) | 1.9 | 0.2 | **-89.0 %** |\n| Score (0-10, mean) | 8.82 | 8.81 | tied |\n\n**32 / 48 (67 %)** tasks are cheaper under C2; quality is at parity.\n\nFull report: [**BENCHMARK_REPORT_FLASK48.md**](BENCHMARK_REPORT_FLASK48.md)\n\n### sklearn48 — `scikit-learn/scikit-learn` (48 paired tasks)\n\n| Metric | C0 (baseline) | C2 (doc-augmented) | Δ |\n|---|---:|---:|---:|\n| Cost / task (mean) | $0.1180 | $0.0834 | **-29.3 %** |\n| Wall / task (mean) | 39.7 s | 28.6 s | **-27.9 %** |\n| Tool calls (mean) | 8.1 | 2.4 | **-70.5 %** |\n| Files read (mean) | 1.8 | 0.6 | **-69.3 %** |\n| Score (0-10, mean) | 8.72 | 8.23 | similar on this sample |\n\n**33 / 48 (69 %)** tasks are cheaper under C2; **28 / 48 (58 %)** are faster.\n\nFull report: [**BENCHMARK_REPORT_SKLEARN48.md**](BENCHMARK_REPORT_SKLEARN48.md)\n\n### Bonus: token-efficiency benchmark\n\nHow many tokens does each strategy require for a model to understand a commit,\nmeasured on the 30 most recent non-merge commits of `pallets/flask`?\n\n| Strategy | Tokens / commit |\n|---|---:|\n| naive (full contents of changed files) | 64,039 |\n| `git diff` only | 14,888 |\n| **`get_context`** | **2,391** |\n\nReduction vs **naive**: **209x mean**, 26.8x pooled, 12.6x median, 1,214x best case.\nReduction vs **`git diff`**: 41.7x mean, 6.2x pooled.\n\nReproduce:\n\n```bash\n.venv/bin/python harness/token_efficiency_bench.py \\\n    --repo repos/pallets/flask --last 30 --min-repowise-tokens 0\n```\n\nRaw data: `results/token_efficiency/results.csv`.\n\n---\n\n## health-defect — Code Health vs. Defect Prediction\n\nA reproducible benchmark proving that deterministic code health scores predict\nreal-world defects in open-source Python projects. Health scores are collected\nat a historical snapshot (T0); bug-fixing commits are counted over the following\n6 months (T0 -\u003e T1); the two are correlated.\n\n### Headline numbers\n\nAcross three public repositories (862 source files, 6-month defect window):\n\n| Repo | Files | Spearman ρ | p-value | Defect ratio | ROC AUC | Precision@20 |\n|------|------:|----------:|---------:|-------------:|--------:|-------------:|\n| Django | 542 | **-0.337** | \u003c0.0001 | **12x** | 0.698 | **70 %** |\n| Pydantic | 216 | -0.229 | 0.0007 | 10x | **0.742** | 30 % |\n| FastAPI | 104 | -0.272 | 0.0053 | 75x | 0.715 | 35 % |\n\n**Files scoring below 4.0 have 10-75x more bug-fixing commits than files\nscoring above 8.0.** The correlation is statistically significant (p \u003c 0.01)\nacross all three codebases.\n\nTop biomarker predictors (by Cliff's delta effect size):\n\n1. `developer_congestion` — δ = +0.78 (Django)\n2. `untested_hotspot` — δ = +0.69 (Django), +0.67 (FastAPI)\n3. `brain_method` — δ = +0.62 (Pydantic), +0.43 (Django)\n\nFull report: [**health-defect/BENCHMARK_REPORT.md**](health-defect/BENCHMARK_REPORT.md)\nReproduction steps: [**health-defect/README.md**](health-defect/README.md)\n\n---\n\n## Repository layout\n\n```\nrepowise-bench/\n├── README.md                         — this file (index of all benchmarks)\n├── requirements.txt                  — shared Python dependencies\n│\n├── harness/                          — shared runner infrastructure (SWE-QA)\n│   ├── run_experiment.py             — entry point: orchestrates a paired run\n│   ├── swe_qa_runner.py              — per-task runner + LLM-as-judge\n│   ├── metrics.py                    — RunMetrics, stream parser, BudgetTracker\n│   └── token_efficiency_bench.py     — token-efficiency mini-benchmark\n│\n├── configs/                          — benchmark configuration files (SWE-QA)\n│   └── swe_qa_flask48.yaml           — canonical SWE-QA / Flask configuration\n│\n├── data/                             — static benchmark datasets\n│   └── swe_qa/tasks.json             — full SWE-QA task corpus\n│\n├── analysis/                         — aggregation scripts (SWE-QA)\n│   └── aggregate_flask48.py\n│\n├── scripts/                          — shared utility scripts\n│   └── download_benchmarks.py        — fetches SWE-QA dataset and clones repos\n│\n├── results/                          — all benchmark outputs (gitignored except baselines)\n│   ├── swe_qa_flask48/               — SWE-QA Flask results\n│   ├── swe_qa_sklearn48/             — SWE-QA scikit-learn results\n│   ├── token_efficiency/             — token-efficiency results\n│   └── health_defect_{repo}/         — one directory per health-defect repo\n│       ├── correlation.json\n│       ├── defect_counts.json\n│       ├── joined_data.json\n│       ├── health_scores.json\n│       └── charts/\n│\n├── BENCHMARK_REPORT_FLASK48.md       — SWE-QA full report: Flask\n├── BENCHMARK_REPORT_SKLEARN48.md     — SWE-QA full report: scikit-learn\n│\n├── health-defect/                    — self-contained health-defect benchmark\n│   ├── README.md                     — benchmark overview and reproduction steps\n│   ├── BENCHMARK_REPORT.md           — full statistical report\n│   ├── config.yaml                   — per-repo configuration\n│   ├── run_benchmark.py              — entry point\n│   └── lib/                          — benchmark library modules\n│\n├── mcp_configs/                      — generated MCP server configs (gitignored)\n├── indexes/                          — generated documentation indexes (gitignored)\n├── repos/                            — cloned target repositories (gitignored)\n└── logs/                             — per-run logs (gitignored)\n```\n\n---\n\n## Adding a new benchmark\n\nEach benchmark gets its own directory. Convention:\n\n1. **Create a directory** at `repowise-bench/\u003cbenchmark-name\u003e/`\n2. **Add a `README.md`** with methodology, headline numbers, and reproduction steps\n3. **Add a `run_benchmark.py`** (or equivalent entry point) runnable from within the directory\n4. **Write results to `../results/\u003cbenchmark_name\u003e_{variant}/`** so outputs land in the shared `results/` tree\n5. **Update this README** — add a row to the [Benchmarks](#benchmarks) table\n\nShared repos and indexes can be reused from `../repos/` and `../indexes/`. New Python dependencies go in the top-level `requirements.txt`.\n\n---\n\n## SWE-QA methodology\n\n### Pairing\n\nEvery task is run under both conditions, and every metric is computed per-task\nbefore being aggregated. We never compare a C0 mean against a C2 mean drawn\nfrom a different subset of tasks. If a task fails to complete under one\ncondition, it is re-run under both conditions and the new pair replaces the\nold one in full.\n\n### Cost accounting\n\nCost is read directly from each task's `estimated_cost_usd` field, populated\nfrom the agent runtime's per-model billing roll-up. This sums cost across\nevery model invoked — both the parent session and any subagents dispatched\nvia the `Agent` tool. Token-based recomputation is intentionally avoided\nbecause it can miss subagent spend not surfaced in the parent stream's\n`usage` blocks.\n\n### Judge\n\nEach (task, configuration) pair is scored by an LLM judge using a fixed\nfive-dimension rubric (correctness, completeness, relevance, clarity,\nreasoning) on a 0-10 scale. The judge does not see the configuration label\nand is the same model in both arms.\n\n### Reproducibility\n\nRuns are deterministic up to LLM nondeterminism. Model versions, prompt\ntemplates, and the SWE-QA task corpus are pinned in this repository. The\nonly external dependencies are the repository checkouts (pinned by commit\nhash in the documentation index metadata) and the Anthropic API.\n\n---\n\n## SWE-QA reproduction\n\nThe full pipeline takes about 30 minutes of wall-clock time per arm and costs\napproximately $5-10 per arm at list prices, depending on retry behavior.\n\n### Prerequisites\n\n- **Python 3.11+**\n- **Claude Code CLI** (`claude`) installed and authenticated (OAuth or\n  `ANTHROPIC_API_KEY`)\n- **repowise CLI** installed and discoverable on `$PATH`, or a local checkout\n  of repowise sibling to this directory\n- ~5 GB free disk space for the checkout, index, and run logs\n\n### 1. Install Python dependencies\n\n```bash\npip install -r requirements.txt\n```\n\n### 2. Fetch the repo checkout and SWE-QA task corpus\n\n```bash\npython scripts/download_benchmarks.py --benchmark swe_qa\n```\n\n### 3. Build the C2 documentation index (optional — built on demand if absent)\n\n```bash\nrepowise init repos/pallets/flask --output-dir indexes\n```\n\n### 4. Run the benchmark\n\n```bash\nPYTHONIOENCODING=utf-8 python harness/run_experiment.py \\\n    --config configs/swe_qa_flask48.yaml\n```\n\nResults are written incrementally to `results/swe_qa_flask48/swe_qa.jsonl`;\nthe run is safe to interrupt and resume.\n\n### 5. Aggregate the results\n\n```bash\npython analysis/aggregate_flask48.py\n```\n\nFor health-defect reproduction steps, see [health-defect/README.md](health-defect/README.md).\n\n---\n\n## SWE-QA output schema\n\nEach row of `results/swe_qa_flask48/swe_qa.jsonl` contains:\n\n| Field | Type | Description |\n|---|---|---|\n| `task_id` | string | Unique task identifier (e.g. `flask_017`) |\n| `benchmark` | string | Always `swe_qa` |\n| `condition` | string | `C0_bare` or `C2_full` |\n| `repo` | string | Source repository (e.g. `pallets/flask`) |\n| `question_type` | string | SWE-QA question category (What / Where / How / Why) |\n| `answer` | string | The agent's final answer |\n| `judge_scores` | dict[str,float] | Judge dimension scores in [0, 10] |\n| `estimated_cost_usd` | float | Total dollar cost across all models invoked |\n| `wall_clock_seconds` | float | End-to-end wall-clock duration |\n| `num_tool_calls` | int | Total tool invocations made by the agent |\n| `files_explored` | list[str] | Distinct file paths opened via `Read` |\n\nFor the health-defect output schema, see [health-defect/README.md](health-defect/README.md).\n\n---\n\n## Citation\n\nIf you use these benchmarks or their results, please cite the relevant report:\n\n```\nRepowise on SWE-QA: A Benchmark Study of Documentation-Augmented Code\nQuestion Answering on Flask. 2026.\n```\n\n```\nRepowise health-defect Benchmark: Code Health Scores as Defect Predictors\nAcross Django, FastAPI, and Pydantic. 2026.\n```\n\n---\n\n## License\n\nThis benchmark harness is released under the Apache 2.0 license. The repository\ncheckouts used as targets are owned by their respective projects and licensed\nseparately. The SWE-QA task corpus is the property of its original authors.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frepowise-dev%2Frepowise-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frepowise-dev%2Frepowise-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frepowise-dev%2Frepowise-bench/lists"}