{"id":50907430,"url":"https://github.com/supermodeltools/dead-code-benchmark-blog","last_synced_at":"2026-06-16T06:31:21.958Z","repository":{"id":343323234,"uuid":"1177100047","full_name":"supermodeltools/dead-code-benchmark-blog","owner":"supermodeltools","description":"Dead code detection benchmark: data, blog post, and analysis for Supermodel's graph-powered dead code tool","archived":false,"fork":false,"pushed_at":"2026-03-27T21:24:07.000Z","size":12177,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-28T03:55:40.702Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/supermodeltools.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-09T17:36:45.000Z","updated_at":"2026-03-27T21:24:11.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/supermodeltools/dead-code-benchmark-blog","commit_stats":null,"previous_names":["supermodeltools/dead-code-benchmark-blog"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/supermodeltools/dead-code-benchmark-blog","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supermodeltools%2Fdead-code-benchmark-blog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supermodeltools%2Fdead-code-benchmark-blog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supermodeltools%2Fdead-code-benchmark-blog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supermodeltools%2Fdead-code-benchmark-blog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/supermodeltools","download_url":"https://codeload.github.com/supermodeltools/dead-code-benchmark-blog/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supermodeltools%2Fdead-code-benchmark-blog/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34393304,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-16T02:00:06.860Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-16T06:31:21.296Z","updated_at":"2026-06-16T06:31:21.947Z","avatar_url":"https://github.com/supermodeltools.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dead Code Benchmark Blog\n\nThis repo contains everything for our dead code detection benchmark: raw data from previous runs, the blog post draft, analysis of false positive patterns, and instructions for running new benchmarks.\n\n## Repository Structure\n\n```\ndead-code-benchmark-blog/\n  README.md                           # This file (planning, methodology, run instructions)\n  blog/\n    2026-03-09-dead-code-graphs-ai-agents.md   # Blog post draft\n  runs/\n    feb-06-initial/                   # First antiwork/helper benchmark\n    feb-17-18-full-suite/             # 16-run benchmark (blogpost-deadcode.zip)\n    feb-20-with-synthetic/            # 10-task MCP vs baseline (fri-feb-20.zip)\n    feb-23-post-fixes/                # 16-task MCP-only post-fixes (sun-feb-23.zip)\n    mar-09-post-parser-improvements/  # NEW: Run after barrel re-export + 7 new phases\n  analysis/\n    fp-taxonomy.md                    # False positive root cause taxonomy\n    parser-improvement-plan.md        # Prioritized improvement roadmap\n    benchmark-improvement-system.md   # Continuous improvement loop design\n```\n\n---\n\n## Previous Benchmark Results Summary\n\n### Timeline\n\n| Date | Run | Tasks | Key Finding |\n|------|-----|-------|-------------|\n| Feb 6 | Initial | antiwork/helper + synthetic | **91.7% recall, 100% precision** on real codebase (resolved!) |\n| Feb 9 | Replication | Same as above | Identical results -- confirmed reproducibility |\n| Feb 17-18 | Full suite | 16 runs (synthetic + 12 real PRs) | MCP wins 9/16, 0 resolved on large repos |\n| Feb 20 | MCP vs baseline | 10 real PRs | **Baseline scored 0% on all 10 tasks.** MCP: 75-100% recall on 6/10 |\n| Feb 23 | Post-fixes (MCP only) | 16 PRs | Directus 80% recall, some improvement |\n| **Mar 9** | **Post parser improvements** | **TBD** | **Run this now -- see instructions below** |\n\n### Best MCP Recall Per Real-World Task (across all runs)\n\n| Task | Best Recall | Best Precision | Run |\n|------|------------|----------------|-----|\n| directus_pr26311 | **100%** | 0.6% | Feb 20 |\n| podman_pr16084 | **100%** | 0.3% | Feb 20 |\n| latitude_pr2300 | **100%** | 0.3% | Feb 20 |\n| tyr_pr258 | **95.5%** | 3.8% | Feb 20 |\n| antiwork/helper | **91.7%** | **100%** | Feb 6 (RESOLVED) |\n| mimir_pr3613 | **77.8%** | 0.6% | Feb 20 |\n| otel_js_pr5444 | **75.0%** | 1.3% | Feb 20 |\n| maskbook_pr12361 | **63.6%** | 0.2% | Feb 20 |\n\n### Parser Improvements Since Last Benchmark (Feb 20-23)\n\nThese changes landed but have NOT been measured in a full benchmark yet:\n\n1. **Barrel re-export filtering** (Mar 6, #444/#469)\n   - Internal testing: recall 59.0% -\u003e 85.2% across 5 repos\n   - tyr: 36.4% -\u003e 86.4%, mimir: 77.8% -\u003e 88.9%, directus: 80% -\u003e 93.3%, latitude: 80% -\u003e 100%\n   - Propagates `isReExport` flag, excludes re-export-only consumers from import counts\n   - Step 2b fixpoint loop: cascading dead import detection\n\n2. **7 new pipeline phases** (Feb 26-Mar 6, #469)\n   - Phase 2b: Vue SFC import extraction\n   - Phase 2c: Dynamic `import()`/`require()` filesystem loading detection\n   - Phase 4c: Library package detection (types/, module/, packages/)\n   - Phase 4d: Cross-package import resolution for monorepos\n   - Phase 6b: Class-to-method reachability propagation\n   - Phase 6d: Exported object literal property reachability\n   - Directus: 2,808 candidates -\u003e 2,178 (-22.4%), preserved 22/23 known TPs\n\n3. **Class rescue via new/instanceof/extends** (Mar 6, #497)\n   - Non-exported classes used via `new`, `instanceof`, `extends` no longer flagged dead\n   - Example: `BlobNotFoundError` in job-worker.service.ts\n\n4. **Same-file rescue patterns** (broadened, #443)\n   - Callback references: `items.map(processItem)`\n   - Object literal methods: `{ handler: myFunction }`\n   - JSX component references: `\u003cTopLinks /\u003e`\n   - Module-scope calls: `const X = getZipLimits()`\n   - Default parameters: `function foo(cb = myHelper) {}`\n\n---\n\n## Running a New Benchmark\n\n### Prerequisites\n\n1. **Docker Desktop** running\n2. **Environment variables** set:\n   ```bash\n   export ANTHROPIC_API_KEY=\"sk-ant-...\"\n   export SUPERMODEL_API_KEY=\"smsk_live_...\"\n   ```\n3. **mcpbr-eval** repo set up (see below)\n\n### Setup (one-time)\n\n```bash\n# The benchmark repo\ncd ~/mcpbr-eval\ngit checkout feat/supermodel-benchmark\n\n# Install dependencies\nuv pip install -e \".[dev]\"\n\n# Verify it works\nuv run pytest -m \"not integration\"\n```\n\n### Run the Benchmark\n\n**Option A: Run all 17 tasks (MCP-only, ~$5-10, ~2 hours)**\n\nThis is the fastest way to get \"after\" numbers for the blog post. No baseline needed since we already have baseline data showing 0% across the board.\n\n```bash\ncd ~/mcpbr-eval\nuv run mcpbr run --config config/supermodel-deadcode-pr.yaml --mcp-only -vv\n```\n\n**Option B: Run specific high-signal tasks (recommended first)**\n\nStart with the 5 tasks that had internal recall improvements:\n\n```bash\ncd ~/mcpbr-eval\n\n# These are the tasks with known recall improvements from parser fixes\nfor task in tyr_pr258 jslpsolver_pr159 mimir_pr3613 directus_pr26311 latitude_pr2300; do\n  echo \"=== Running $task ===\"\n  uv run mcpbr run --config config/supermodel-deadcode-pr.yaml --mcp-only -t $task -vv --no-incremental\ndone\n```\n\n**Option C: Run with baseline comparison (~$15-20, ~4 hours)**\n\nFull A/B comparison. Only needed if we want fresh baseline numbers.\n\n```bash\ncd ~/mcpbr-eval\nuv run mcpbr run --config config/supermodel-deadcode-pr.yaml -vv\n```\n\n### After the Run\n\nResults will be in the latest `.mcpbr_run_*` directory:\n\n```bash\n# Find the latest run\nls -td ~/mcpbr-eval/.mcpbr_run_* | head -1\n\n# Copy results to this repo\nLATEST=$(ls -td ~/mcpbr-eval/.mcpbr_run_* | head -1)\ncp -r \"$LATEST\" ~/dead-code-benchmark-blog/runs/mar-09-post-parser-improvements/\n\n# Quick summary\ncat \"$LATEST/results.json\" | python3 -c \"\nimport json, sys\nd = json.load(sys.stdin)\nprint(f\\\"MCP: {d['summary']['mcp']['resolved']}/{d['summary']['mcp']['total']} resolved\\\")\nprint(f\\\"Cost: \\${d['summary']['mcp']['total_cost']:.2f}\\\")\nfor t in d['tasks']:\n    m = t['mcp']\n    print(f\\\"  {t['instance_id']}: P={m.get('precision',0):.1%} R={m.get('recall',0):.1%} F1={m.get('f1_score',0):.1%} ({m.get('true_positives',0)} TP, {m.get('false_positives',0)} FP)\\\")\n\"\n```\n\n### Configuration Reference\n\nThe benchmark config is at `~/mcpbr-eval/config/supermodel-deadcode-pr.yaml`. Key settings:\n\n| Setting | Value | Notes |\n|---------|-------|-------|\n| `model` | `claude-sonnet-4-20250514` | Claude Sonnet 4 |\n| `agent_harness` | `claude-code` | Uses Claude Code as agent |\n| `max_iterations` | 30 | Agent turn limit |\n| `timeout_seconds` | 1200 | 20 min per task |\n| `supermodel_api_base` | staging.api.supermodeltools.com | Uses staging API |\n| `resolved_threshold` | 0.8 | P\u003e=80% AND R\u003e=80% to \"resolve\" |\n\n### Task Quality Notes\n\nFrom the config file comments and benchmark experience:\n\n| Task | Quality | Notes |\n|------|---------|-------|\n| tyr_pr258 | Best | 95.5% API recall (21/22 GT), clean TypeScript monorepo |\n| directus_pr26311 | Good | 15 GT items, multi-package structure |\n| mimir_pr3613 | Good | 9 GT items, Statistics Norway portal |\n| latitude_pr2300 | Good | 5 GT items, monorepo with packages |\n| jslpsolver_pr159 | Good | 10 GT items, smaller repo (baseline previously outperformed) |\n| podman_pr16084 | OK | Only 2 GT items (small sample) |\n| otel_js_pr5444 | OK | 4 GT items, non-determinism observed |\n| maskbook_pr12361 | OK | 22 GT items but 6000+ candidates (analysis dump risk) |\n| gemini_cli_pr18681 | Poor | 33% API recall, test-import gap |\n| prisma_pr28485 | Poor | 0% recall (GT items only referenced by tests) |\n| n8n_pr23572 | Invalid | Feature removal, not dead code (GT items are alive) |\n| typescript_pr56817 | Blocked | Too large for staging API without cached_analysis |\n\n---\n\n## False Positive Taxonomy\n\n### 7 Categories Ranked by Impact\n\n| Priority | Root Cause | % of FPs | Status (Mar 9) |\n|----------|-----------|----------|----------------|\n| **P0** | Import resolution: too many root files (145 in Tyr, should be ~3) | ~51% | **Partially fixed** (barrel re-exports done, framework wiring still missing) |\n| **P1** | `export type { X } from` re-exports not tracked | ~15% | Not fixed |\n| **P2** | `export default X` not detected as export | ~5% | Not fixed |\n| **P3** | JSX usage not recognized as calls (React components) | ~20% in React repos | **Partially fixed** (same-file rescue catches `\u003cFoo /\u003e`, cross-file misses) |\n| **P4** | Test/script/config file imports not scanned | 11/28 scream test FPs | Not fixed |\n| **P5** | Python `isExported` broken (only checks `__all__`) | Disables Python pipeline | Not fixed |\n| **P6** | Type/interface structural typing references | ~21% in TS repos | Hard to fix |\n\n### Detail: P0 - Import Resolution\n\nThe dominant false positive source. When import resolution fails to trace how a file is consumed (e.g., Express `app.use(router)`, dynamic `require()`, framework wiring), the file becomes a \"root file\" with no importers. All its exports then get flagged as \"exported but file never imported.\"\n\nIn the Tyr benchmark, there were **145 root files** (should be ~3-5). This single issue produced ~396 of 775 candidates (51%).\n\n**What's fixed:** Barrel re-exports now correctly traced. Step 2b fixpoint loop handles cascading dead imports.\n\n**What's still broken:** Framework-level wiring (Express router mounting, NestJS module registration, dynamic requires).\n\n### Detail: P4 - Test/Script Imports\n\nSymbols imported only from test files, build scripts, or config files (e.g., `vitest.config.ts`, `rollup.config.mjs`) appear to have \"no importers\" because the scanner doesn't cover those file types. This was 11 of 28 FPs in the jslpsolver scream test.\n\n**Highest-ROI unfixed item.** Scanning test file imports would eliminate a huge chunk of FPs with minimal regression risk.\n\n---\n\n## Benchmark-Driven Parser Improvement System\n\n### The Loop\n\n```\n1. Run benchmark -\u003e measure P/R/F1 per task\n2. Sample FPs -\u003e categorize root causes (using taxonomy above)\n3. Pick highest-impact root cause -\u003e implement fix\n4. Run benchmark again -\u003e verify improvement, check for regressions\n5. Repeat\n```\n\n### Tiered Testing\n\n| Tier | Speed | Cost | What It Tests | When to Run |\n|------|-------|------|--------------|-------------|\n| **Tier 1** | Seconds | Free | Unit tests: does parser extract `export default`? | Every commit |\n| **Tier 2** | Minutes | Free | Candidate-level regression: run analysis on 5 repos, diff candidate counts | Every PR |\n| **Tier 3** | Hours | $5-50 | Full agent benchmark via mcpbr: measures whole pipeline end-to-end | Weekly / pre-release |\n\n### Tier 2: Candidate-Level Regression (Cheap \u0026 Fast)\n\nBefore running an expensive Tier 3 benchmark, verify parser changes with a candidate-level check:\n\n```bash\n# 1. Check out the benchmark repo at pre-PR commit\n# 2. Run Supermodel dead code analysis (just the API, no agent)\n# 3. Count candidates and check if known TPs are still detected\n# 4. Compare candidate count to baseline\n\n# Example for Directus:\n# Before fix: 2,808 candidates, 22/23 TPs detected\n# After fix:  2,178 candidates (-22.4%), 22/23 TPs still detected\n# = Good: fewer FPs, no TP regression\n```\n\n### Ground Truth Registry\n\nFormalize the 12 PRs as a regression test suite:\n\n```json\n{\n  \"tasks\": [\n    {\n      \"id\": \"tyr_pr258\",\n      \"repo\": \"uncovering-world/track-your-regions\",\n      \"merge_commit\": \"6f480121...\",\n      \"ground_truth_count\": 22,\n      \"best_api_recall\": 0.955,\n      \"best_agent_recall\": 0.955,\n      \"known_misses\": [\"one item missed by API\"]\n    }\n  ]\n}\n```\n\n### FP Classification Pipeline\n\nAfter each benchmark run, automatically categorize FPs:\n\n1. For each FP, check: imported by test file? Framework entry point? Type re-export? Generated directory?\n2. Tag each FP with root cause category\n3. Track category volumes over time\n4. Build a precision/recall dashboard per root cause\n\n---\n\n## Key Files in Related Repos\n\n| What | Where |\n|------|-------|\n| **mcpbr-eval** (benchmark runner) | `~/mcpbr-eval` (branch: `feat/supermodel-benchmark`) |\n| **Benchmark config** | `~/mcpbr-eval/config/supermodel-deadcode-pr.yaml` |\n| **Supermodel benchmark code** | `~/mcpbr-eval/src/mcpbr/benchmarks/supermodel/` |\n| **Dead code endpoint impl** | `~/mcpbr-eval/src/mcpbr/benchmarks/supermodel/endpoints/dead_code.py` |\n| **Parser (tree-sitter)** | `~/supermodel-public-api/src/data-plane/src/parsers/tree-sitter.ts` |\n| **Dead code pipeline** | `~/supermodel-public-api/src/data-plane/src/services/job-worker.service.ts:1344-3181` |\n| **Blog post draft** | `~/jonathanpopham.github.io/_drafts/2026-03-09-dead-code-graphs-ai-agents.md` |\n| **Previous run data (Downloads)** | `~/Downloads/blogpost-deadcode/`, `fri-feb-20-*`, `sun-feb-23-*` |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsupermodeltools%2Fdead-code-benchmark-blog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsupermodeltools%2Fdead-code-benchmark-blog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsupermodeltools%2Fdead-code-benchmark-blog/lists"}