{"id":51508553,"url":"https://github.com/blackwell-systems/context-retrieval-benchmark","last_synced_at":"2026-07-08T03:30:39.723Z","repository":{"id":361963997,"uuid":"1256650914","full_name":"blackwell-systems/context-retrieval-benchmark","owner":"blackwell-systems","description":"Reproducible evaluation for context retrieval systems. 308 tasks, 16 repos, 8 languages. P@10, R@10, NDCG, MRR.","archived":false,"fork":false,"pushed_at":"2026-06-02T01:20:18.000Z","size":3,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-02T03:14:23.712Z","etag":null,"topics":["agentic","ai","benchmark","code-context","code-intelligence","code-navigation","code-search","developer-tools","evaluation","ground-truth","information-retrieval","knowledge-graph","llm-tools","mcp","precision","precision-at-k","reproducible-research","retrieval","static-analysis","tree-sitter"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blackwell-systems.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-02T01:18:01.000Z","updated_at":"2026-06-02T01:46:43.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/blackwell-systems/context-retrieval-benchmark","commit_stats":null,"previous_names":["blackwell-systems/code-context-benchmark"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/blackwell-systems/context-retrieval-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackwell-systems%2Fcontext-retrieval-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackwell-systems%2Fcontext-retrieval-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackwell-systems%2Fcontext-retrieval-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackwell-systems%2Fcontext-retrieval-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blackwell-systems","download_url":"https://codeload.github.com/blackwell-systems/context-retrieval-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackwell-systems%2Fcontext-retrieval-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35251015,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-08T02:00:06.796Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agentic","ai","benchmark","code-context","code-intelligence","code-navigation","code-search","developer-tools","evaluation","ground-truth","information-retrieval","knowledge-graph","llm-tools","mcp","precision","precision-at-k","reproducible-research","retrieval","static-analysis","tree-sitter"],"created_at":"2026-07-08T03:30:39.094Z","updated_at":"2026-07-08T03:30:39.711Z","avatar_url":"https://github.com/blackwell-systems.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/blackwell-systems\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/blackwell-systems/blackwell-docs-theme/main/badge-trademark.svg\" alt=\"Blackwell Systems\"\u003e\u003c/a\u003e\n  \u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-MIT-blue.svg\" alt=\"License\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n# code-context-benchmark\n\nReproducible evaluation for code context retrieval systems. 308 tasks, 16 repos, 8 languages.\n\n---\n\n## Why this exists\n\nNo code context retrieval system publishes reproducible precision metrics. codegraph (19K stars), GitNexus (40K stars), Gortex, codebase-memory, and Aider all ship without published P@10, ground truth corpora, or reproducibility tooling.\n\nThis benchmark is the first multi-system evaluation in the space. It is designed to be run by anyone, on any system, with pinned inputs and deterministic scoring.\n\n## What it measures\n\n**P@10 (Precision at 10):** Of the top 10 symbols returned for a task, what fraction are actually relevant? This is the headline metric because it directly measures whether the context an agent receives is useful.\n\nAlso measured: R@10, NDCG@10, MRR, token efficiency, latency.\n\n## Corpus\n\n| Stat | Value |\n|------|-------|\n| Tasks | 308 |\n| Repos | 16 |\n| Languages | Go, Python, TypeScript, Java, C#, Rust, Ruby, multi |\n| Ground truth | Hand-curated, expert-verified, DB-validated |\n| Matching | Dot-bounded containment (no mid-word substring inflation) |\n\nEach repo is pinned to an exact commit. See `corpus/MANIFEST.yaml`.\n\n### Repos\n\n| Repo | Language | Tasks |\n|------|----------|------:|\n| Django | Python | 33 |\n| Terraform | Go | 20 |\n| Caddy | Go | 20 |\n| Jekyll | Ruby | 20 |\n| Rails | Ruby | 20 |\n| Ocelot | C# | 20 |\n| FastAPI | Python | 20 |\n| Spark-Java | Java | 20 |\n| Ripgrep | Rust | 20 |\n| Kafka | Java | 19 |\n| Kubernetes | Go | 19 |\n| Flask | Python | 19 |\n| Cargo | Rust | 19 |\n| VS Code | TypeScript | 19 |\n| Saleor | Python | 11 |\n| Cross-cutting | Mixed | 9 |\n\n## Quick start\n\n```bash\n# 1. Clone this repo\ngit clone https://github.com/blackwell-systems/code-context-benchmark.git\ncd code-context-benchmark\n\n# 2. Clone corpus repos at pinned commits\n./corpus/corpus-setup.sh clone\n\n# 3. Index (requires a code context system that produces graph DBs)\n# Example with knowing:\nKNOWING_BIN=knowing ./corpus/corpus-setup.sh index\n\n# 4. Clear any stale state\nfor db in corpus/repos/*/.knowing/graph.db; do\n  sqlite3 \"$db\" \"DELETE FROM task_memory; DELETE FROM feedback;\" 2\u003e/dev/null\ndone\n\n# 5. Run the benchmark\ngo test ./... -run '^TestCrossSystem$' -v -timeout 0\n```\n\n## Writing an adapter\n\nImplement the `Adapter` interface:\n\n```go\ntype Adapter interface {\n    Name() string\n    Index(repoPath string) (indexTimeMs int64, err error)\n    Retrieve(repoPath string, task Task, tokenBudget int) (RetrievalResult, error)\n    SupportsLearning() bool\n    RecordFeedback(repoPath string, task Task, relevantSymbols []string) error\n    Reset(repoPath string) error\n}\n```\n\nPut your adapter in `adapters/` and register it in the harness. The benchmark runs every registered adapter on every task and reports comparative metrics.\n\n## Methodology\n\n- **Same input, different systems.** Every system receives identical task descriptions.\n- **Cold start.** No pre-existing feedback, session history, or learned state.\n- **Manual ground truth.** Hand-labeled by a developer who read the source code.\n- **Paired statistical tests.** Wilcoxon signed-rank (non-parametric).\n- **Effect size over p-values.** Cohen's d reported alongside significance.\n\nFull methodology: [METHODOLOGY.md](METHODOLOGY.md)\n\n## Task format\n\n```yaml\nid: django-easy-003\nrepo: django\ntier: easy\ndescription: \"Write a management command that exports user data to CSV\"\nground_truth:\n  - BaseCommand\n  - BaseCommand.handle\n  - BaseCommand.add_arguments\n  - OutputWrapper\n  - call_command\ntags: [management-commands, cli]\n```\n\nTasks are in `corpus/tasks/\u003crepo\u003e/\u003ctier\u003e/`.\n\n## Contributing\n\nWe welcome:\n- New task fixtures (with ground truth validated against the graph DB)\n- New corpus repos (with pinned commits and MANIFEST entry)\n- New adapter implementations\n- Methodology improvements\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackwell-systems%2Fcontext-retrieval-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblackwell-systems%2Fcontext-retrieval-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackwell-systems%2Fcontext-retrieval-benchmark/lists"}