https://github.com/blackwell-systems/context-retrieval-benchmark
Reproducible evaluation for context retrieval systems. 308 tasks, 16 repos, 8 languages. P@10, R@10, NDCG, MRR.
https://github.com/blackwell-systems/context-retrieval-benchmark
agentic ai benchmark code-context code-intelligence code-navigation code-search developer-tools evaluation ground-truth information-retrieval knowledge-graph llm-tools mcp precision precision-at-k reproducible-research retrieval static-analysis tree-sitter
Last synced: 2 days ago
JSON representation
Reproducible evaluation for context retrieval systems. 308 tasks, 16 repos, 8 languages. P@10, R@10, NDCG, MRR.
- Host: GitHub
- URL: https://github.com/blackwell-systems/context-retrieval-benchmark
- Owner: blackwell-systems
- License: mit
- Created: 2026-06-02T01:18:01.000Z (about 1 month ago)
- Default Branch: main
- Last Pushed: 2026-06-02T01:20:18.000Z (about 1 month ago)
- Last Synced: 2026-06-02T03:14:23.712Z (about 1 month ago)
- Topics: agentic, ai, benchmark, code-context, code-intelligence, code-navigation, code-search, developer-tools, evaluation, ground-truth, information-retrieval, knowledge-graph, llm-tools, mcp, precision, precision-at-k, reproducible-research, retrieval, static-analysis, tree-sitter
- Size: 2.93 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
---
# code-context-benchmark
Reproducible evaluation for code context retrieval systems. 308 tasks, 16 repos, 8 languages.
---
## Why this exists
No code context retrieval system publishes reproducible precision metrics. codegraph (19K stars), GitNexus (40K stars), Gortex, codebase-memory, and Aider all ship without published P@10, ground truth corpora, or reproducibility tooling.
This benchmark is the first multi-system evaluation in the space. It is designed to be run by anyone, on any system, with pinned inputs and deterministic scoring.
## What it measures
**P@10 (Precision at 10):** Of the top 10 symbols returned for a task, what fraction are actually relevant? This is the headline metric because it directly measures whether the context an agent receives is useful.
Also measured: R@10, NDCG@10, MRR, token efficiency, latency.
## Corpus
| Stat | Value |
|------|-------|
| Tasks | 308 |
| Repos | 16 |
| Languages | Go, Python, TypeScript, Java, C#, Rust, Ruby, multi |
| Ground truth | Hand-curated, expert-verified, DB-validated |
| Matching | Dot-bounded containment (no mid-word substring inflation) |
Each repo is pinned to an exact commit. See `corpus/MANIFEST.yaml`.
### Repos
| Repo | Language | Tasks |
|------|----------|------:|
| Django | Python | 33 |
| Terraform | Go | 20 |
| Caddy | Go | 20 |
| Jekyll | Ruby | 20 |
| Rails | Ruby | 20 |
| Ocelot | C# | 20 |
| FastAPI | Python | 20 |
| Spark-Java | Java | 20 |
| Ripgrep | Rust | 20 |
| Kafka | Java | 19 |
| Kubernetes | Go | 19 |
| Flask | Python | 19 |
| Cargo | Rust | 19 |
| VS Code | TypeScript | 19 |
| Saleor | Python | 11 |
| Cross-cutting | Mixed | 9 |
## Quick start
```bash
# 1. Clone this repo
git clone https://github.com/blackwell-systems/code-context-benchmark.git
cd code-context-benchmark
# 2. Clone corpus repos at pinned commits
./corpus/corpus-setup.sh clone
# 3. Index (requires a code context system that produces graph DBs)
# Example with knowing:
KNOWING_BIN=knowing ./corpus/corpus-setup.sh index
# 4. Clear any stale state
for db in corpus/repos/*/.knowing/graph.db; do
sqlite3 "$db" "DELETE FROM task_memory; DELETE FROM feedback;" 2>/dev/null
done
# 5. Run the benchmark
go test ./... -run '^TestCrossSystem$' -v -timeout 0
```
## Writing an adapter
Implement the `Adapter` interface:
```go
type Adapter interface {
Name() string
Index(repoPath string) (indexTimeMs int64, err error)
Retrieve(repoPath string, task Task, tokenBudget int) (RetrievalResult, error)
SupportsLearning() bool
RecordFeedback(repoPath string, task Task, relevantSymbols []string) error
Reset(repoPath string) error
}
```
Put your adapter in `adapters/` and register it in the harness. The benchmark runs every registered adapter on every task and reports comparative metrics.
## Methodology
- **Same input, different systems.** Every system receives identical task descriptions.
- **Cold start.** No pre-existing feedback, session history, or learned state.
- **Manual ground truth.** Hand-labeled by a developer who read the source code.
- **Paired statistical tests.** Wilcoxon signed-rank (non-parametric).
- **Effect size over p-values.** Cohen's d reported alongside significance.
Full methodology: [METHODOLOGY.md](METHODOLOGY.md)
## Task format
```yaml
id: django-easy-003
repo: django
tier: easy
description: "Write a management command that exports user data to CSV"
ground_truth:
- BaseCommand
- BaseCommand.handle
- BaseCommand.add_arguments
- OutputWrapper
- call_command
tags: [management-commands, cli]
```
Tasks are in `corpus/tasks///`.
## Contributing
We welcome:
- New task fixtures (with ground truth validated against the graph DB)
- New corpus repos (with pinned commits and MANIFEST entry)
- New adapter implementations
- Methodology improvements
## License
MIT