{"id":51690611,"url":"https://github.com/lazyisefficient/agentic-benchmark","last_synced_at":"2026-07-16T02:00:42.063Z","repository":{"id":370349950,"uuid":"1292514425","full_name":"LazyIsEfficient/agentic-benchmark","owner":"LazyIsEfficient","description":null,"archived":false,"fork":false,"pushed_at":"2026-07-16T00:17:30.000Z","size":4472,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-16T01:22:50.761Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LazyIsEfficient.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-07-07T15:33:00.000Z","updated_at":"2026-07-16T00:13:34.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/LazyIsEfficient/agentic-benchmark","commit_stats":null,"previous_names":["lazyisefficient/agentic-benchmark"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/LazyIsEfficient/agentic-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LazyIsEfficient%2Fagentic-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LazyIsEfficient%2Fagentic-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LazyIsEfficient%2Fagentic-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LazyIsEfficient%2Fagentic-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LazyIsEfficient","download_url":"https://codeload.github.com/LazyIsEfficient/agentic-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LazyIsEfficient%2Fagentic-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35527286,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-16T02:00:06.687Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-16T02:00:25.495Z","updated_at":"2026-07-16T02:00:42.055Z","avatar_url":"https://github.com/LazyIsEfficient.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CLAUDE.md Variant Benchmarking Harness\n\nMeasures how well different `CLAUDE.md` system-prompt variants perform on real\ncoding tasks, scored on five lexicographic axes — Correctness, Adherence, Craft,\nEfficiency, Reliability. Everything that can be measured deterministically is\nmeasured by the harness; a strong judge model scores only the qualitative\nresidual. Turns \"which doctrine feels better\" into reproducible, auditable\nverdicts — never a weighted mega-score.\n\nEvery agent and judge invocation runs through the official `claude` CLI inside a\nDocker container that is fully isolated from your real `~/.claude`. No Anthropic\nAPI keys, no direct API calls, no Python.\n\n## How it works\n\n```\n                 host (tsx orchestrator)\n                          │\n      for each (executor model × task × variant):\n                          │\n  1. prepareWorkspace  ──▶ temp dir, git baseline, drop variant CLAUDE.md\n                          │\n  2. runExecutor       ──▶ docker run claude -p (tools on, stream-json, --model)\n                          │        auth via env var; workspace→/work bind-mount\n                          │\n  3. captureArtifacts  ──▶ git diff, classify files, extract transcript, signals\n                          │\n  4. deterministic axes ▶ testCommand run in-container, anchor grading, slop\n                          │        metrics, expected-surface scoping (harness)\n                          │\n  5. runJudge          ──▶ docker run claude -p (tools off, prompt over stdin)\n                          │        diff + read-only deterministic context → strict\n                          │        JSON (craft, blast radius, correctness fallback)\n                          │\n  6. pairwise judging  ──▶ A/B craft comparisons per variant pair (on by default)\n                          │\n                 aggregate → reports/\u003crun-folder\u003e/{report.md, report.json, results/}\n```\n\nThe orchestrator runs on the host via `tsx` (no build step) and shells out to\n`docker run` for each container invocation.\n\n## The five axes\n\n**The judge only scores what cannot be measured deterministically.** Correctness\n(tests), adherence (anchors), efficiency (telemetry), and reliability (cross-run\nvariance) are computed by the harness and handed to the judge as read-only\ncontext. The judge owns three things: the qualitative residual of Craft, intent\nclassification for Blast Radius, and a fail-closed correctness assessment ONLY\nwhen no executable check exists.\n\n| Axis | Source | Judge role |\n|------|--------|------------|\n| **Correctness** | the task's optional `testCommand`, run in the workspace container after the executor finishes (pass = exit 0) | fallback verdict (`likely_correct` / `likely_incorrect` / `unknown`) only when the task declares no tests; fail-closed |\n| **Adherence** | graded anchors — `held-by-abstraction` \u003e `held-by-literal` \u003e `held-by-inertia` \u003e `drift` \u003e `trap` (harness, deterministic) | none — read-only context |\n| **Craft** | deterministic slop metrics (duplication windows, churn ratio, residue, test-tamper grep) + judge residual: naming / structure / consistency / economy, each 0–4 ordinal with cited evidence | scores the qualitative residual only |\n| **Efficiency** | tokens, turns, wall-clock, cost (telemetry) | none |\n| **Reliability** | cross-run variance via `--repeats N` | none |\n\nReporting is **lexicographic, never a weighted mega-score**:\n\n1. **Correctness gates everything.** A failing `testCommand` is a failing cell —\n   no amount of craft polish outranks it.\n2. **Adherence is the campaign headline** — did the convention hold, and *how*.\n3. **Craft** = slop deltas + judge medians + pairwise win-rates. Cross-bundle\n   ranking uses A/B pairwise judging with randomized assignment and a\n   position-bias audit — **absolute craft scores are never compared across\n   bundles**.\n4. **Efficiency** and **Reliability** are reported columns.\n\n### Correctness\n\nA task (or campaign link) may declare a `testCommand` in its `meta.json`; the\nharness runs it in the workspace container after the executor finishes\n(`BENCH_TEST_TIMEOUT_MS`, default 300 s) and the exit code is the verdict.\nPass/fail counts are parsed best-effort from node:test / jest / vitest output —\nthe exit code stays authoritative, counts are never fabricated. Campaign links\nmay carry per-link `testCommand`s, since later links accrete code and typically\nneed a wider command than earlier ones.\n\nWhen no `testCommand` exists, the judge supplies a deliberately hedged fallback\nverdict: `likely_correct` / `likely_incorrect` / `unknown`, based solely on\nreading the diff against the task. When in doubt it must say `unknown` — a\nstatic read of a diff is not a test run, and downstream reporting weights it\naccordingly.\n\n### Adherence — graded anchors\n\nAnchors are deterministic detectors (no judge involvement) over the run's diff\nand trace. The graded detector refines \"did the convention hold\" into *how*:\n\n- **`held-by-abstraction`** — the required signals are absent from this link's\n  own diff but present in the cumulative chain diff: the convention persists via\n  a helper built earlier in the chain (campaign mode). The strongest hold.\n- **`held-by-literal`** — the convention was re-emitted literally in this link's\n  own diff.\n- **`held-by-inertia`** — the link never exercised the rule's surface: none of\n  the rule's `appliesIf` regexes matched its diff, so the hold is vacuous, not\n  earned.\n- **`drift`** — the link added real code but the required signals are absent.\n- **`trap`** — a forbidden signal is present in the link's own diff.\n- **`unknown`** — fail-closed when the detector cannot grade (no added lines,\n  malformed rule pattern).\n\nFor `rule` anchors the precedence is exact: trap → literal → inertia →\nabstraction → drift/unknown. Inertia is checked *before* abstraction so a link\nthat never faced the rule can't be spuriously credited via the cumulative diff,\nand forbidden signals are only ever tested against the link's own diff — a\nmarker inherited from an earlier link doesn't poison this link's grade.\n\n### Craft — mechanical slop + judged residual\n\nThe mechanical half is computed by the harness from the diff (pure string\nfunctions — the numbers can't be argued with, and a reader can re-derive any\ncount by hand):\n\n- **`duplicationDelta`** — duplicated 4-line added-line windows\n  (whitespace-normalized, never straddling a file boundary, brace/import noise\n  filtered); N identical windows count as N−1.\n- **`churnRatio`** — campaign links only: the fraction of lines added by earlier\n  links that this link deletes — high churn means the chain is rewriting its own\n  work. `null` for single-shot cells (not measurable, never a fake clean 0).\n- **`residue`** — shipped work-in-progress: TODO/FIXME markers, debug logging\n  (`console.log`/`debugger`), commented-out code — counted on added lines only.\n- **`testTamper`** — signals that the run weakened tests to pass: added\n  `.skip(` / `.only(` / `xit(` / `xdescribe(` / `test.todo(` / `eslint-disable` /\n  `@ts-nocheck` / `--passWithNoTests`, plus deleted `expect(`/`assert` lines —\n  each hit quoted as evidence.\n\nThe judge scores only the residual — four dimensions, each on a 0–4 ordinal\nscale (definitions, not vibes):\n\n- **0 — actively harmful** (misleading names, structure that hides a bug,\n  copy-paste divergence waiting to happen)\n- **1 — poor** (works, but a maintainer would rewrite it)\n- **2 — acceptable** (unremarkable, no objections in review)\n- **3 — good** (a reviewer would approve without comments)\n- **4 — exemplary** (the solution a strong senior engineer would write;\n  teachable)\n\nDimensions: **naming** (identifiers communicate intent), **structure**\n(right-sized functions/modules — both under-abstraction and speculative\nover-abstraction are penalized), **consistency** (matches the surrounding repo's\nidioms, judged against the seed code visible in diff context lines), and\n**economy** (the diff is proportionate to the task; no drive-by rewrites, no\npadding, no dead code). Every score must cite `file:line — quote` evidence from\nthe diff (quotes capped at 10 words); a score without evidence is invalid and\nrecorded as `unknown`. The judge is explicitly told to judge the diff, not the\nagent's narration; verbosity is not rewarded and brevity is not penalized.\n\n### Craft across bundles — pairwise A/B judging\n\nAbsolute craft scores drift across judge calls and are not comparable across\nagent configurations, so cross-bundle ranking comes from pairwise win-rates\ninstead: for each variant pair on the same (task/link × executor model ×\nrepeat), a second judge call compares the two diffs head-to-head per craft\ndimension plus an overall verdict. The A/B assignment is randomized per call to\ncancel position bias, the resolved mapping is recorded so winners always map\nback to variant names, and the report carries a position-bias audit. `tie` is a\nlegitimate and expected verdict; a judge failure degrades to all-tie (a dead\njudge can never move rankings). On by default — disable with `--no-pairwise` /\n`BENCH_PAIRWISE=0` to halve judge cost on exploratory runs.\n\nPer-call randomization cancels position bias only *in expectation*; at the ~4–8\ndecisive comparisons a real run produces it does not cancel *within* a run.\n`--pairwise-both-orders` (opt-in — it doubles the pairwise judge cost) instead\njudges each pair in BOTH seatings and combines: a variant wins the comparison\nonly if it wins in both orders (or wins one and ties the other); a winner that\nflips on order-swap resolves to **tie** (the verdict was position-dependent, not\ncraft-driven). The same rule applies per craft dimension and to the severity\ntag, and if either order fails the whole comparison degrades to tie. In this\nmode the position-bias audit is ~50% by construction, so the report renders\n\"position-cancelled\" rather than a slot percentage. Default off preserves\ntoday's single-order cost and behavior.\n\n### Blast radius (`expectedSurface`)\n\nA task — or an individual campaign link, whose declaration overrides the task's\nwholesale, never merged — may declare `expectedSurface`: glob patterns of the\nfiles the agent is expected to touch. The harness mechanically lists every\nchanged file matching no pattern; the judge only classifies each excursion — it\nnever decides what counts as out-of-scope:\n\n- **`necessary`** — the task could not be completed without it\n- **`defensible`** — not required, but a reasonable reviewer would accept it\n- **`overreach`** — unrequested change with no task justification\n- **`adversarial`** — weakens a check to make the task appear complete\n  (test-expectation edits, disabled lint rules, skipped tests, loosened\n  assertions); must quote the exact weakened check\n\nAny single `adversarial` classification **hard-disqualifies the cell**: it is\nexcluded from every aggregate and reported — a gaming attempt must never be\naveraged away. An absent `expectedSurface` means scoping is off for that task\n(deliberately fail-open: scoping is opt-in per fixture); an explicit `[]` means\n\"this run may touch nothing\", so every touched file is out of scope.\n\n### Fail-closed judge contract\n\n- **The judge sees the DIFF only, never the transcript.** Transcripts are\n  provider-fingerprinted (they leak which harness produced them), so craft is\n  judged from the diff alone. The deterministic context (anchor verdict, test\n  results, slop metrics, out-of-scope files) is rendered read-only in the\n  prompt; the judge is instructed not to re-derive or dispute it.\n- **Strict JSON output.** A parse failure triggers exactly one re-ask with the\n  raw output quoted back plus \"Output valid JSON only.\"; a second failure\n  records the judge's fields as `unknown`/empty — the deterministic axes\n  survive intact.\n- **Out-of-range values are never clamped.** A craft score outside 0–4, or a\n  numeric score with no cited evidence, becomes `unknown` with a visible\n  `invalid:*` flag — a judge malfunction must not masquerade as a real low\n  score. Malformed blast-radius entries are dropped (flagged); an invalid\n  correctness verdict degrades to `unknown`.\n- **Evidence is capped** — 10 words per quoted snippet in-prompt, 120 chars\n  kept at parse — so verdicts stay auditable without republishing the diff.\n- **Truncation fails closed.** The diff embedded in the judge prompt is\n  byte-capped (`BENCH_MAX_DIFF_BYTES`) with a visible `[DIFF TRUNCATED]`\n  marker, and the prompt's own rule tells the judge to output `unknown` for\n  anything it cannot see.\n\n### Task metadata that feeds the axes (`meta.json`)\n\n- **`testCommand`** — arms the deterministic Correctness axis; campaign links\n  may carry per-link overrides.\n- **`expectedSurface`** — arms blast-radius scoping; a campaign link's\n  declaration replaces the task-level one wholesale.\n- **`appliesIf`** (on `rule` anchors) — regexes describing the code surface that\n  exercises the rule; when none match a link's diff the grade is\n  `held-by-inertia` (vacuous hold) instead of a credited literal/abstraction\n  hold. Omitted = applicability unknown; grading falls back to the\n  required/forbidden signals alone.\n\n## Setup\n\nPrerequisites: Docker Desktop and Node 22.\n\n```bash\nnpm install                 # dev deps only (tsx, typescript, @types/node)\nnpm run build-image         # builds claude-bench:latest\nnpm run setup-auth          # ONE-TIME interactive login (see below)\n```\n\n### One-time auth (token model)\n\n`claude setup-token` (requires a Claude subscription) **prints** a long-lived\nsubscription token to the terminal — it does not persist a credential file.\nHeadless/Docker runs supply that token via the `CLAUDE_CODE_OAUTH_TOKEN` env var.\n\n`npm run setup-auth` walks you through it: it runs `claude setup-token`\ninteractively in the image (a browser opens to authorize; if it shows a code\ninstead of redirecting, paste that code at the prompt), then asks you to paste\nthe printed token back. The token is stored at `./.bench-config/oauth-token`\n(chmod 600, gitignored) and is never echoed back. This is interactive and must\nbe run by you.\n\nThe harness resolves the token in this order:\n\n1. `CLAUDE_CODE_OAUTH_TOKEN` if exported (in which case you can skip\n   `setup-auth` entirely), then\n2. the `./.bench-config/oauth-token` file.\n\nEvery container run receives the token as an env var. No credential file is ever\nmounted into a container, and the real `~/.claude` / macOS keychain is never\ntouched.\n\n## Usage\n\n```bash\nnpm run bench -- --list                    # list variants + tasks (no auth needed)\nnpm run bench -- --all                      # full matrix: every variant × every task\nnpm run bench -- --variant tdd-first        # one variant\nnpm run bench -- --variant minimal --variant security-first\nnpm run bench -- --task safe-redirect       # restrict to one task\nnpm run bench -- --all --models fable,sonnet,opus   # compare across executor models\nnpm run bench -- --all --concurrency 3      # run up to 3 cells in parallel\nnpm run bench -- --all --delay-ms 5000      # pace cells to ease rate limits\nnpm run bench -- --all --repeats 3          # run each cell 3× (Reliability: cross-run variance)\nnpm run bench -- --all --no-pairwise        # skip A/B craft judging (halves judge cost)\nnpm run bench -- --report reports/\u003crun\u003e/    # regenerate a report from a finished run (offline)\nnpm run bench -- --report reports/\u003crun\u003e/ --focus craft   # focused re-render: one concern only\n```\n\n### Focused reports: `--focus \u003caxis\u003e`\n\n`--focus \u003caxis\u003e` renders a **focused report** — only the named concern's\nsection(s) plus the run header — instead of the full multi-axis report. Handy\nwhen iterating on one dimension (e.g. tuning a CLAUDE.md's craft or memory\ndoctrine) without scrolling past the other axes. Accepted axes:\n\n| `--focus` | Renders |\n| --- | --- |\n| `correctness` | Correctness table |\n| `memory` | Memory effect (single-shot and/or campaign) |\n| `craft` | Slop, judge craft, pairwise, Craft Score |\n| `efficiency` | Cost / time / turns table |\n| `blast-radius` | Out-of-scope file excursions |\n\nAn unknown axis fails fast with the accepted set enumerated. The cross-task\ninsight callout and the observational tails (Reliability, Excluded cells,\nBehavioral signals) are full-report only and are dropped in a focused render.\n`report.json` is always the complete payload — `--focus` only narrows the\nrendered `report.md`, so it composes with `--report` for an offline re-render.\n\n### The benchmark matrix: variant × task × executor model\n\nThe executor model is a first-class dimension. `--models \u003ccsv\u003e` (comma/space\nseparated, repeatable) runs each selected (variant × task) against each executor\nmodel, so you can compare how a CLAUDE.md doctrine performs on e.g. Fable vs\nOpus vs Sonnet. Without `--models`, a single model (`BENCH_EXECUTOR_MODEL`,\ndefault `sonnet`) is used — behavior is identical to before.\n\nModel aliases: `fable`, `opus`, `sonnet`, `haiku`.\n\n**The judge model is held FIXED** (`BENCH_JUDGE_MODEL`, default `opus`) across\nevery run. Varying the judge would make scores non-comparable, so only the\nexecutor varies; every result is tagged with both `executorModel` and the fixed\n`judgeModel`. Preflight probes auth for each distinct executor model plus the\njudge model.\n\n### Concurrency\n\n`--concurrency N` (alias `-c`, default `3`, or `BENCH_CONCURRENCY`) runs up to\nN matrix cells (`executorModel × task × variant`) at once through a bounded\nworker pool. At `N=1` the run is fully sequential with live per-line logging.\nAt `N\u003e1` each cell's log lines are **buffered** and emitted as\none contiguous block with a `[k/total]` progress counter, so concurrent cells\nstay readable instead of interleaving.\n\nThe report is sorted deterministically (executor model → variant → task) before\nrendering, so output is stable regardless of `N`.\n\nTradeoffs: higher N trims wall-clock but multiplies CPU/RAM pressure from\nconcurrent resource-heavy executor containers (each may run `npm install`, build,\ntest) and raises the chance of transient API rate-limit/overload responses —\nwhich the existing retry-with-backoff absorbs. `2`–`3` is a sane range; values\nabove `16` are clamped. Invalid or `\u003c 1` values are rejected with a clear error.\n\n### Timeouts\n\nEach executor run is bounded by `BENCH_EXECUTOR_TIMEOUT_MS` (default 1800 s;\njudge by `BENCH_JUDGE_TIMEOUT_MS`, default 300 s). Enforcement is authoritative:\non timeout the harness `docker kill`s the container **and** SIGKILLs the local\n`docker run` client, and if the client still hasn't exited within a grace window\n(`BENCH_CONTAINER_KILL_GRACE_MS`, default 10 s) the run is force-resolved as\ntimed-out. This caps a run's wall-clock at ≈ timeout + grace — earlier a hung,\nrate-limited client lingered ~14 minutes past the kill.\n\n**A timeout is terminal for that cell — it is NOT retried.** A timeout almost\nalways means the session hung (usually API rate-limiting), so a retry would just\nburn another full timeout; the cell is recorded as failed and the matrix moves\non. Genuine transient failures (spawn error, non-timeout non-zero exit) still\nretry with backoff.\n\n### Rate limits\n\nSustained large matrices — especially many Opus cells back-to-back — can hit\nsubscription rate limits. The symptom is executor runs that stall (the trace\nshows repeated `api_retry` events) and eventually hit the timeout. Mitigate by:\n\n- running smaller `--models` batches,\n- adding `--delay-ms N` (default 0) to pause N ms between cells — both\n  sequentially and before each pool dispatch — easing sustained pressure, and/or\n- lowering `--concurrency`.\n\nBefore running the matrix, the CLI performs a preflight auth probe in a\ncontainer. If it is not logged in, it prints instructions to run\n`npm run setup-auth` and exits non-zero — it never tries to auth for you.\n\n### Output: durable, never-overwritten per-run folders\n\nEvery `bench` invocation gets a **GUID `runId`** (`crypto.randomUUID()`) and its\nown self-contained folder under `reports/` — the report and all cell results\nlive together, and no run ever overwrites another:\n\n```\nreports/\n  \u003crevkey\u003e__\u003ciso\u003e__\u003cshort8\u003e/\n    report.md            # includes the full runId GUID in its header\n    report.json          # meta.runId = full GUID\n    results/\n      \u003ccellId\u003e/          # cellId = task__variant__modelSlug (unique within a run)\n        diff.patch  trace.ndjson  transcript.txt  result.json  workspace/\n```\n\n- **`\u003crevkey\u003e`** is a fixed-width, zero-padded **reverse-time key**\n  (`HORIZON_MS - runEpochMs`). A newer run yields a *smaller* number, so a plain\n  ascending name sort (the default in most tools) lists the **newest run first**.\n  The leading segment is deliberately this sort key, not a readable date — the\n  human-readable UTC stamp follows it.\n- **`\u003ciso\u003e`** is a filesystem-safe UTC stamp (`2026-07-08T15-30-00Z`).\n- **`\u003cshort8\u003e`** is the first 8 hex of the GUID for a readable folder name; the\n  full GUID is in `report.json` and the `report.md` header.\n\nTwo ids, disambiguated:\n- **`runId`** — the per-execution GUID (one per `bench` invocation / whole matrix).\n- **`cellId`** — the per-(variant × task × model) id, `task__variant__modelSlug`,\n  unique within a run (no timestamp needed).\n\n`reports/` is gitignored.\n\n### Environment overrides\n\n| Variable | Default | Meaning |\n|----------|---------|---------|\n| `BENCH_IMAGE` | `claude-bench:latest` | Docker image tag |\n| `BENCH_EXECUTOR_MODEL` | `sonnet` | Model that does the coding |\n| `BENCH_JUDGE_MODEL` | `opus` | Model that scores |\n| `BENCH_EXECUTOR_TIMEOUT_MS` | `1800000` | Per-executor-run timeout |\n| `BENCH_CONCURRENCY` | `3` | Default cell concurrency (overridden by `--concurrency`) |\n| `BENCH_JUDGE_TIMEOUT_MS` | `300000` | Per-judge-run timeout |\n| `BENCH_TEST_TIMEOUT_MS` | `300000` | Timeout for a task's `testCommand` container (Correctness axis) |\n| `BENCH_SETUP_TIMEOUT_MS` | `300000` | Per setup-bundle pre-step timeout |\n| `BENCH_CONTAINER_KILL_GRACE_MS` | `10000` | Grace before force-resolving a hung run after timeout |\n| `BENCH_INTER_CELL_DELAY_MS` | `0` | Default pause between cells (overridden by `--delay-ms`) |\n| `BENCH_PAIRWISE` | on | Pairwise A/B craft judging; only `0`/`false` disable (any other value, including a typo, stays on). Overridden by `--no-pairwise` |\n| `BENCH_REPEATS` | `1` | Runs per (variant × task × model) cell (Reliability axis). Overridden by `--repeats N` |\n| `BENCH_MAX_DIFF_BYTES` | `200000` | Byte cap on the diff shown to the judge |\n| `BENCH_MAX_TRANSCRIPT_BYTES` | `200000` | Byte cap on the captured transcript (artifact only — the judge never sees it) |\n\n### Evidence handling\n\n- **Prompts are sent over stdin**, not as a `-p \"\u003c...\u003e\"` argv element, so a large\n  evidence bundle can never hit the OS `ARG_MAX` limit (`spawn E2BIG`).\n- **Dependency/build artifacts are never captured.** Each workspace's\n  `.git/info/exclude` ignores `node_modules/`, `dist/`, `build/`, `coverage/`,\n  `.next/`, `.turbo/`, `*.log`, `.DS_Store` (and the variant `CLAUDE.md`), so a\n  legitimate `npm install` is not counted as the agent's work.\n- **Evidence is size-capped.** The redacted diff is truncated at\n  `BENCH_MAX_DIFF_BYTES` before the judge sees it, with a visible truncation\n  marker — and a truncated diff triggers the judge's own fail-closed rule\n  (output `unknown` for anything it cannot see). The transcript — which the\n  judge never sees — is still captured, redacted, and capped at\n  `BENCH_MAX_TRANSCRIPT_BYTES` as an artifact. Truncation sets\n  `evidenceTruncated` on the result and shows a note in the report. The full\n  redacted diff/transcript are still written to `results/\u003crunId\u003e/`.\n\n### Run metrics (cost \u0026 time KPIs) — the Efficiency axis\n\nEach run also records how expensive and slow the variant was to execute. These\nremain **observational per cell** and now surface as the **Efficiency axis** —\na reported column, still never folded into the other axes or their ranking.\n\n- **Wall-clock time** (headline): host-measured around the whole `docker run`\n  (spawn → exit), so it includes container startup, `npm install`, etc. — the\n  truest \"actual time spent\". The CLI's own `duration_ms` (agent-session time)\n  and `duration_api_ms` are captured too.\n- **Cost**: `total_cost_usd` from the claude result event. On a subscription\n  this is the *would-be* API cost — a useful proxy, not an actual charge. The\n  **token counts** (`input`/`output`, plus cache read/create) are the ground\n  truth, so both are captured.\n- **Turns**: the agent's `num_turns`.\n\nBoth the executor and judge calls are measured. Where they appear:\n\n- **Console** — appended to each cell's judged log line, alongside that cell's\n  craft summary and anchor grade: exec wall time, cost, tokens in/out, turns.\n- **`reports/report.md`** — the Efficiency section: a per-(variant × model)\n  table with exec time, exec cost, input/output tokens, turns, judge cost.\n  Absent CLI fields render as `—` (never `undefined`/`NaN`). Cost/tokens/time\n  are **summed** across each (variant × model)'s tasks (total spend).\n- **`\u003crunDir\u003e/report.json`** and each **`\u003crunDir\u003e/results/\u003ccellId\u003e/result.json`**\n  — the full raw `metrics` object (executor + judge `CallMetrics`, all fields).\n\n### Report sections\n\n`report.md` walks the axes in lexicographic order — the same priority the\nscoring uses:\n\n- **Correctness** — per-cell `testCommand` verdicts (pass/fail, plus parsed test\n  counts when the runner output was parseable); cells without executable tests\n  show the judge's hedged fallback verdict instead.\n- **Memory effect** (adherence) — the campaign headline: graded anchor verdicts,\n  rendered with graded symbols — `✓A` held-by-abstraction, `✓L` held-by-literal,\n  `~I` held-by-inertia, `✗` drift, `⚠` trap, `?` unknown.\n- **Craft** — the slop-metrics table, per-cell judge craft medians, and pairwise\n  win-rates with the position-bias audit. Absolute craft medians are comparable\n  only within a bundle; cross-bundle ranking reads the win-rates.\n- **Efficiency** — cost/time/token columns per (variant × model).\n- **Reliability** — cross-run variance when `--repeats N` \u003e 1.\n- **Blast radius** — out-of-scope touches with the judge's classification;\n  adversarial entries are called out with the cell's disqualification.\n- Behavior comparison and **`## Excluded cells (not scored)`** — as before.\n\nMulti-model runs (`--models`) group each section per executor model. The\naggregation unit is `(variant × executorModel)` across tasks — axis values are\n**never averaged or ranked across models** (they aren't interchangeable), and\npairwise comparisons never cross models either.\n\n`report.json` keeps a flat `results` array (each entry tagged with\n`executorModel`/`judgeModel`, so it's groupable by model, plus a derived\n`scored` flag / `excludedReason`), an `executorModels` list, and the fixed\n`judgeModel`; runs that exercised campaigns or pairwise judging additionally\ncarry `campaigns` (per-link trajectories) and `pairwise` (A/B comparisons)\narrays.\n\n### Scored vs excluded cells (only real verdicts count)\n\nAggregates are computed over **scored cells only** — a cell counts iff it\nproduced a real judge verdict (executor OK **and** no judge failure; timeouts\nalready fail the executor). This keeps wall-clock (the executor timeout) from\npolluting the ranking:\n\n- A failed/timed-out cell is a **coverage gap, never a fabricated\n  bottom-of-scale verdict**. It is excluded from every aggregate and listed\n  under **`## Excluded cells (not scored)`** with its reason.\n- Per-unit **coverage** is reported — e.g. `2/3 scored, 1 excluded`. A unit with\n  **zero** scored cells reads as excluded, never as a real verdict, and ranks\n  **last**.\n- A genuine judged verdict — however unflattering — **does** count: only the\n  failure path is excluded, never a real verdict.\n- Disqualified cells (any `adversarial` blast-radius entry) are likewise\n  excluded from every aggregate and reported — gaming is surfaced, never\n  averaged away.\n- Time/cost KPIs are unchanged: Efficiency remains observational, still includes\n  failed/timed-out attempts (honest cost accounting), and never feeds the other\n  axes.\n\n### Regenerating a report (`--report`, offline)\n\n`npm run bench -- --report \u003cpath\u003e` rebuilds `report.md` + `report.json` from a\nfinished run — `\u003cpath\u003e` is a run folder (containing `report.json`) or a\n`report.json` directly. It reloads the saved results, re-runs the current\naggregation + rendering, and rewrites both files in place. Pure and offline: no\nDocker, no auth, no executor/judge calls — so historical runs can be\nre-aggregated after a methodology change like this one.\n\n## Variant types\n\nA variant is a `prompts/\u003cname\u003e/` directory. Its shape is declared by an optional\n`variant.json` manifest — absent ⇒ `claude-md` (today's behavior):\n\n```jsonc\n{\n  \"type\": \"claude-md\" | \"bundle\",   // default \"claude-md\"\n  \"claudeMd\": \"CLAUDE.md\",           // CLAUDE.md filename within the dir\n  \"configDir\": \"claude\",             // .claude tree dirname (copy bundles)\n  \"install\": \"copy\" | \"setup\",       // bundles only; default \"copy\"\n  \"setupCommand\": \"…\",               // required when install=\"setup\"\n  \"description\": \"…\"                 // shown in --list\n}\n```\n\n- **`claude-md`** — a lone `CLAUDE.md`. Its content is written to\n  `\u003cworkspace\u003e/CLAUDE.md`. (An empty file — the `naked` variant — models an empty\n  project memory.)\n- **`bundle`** — a `CLAUDE.md` PLUS a `.claude/` tree of skills/agents/commands/\n  hooks/settings, installed at **project scope** (`\u003cworkspace\u003e/.claude/`, where\n  Claude Code reads them). Two install modes:\n  - **`install: \"copy\"`** (default, e.g. `agentic-os`) — a vendored\n    `.claude/` tree (`configDir`, e.g. `prompts/\u003cname\u003e/claude/`) is copied in and\n    every shipped `*.sh` is `chmod +x`ed so hooks run.\n  - **`install: \"setup\"`** (e.g. `gstack`) — the bundle ships no vendored\n    `.claude/`; instead its source is **baked into the image** and a\n    `setupCommand` registers the skills at runtime. Before the executor runs, the\n    harness injects the CLAUDE.md, then runs a **setup pre-step container**\n    (mounting `/work`, executing `setupCommand`) that populates\n    `\u003cworkspace\u003e/.claude/skills`. Its output is saved to `results/\u003ccellId\u003e/setup.log`.\n    If the pre-step leaves `.claude/skills` empty, the cell is recorded as failed\n    (\"bundle setup failed\") and the judge is skipped — the matrix continues.\n- **Diff isolation:** for every bundle, both `CLAUDE.md` and the entire\n  `.claude/` tree are added to the workspace's `.git/info/exclude`, so none of the\n  shipped/registered config (skills, agents, hooks) appears in the captured diff\n  or `changedFiles` — only the agent's own work is scored.\n\n### Variants shipped\n\n- `naked` — empty `CLAUDE.md` (baseline: no guidance at all).\n- `minimal` — almost no guidance.\n- `senior-verbose` — clean architecture / SOLID / readability; no testing or\n  security mandate.\n- `tdd-first` — mandates a real testing framework and thorough tests.\n- `security-first` — mandates an explicit security review, threat modeling,\n  secure defaults, and docs.\n- `agentic-os` — **copy bundle**: the full agentic-os v3.0.1 harness\n  ([github.com/LazyIsEfficient/agentic-os](https://github.com/LazyIsEfficient/agentic-os))\n  — CLAUDE.md + a vendored `.claude/` (38 skills, agents, commands, hooks, rules,\n  settings), copied in at project scope.\n- `gstack` — **setup bundle**: Garry's Stack 1.58.5.0 `@11de390`\n  ([github.com/garrytan/gstack](https://github.com/garrytan/gstack)) — Bun-native.\n  Its source is baked into the image at `/opt/gstack`; a runtime `setup --local`\n  registers **55 skills** into `\u003cworkspace\u003e/.claude/skills` before the executor\n  runs.\n\nThe lone-CLAUDE.md variants are designed to produce genuinely different outcomes\nacross the axes so the harness visibly discriminates between doctrines; the\nbundles test whether a full skills/agents harness beats a plain prompt.\n\n### Bundle caveats\n\n- **Cost \u0026 latency.** Bundles that dispatch subagents (agentic-os leans on the\n  Task tool) multiply API calls per cell → far higher cost/latency and much more\n  rate-limit exposure. Prefer low `--concurrency`, `--delay-ms`, and small\n  `--models` batches when benchmarking bundles.\n- **Global-path hooks/scripts.** A bundle's hooks/scripts that assume a global\n  `~/.claude` (or a `~/.cursor`) path won't resolve under a project-scope install\n  and may no-op. For agentic-os v3.0.1 the auto-run hooks in `settings.json` are\n  all project-relative (`bash .claude/hooks/*.sh`) and self-contained, so they\n  run; but some **agent/command instruction docs** reference repo-root\n  `scripts/*.sh` and a `$HOME/.cursor/.../ledger.py` findings-ledger that do not\n  exist at project scope — if an agent tries to invoke those they will fail/no-op\n  (a benign degradation, not a correctness issue). Repo-root `scripts/` is\n  deliberately NOT vendored because no hook references it.\n- **v3.0.1 memory loop — write hook needs `jq` (now in the image), read hook\n  doesn't.** agentic-os **writes** durable facts via a `Stop` hook\n  (`.claude/hooks/memory-extract.sh`, the `memory-extraction` skill) — the fix for\n  upstream issue #217 — and, as of v3.0.1, **reads them back** via a `SessionStart`\n  hook (`.claude/hooks/memory-inject.sh`) that injects the `.claude/memory/MEMORY.md`\n  index into each fresh session — the fix for issue #225. The write hook is\n  **fail-open on `jq`**: with no `jq` on `PATH` it emits `{}` and silently no-ops, so\n  the benchmark `Dockerfile` installs `jq`; strip it and recording stops. The read\n  hook needs no `jq` (grep + cat). In **campaign mode** (where `.claude/memory/`\n  persists across the chain) these two hooks together are what let a convention\n  recorded in one link actually influence a later, context-reset link.\n- **MCP-dependent skills.** Skills that rely on MCP servers are unavailable\n  inside the container.\n\n### gstack setup bundle — image \u0026 fidelity notes\n\n- **Bun + baked G-Stack.** The image installs Bun 1.3.10 and bakes a *built*\n  G-Stack at `/opt/gstack` (`bun install \u0026\u0026 bun run build`). The build must run\n  on the image's own filesystem — `bun run build` fails over the macOS Docker\n  bind-mount (virtiofs `rename ENOENT`) — so G-Stack is never built in `/work`.\n  `/opt/gstack/bin` is on `PATH` so its tools (browse, make-pdf) resolve.\n- **Image size.** Bun, G-Stack's `node_modules`, the ~101 MB `browse` binary, and\n  a baked Playwright Chromium + its host libs grow the image substantially\n  (multiple GB). All variants share this one image; `naked`/`agentic-os`/\n  claude-md variants are functionally unaffected (G-Stack skills are only\n  registered for the `gstack` variant's cells).\n- **Chromium is required for setup.** G-Stack's `setup` hard-gates on a\n  launchable Playwright Chromium under `set -e` (it aborts before registering\n  skills otherwise), and it honors `GSTACK_SKIP_FONTS`/`GSTACK_SKIP_COREUTILS`/\n  `GSTACK_SKIP_GBRAIN_REGEN` (all set `=1` in the image) but has **no** flag to\n  skip the browser check. So Chromium + its apt libs are baked in; with them\n  present, `setup --local -q --no-prefix` exits 0 and registers 55 skills.\n- **Fidelity boundary.** Core skills and the Bun toolchain work in-container.\n  Browser/Chrome-driven skills are only partially functional (headless Chromium\n  is present, but full browser automation in a container is not validated here),\n  and any skill needing external services/MCP is unavailable.\n\n## The task\n\n`tasks/safe-redirect` — implement `resolveSafeRedirect(userSuppliedUrl,\nallowlist)` that prevents open redirects (allowlisted scheme/host, rejects\n`javascript:`/`data:`/protocol-relative/credentials-in-URL, normalizes relative\npaths) plus a small Express-style handler. It is logic-bearing and\nsecurity-relevant. The task prompt states the requirement but deliberately does\n**not** ask for tests or a security review — that is exactly what the variants\ninfluence and the axes measure.\n\n## Security \u0026 isolation model\n\n- **Auth is a single env var — no credential file is mounted.** Every container\n  run receives `CLAUDE_CODE_OAUTH_TOKEN` as an env var (resolved from the host\n  env or `./.bench-config/oauth-token`). Nothing from the host is bind-mounted\n  for auth. The container keeps its own image-provided writable `/cfg`, which is\n  ephemeral and discarded by `--rm`.\n- **What IS bind-mounted:** only a fresh per-run workspace dir → `/work`\n  (read-write), so the host can read the resulting `git diff` after the\n  container exits. Nothing else.\n- **What is NEVER mounted or read:** your real `~/.claude` and the macOS\n  keychain. Isolation is enforced by Docker plus a scoped env var, not by host\n  config. A container with no token reports \"Not logged in\", which is how the\n  preflight probe validates auth.\n- **`--dangerously-skip-permissions`** is passed to the *executor* so the agent\n  can work unattended — but only ever **inside the container**, against a\n  throwaway workspace. It is never used on the host.\n- **The judge runs with `--tools \"\"`** (all tools disabled). It cannot read\n  files or run commands; it only reasons over the evidence embedded in its\n  prompt — the diff plus read-only deterministic context, never the transcript.\n  This keeps scoring reproducible and prevents the judge from wandering.\n- **`--no-session-persistence`** (executor) keeps runs from leaking state into\n  each other.\n- The variant `CLAUDE.md` is dropped into the workspace *after* the git baseline\n  commit and registered in `.git/info/exclude`, so it is visible to the agent as\n  its system prompt but never counted as the agent's own work in the diff.\n- **Defense in depth around the token:** before any captured artifact\n  (`diff.patch`, `transcript.txt`) is written to the host `results/` dir, a\n  redaction pass replaces any occurrence of the OAuth token with\n  `[REDACTED-CREDENTIAL]`. The token is only ever held in-memory for this\n  comparison — never logged or persisted anywhere else.\n- **Known limitation — the executor's agent can read its own token.** The token\n  is delivered as an env var to the executor container, so the benchmarked agent\n  can read `CLAUDE_CODE_OAUTH_TOKEN` from its environment. This is inherent to the\n  headless subscription model (the agent needs the token to call the API). It is\n  acceptable here because you are benchmarking your own prompts and the token is\n  user-revocable (`claude setup-token` can re-issue). Results redaction stops the\n  token from being persisted to disk, but only run variants and tasks you trust.\n- **Known limitation — network egress is open.** Containers need outbound network\n  to reach the Anthropic API, so egress is not restricted. A hostile task or\n  variant in the executor could attempt to exfiltrate over the wire. Locking\n  egress to the API endpoints (an egress proxy or firewalled Docker network)\n  would close the gap if needed.\n\n## Development\n\n```bash\nnpm run typecheck    # tsc --noEmit\nnpm test             # node --test on the pure-function unit tests\n```\n\nThe harness dogfoods its own scoring: the pure functions behind the axes — slop\nmetrics, anchor grading, surface scoping, judge-output parsing, classification,\nreporting — are unit-tested with the zero-dependency `node:test` runner.\nContainer-dependent code paths (Docker spawn, executor, judge I/O) are exercised\nby the real benchmark run.\n\n## Layout\n\n```\nDockerfile              pinned claude CLI image\nscripts/                build-image.sh, setup-auth.sh\nsrc/\n  config.ts             paths, image, models, timeouts, scoring toggles (env-overridable)\n  auth.ts               OAuth token resolution (env var → token file)\n  types.ts              domain types\n  variant.ts            variant.json manifest parse + defaults\n  rubric.ts             cell-judge prompt (craft/blast/correctness residual) + output contract\n  docker.ts             docker run wrappers (executor, judge, auth probe)\n  workspace.ts          per-run workspace prep + git baseline + bundle materialize\n  capture.ts            diff, file classification, transcript, signals\n  anchors.ts            deterministic anchor detectors + graded verdicts (Adherence)\n  slop.ts               mechanical slop metrics over the diff (Craft, deterministic half)\n  surface.ts            expectedSurface glob scoping (blast radius, deterministic half)\n  metrics.ts            parse cost/time KPIs from the result event + formatters\n  pool.ts               bounded-concurrency worker pool (order-preserving)\n  runmeta.ts            per-run GUID folder naming + reverse-time sort key\n  executor.ts           run one (variant × task × model) cell\n  judge.ts              evidence → cell judge → fail-closed parse → verdict\n  pairwise.ts           A/B craft comparisons (randomized order, fail-closed ties)\n  report.ts             markdown + JSON aggregation\n  cli.ts                arg parsing, preflight, orchestration\nprompts/                variants under test — claude-md dirs + bundle dirs\n  \u003cname\u003e/CLAUDE.md      lone-CLAUDE.md variant (or a bundle's injected doctrine)\n  \u003cname\u003e/variant.json   optional manifest (declares bundle + install mode)\n  \u003cname\u003e/claude/        vendored .claude tree (copy bundles: skills/agents/hooks/…)\n  gstack/gstack-src/    G-Stack source, COPY'd to /opt/gstack + built in the image\ntasks/                  the benchmark task(s)\nreports/                generated per-run folders (gitignored)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flazyisefficient%2Fagentic-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flazyisefficient%2Fagentic-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flazyisefficient%2Fagentic-benchmark/lists"}