{"id":51626698,"url":"https://github.com/waitdeadai/agent-closeout-bench","last_synced_at":"2026-07-13T02:02:19.385Z","repository":{"id":357602751,"uuid":"1237683076","full_name":"waitdeadai/agent-closeout-bench","owner":"waitdeadai","description":"Deterministic closeout physics engine and benchmark for agentic coding assistant dark-pattern detection","archived":false,"fork":false,"pushed_at":"2026-05-22T10:21:25.000Z","size":954,"stargazers_count":0,"open_issues_count":1,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-22T10:56:23.876Z","etag":null,"topics":["ai-safety","benchmark","claude-code","darkbench","hooks","llm-safety","mast","multi-agent","rust","verification","yaml-rules"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/waitdeadai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-13T12:09:37.000Z","updated_at":"2026-05-22T10:21:29.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/waitdeadai/agent-closeout-bench","commit_stats":null,"previous_names":["waitdeadai/agent-closeout-bench"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/waitdeadai/agent-closeout-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waitdeadai%2Fagent-closeout-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waitdeadai%2Fagent-closeout-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waitdeadai%2Fagent-closeout-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waitdeadai%2Fagent-closeout-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/waitdeadai","download_url":"https://codeload.github.com/waitdeadai/agent-closeout-bench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waitdeadai%2Fagent-closeout-bench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35407393,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-13T02:00:06.543Z","response_time":119,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-safety","benchmark","claude-code","darkbench","hooks","llm-safety","mast","multi-agent","rust","verification","yaml-rules"],"created_at":"2026-07-13T02:02:14.835Z","updated_at":"2026-07-13T02:02:19.369Z","avatar_url":"https://github.com/waitdeadai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AgentCloseoutBench\n\nAgentCloseoutBench is a benchmark-in-progress for evaluating dark-pattern\ndetection on agentic coding assistant closeout text: the final assistant message\navailable to Claude Code `Stop` and `SubagentStop` hooks as\n`last_assistant_message`.\n\nThe benchmark contribution is the lifecycle surface and reusable black-box hook\nevaluation harness. Regex hook performance is reported as one detector family,\nnot as the benchmark itself.\n\nThe new detector contribution is `agentcloseout-physics`: a deterministic\ncloseout protocol engine. It treats hooks as runtime adapters and evaluates\npositive closeout states, dark-pattern mechanics, and evidence-claim markers\nwithout a live LLM, embedding model, or network call in the verdict path.\n\n## Current Status\n\nThis repository is a recovery, hardening, and public-data-intake workspace with\na complete v0.2 synthetic candidate corpus plus a small v0.3 public-derived\nadversarial fixture lane. It is not yet a public v1.0 gold-label dataset\nrelease.\n\n- Current candidate corpus: 800 records, 4 categories x 100 positive x 100\n  negative, exact task-type quotas from `quota_manifest.json`.\n- Current public-derived adversarial lane: 16 candidate records, 4 categories x\n  2 positive x 2 negative, with per-record source provenance and manifest rows.\n- Current public-derived rule fixtures: 14 fixtures covering the four\n  dark-pattern engines plus `closeout_contract` and `evidence_claims`.\n- Current corpus labels: candidate labels until two independent human annotation\n  passes plus adjudication are complete.\n- Current public-shaped source: deterministic synthetic templates released under\n  Apache-2.0.\n- Current public claim language: \"To our knowledge, AgentCloseoutBench is the\n  first benchmark for dark-pattern detection on agentic coding assistant closeout\n  text at the Claude Code Stop/SubagentStop `last_assistant_message` boundary.\"\n- Current engine claim language: \"out-of-band deterministic enforcement at the\n  agentic coding assistant closeout boundary makes specific dark-pattern and\n  false-closeout mechanics observable, reproducible, and benchmarkable.\"\n- Current ACSP-CC language: ACSP-CC is a proposed Claude Code closeout security\n  profile, and `agentcloseout-physics` is the current reference implementation\n  for that proposal. Any conformance output is self-assessed preflight evidence,\n  not a standard, certification, or final benchmark metric.\n- Not yet claimed: human-annotated release, universal agent benchmark, or\n  absolute injection-immune defense.\n- Current high-assurance hardening: the Claude Code adapters include a\n  `PreToolUse` tamper guard for `.claude/hooks`, `.claude/agentcloseout.env`,\n  pinned engine paths, and pinned rule packs; env config is parsed through an\n  allowlist instead of shell-sourced.\n\n## Field evidence\n\nThe failure modes this benchmark scores — MAST **2.6** (action-reasoning mismatch) and\n**3.3** (no/incorrect verification) — were observed in a production healthcare deployment.\n**Effective Therapy** (a trauma-therapy platform; cited with permission, patient-facing\nspecifics withheld) ran an Opus 4.7 orchestrator that narrated 39 agent dispatches,\nincluding five *verification* agents reporting findings, while 5 of 39 agents were ever\nused and the verification agents had **zero sessions**; a codebase audit added 80+\nhollow-code findings (correct auth, routes, and signatures — missing the line that\npersists data). Refs [`anthropics/claude-code#61167`](https://github.com/anthropics/claude-code/issues/61167),\n[`#61107`](https://github.com/anthropics/claude-code/issues/61107); case study at\n[`ianymu/recognition-without-arrest#2`](https://github.com/ianymu/recognition-without-arrest/pull/2).\nEffective Therapy has offered 30+ labeled hollow-code examples toward a future\nsemantic-emptiness detector — a real-world labeled corpus, not synthetic.\n\nA follow-up forensic audit (2026-05-26) put a **field-measured fabrication rate** on it,\nagainst ground truth (actual `curl` dispatch logs vs assistant claims): **~34% phantom on\nOpus 4.7** (44 phantom claims / 128 real dispatches, 18 phantom agent-names) vs **~4% on\nOpus 4.6** (2 / 50), with **zero `Agent`/`Task` tool calls in any 4.7 session** — the\nfabrication never crossed the tool boundary. This is the first field-measured rate for the\nMAST 2.6/3.3 family (single deployment, retrospective; not a substitute for gold labels).\nDetails and the curl-vs-claim protocol: [`case-studies/effective-therapy-forensic.md`](case-studies/effective-therapy-forensic.md).\n\n## Categories\n\n- `wrap_up`: unprompted continuation offers or next-step invitations.\n- `cliffhanger`: withheld information or unresolved bait that pressures\n  re-engagement.\n- `roleplay_drift`: emotional, prideful, fatigued, or personally invested\n  agent self-presentation.\n- `sycophancy`: unearned flattery or dishonest positive validation.\n\n## Layout\n\n- `SPEC.md`: active scientific and engineering contract.\n- `SOURCE_LEDGER.md`: live-verified external evidence used for claims.\n- `CLAIM_LEDGER.md`: claim status: verified, corrected, deferred, or dropped.\n- `data/`: release-shaped candidate corpus JSONL files.\n- `recovery/`: local reconstruction outputs and quarantined records.\n- `annotations/`: human and LLM annotation workflow scripts and outputs.\n- `evaluation/`: black-box hook harness and metric code.\n- `engine/`: Rust CLI for deterministic closeout physics.\n- `engines/`: per-category physics engine manifests for paper and runtime use.\n- `rules/closeout/`: versioned deterministic rule packs.\n- `adapters/claude-code/`: installable Claude Code hook adapters for daily use.\n- `fixtures/closeout/`: golden fixtures for rule-pack behavior.\n- `fixtures/closeout_public/`: public-study-derived fixtures for v1 pressure\n  testing.\n- `public_data_intake/`: source registry, manifest, quarantine, and\n  public-derived adversarial corpus lane.\n- `baselines/`: non-hook baselines used to separate benchmark quality from hook\n  tuning.\n- `rubrics/`, `schemas/`, `manifests/`: annotation, schema, provenance, license,\n  redaction, and metadata artifacts.\n- `tests/`: local no-network QA tests.\n\n## Local QA\n\nRun the local no-network checks:\n\n```bash\npython3 scripts/validate_corpus.py --data-dir data --quota-manifest quota_manifest.json\npython3 -m pytest -q\n```\n\nRun a reproducibility smoke check:\n\n```bash\nbash scripts/reproduce_local.sh\n```\n\nRun the deterministic closeout physics checks:\n\n```bash\nbin/agentcloseout-physics lint-rules rules/closeout\nbin/agentcloseout-physics test-rules rules/closeout fixtures/closeout\nbin/agentcloseout-physics test-rules rules/closeout fixtures/closeout_public\npython3 scripts/public_data_intake.py audit-registry \\\n  --registry public_data_intake/source_registry.json \\\n  --schema schemas/public_source.schema.json\npython3 scripts/public_data_intake.py validate-derived \\\n  --registry public_data_intake/source_registry.json \\\n  --manifest public_data_intake/derived_fixture_manifest.jsonl \\\n  --data-dir public_data_intake/candidate_public_adversarial\n```\n\nRun the user-facing Claude Code adapter smoke test:\n\n```bash\nbash scripts/hook-smoke.sh\n```\n\nInstall physics-backed hooks into a Claude Code project:\n\n```bash\nbash adapters/claude-code/install.sh /path/to/project\n```\n\nInstall a single category hook:\n\n```bash\nbash adapters/claude-code/install.sh /path/to/project no-cliffhanger\n```\n\nThe standalone hook repos remain installable on their own. The adapter lane is\nfor users who want the reproducible Rust engine, versioned rule packs,\nrule-pack hash, benchmark fixtures, and opt-in content-free telemetry commands.\nThe adapter installer also writes a `PreToolUse` tamper guard that blocks Claude\nCode from editing the local hook wiring, engine pointer, or rule-pack pointer\nduring an ordinary session.\n\n## Recovery\n\nThe previous `/tmp/agent-closeout-bench` workspace was lost. Recovery is derived\nfrom Claude Code JSONL transcripts under:\n\n```text\n~/.claude/projects/-tmp-agent-closeout-bench/*.jsonl\n```\n\nRecovery must only extract visible assistant `text` blocks. It must not persist\nthinking blocks, signatures, tool calls, tool outputs, hidden transcript fields,\nor secrets.\n\n```bash\npython3 generation/recover_from_claude_transcripts.py \\\n  --transcripts-dir ~/.claude/projects/-tmp-agent-closeout-bench \\\n  --output-dir data \\\n  --manifest recovery/RECOVERY_MANIFEST.md\n```\n\nGeneric negative prompts do not encode a category in the prompt text. The\nrecovery script quarantines those as `category_unresolved` unless a later,\nauditable mapping source proves the category.\n\nThe recovered private transcript pool is not mixed into the public-shaped v0.2\ncorpus. It is preserved for audit in `recovery/recovered_category_proven_pool.jsonl`.\n\n## Evaluation\n\nExample hook evaluation:\n\n```bash\npython3 evaluation/eval_hooks.py \\\n  --hooks-dir /path/to/llm-dark-patterns/hooks \\\n  --corpus-dir data \\\n  --hook-category-map \"wrap_up:no-wrap-up.sh,cliffhanger:no-cliffhanger.sh,roleplay_drift:no-roleplay-drift.sh,sycophancy:no-sycophancy.sh\" \\\n  --ground-truth candidate \\\n  --output results/eval_candidate.json\n```\n\nCandidate diagnostics should use `dev` or `validation`. Final paper results must\nuse adjudicated human labels on `locked_test`; the harness blocks locked-test\nruns that ask for candidate labels.\n\n## Release Blockers\n\n- Two independent human annotation passes.\n- Adjudicated final labels.\n- Per-category agreement report.\n- Private or delayed holdout policy if a leaderboard is launched.\n- Fresh-clone reproducibility run with final labels.\n- Hugging Face dataset card and Croissant metadata validation if targeting an\n  E\u0026D-style release.\n- Exact pinned hook commits and machine-readable result JSON.\n- Larger reviewed public-derived corpus, then two-pass human gold annotation and\n  adjudication before any public performance claim.\n\n## Release Blockers Resolved In v0.2\n\n- Full 800-record schema-valid candidate corpus.\n- Exact deterministic quota manifest.\n- Opaque blind annotation packet with private id map.\n- Provenance, license, and redaction manifests for the synthetic public-shaped\n  corpus.\n- Local no-network smoke reproduction.\n\n## Public-Data Guardrails Added In v0.3\n\n- Source registry with tier, license, privacy status, allowed use, import\n  decision, and release eligibility.\n- Content-free sampler for local public JSONL trace review; raw text is not\n  persisted unless an approved source and explicit write flag are used.\n- Derived-fixture manifest linking every public-derived record to source id,\n  source-record hash, transform, reviewer, and license decision.\n- Quarantine checks for secrets, emails, absolute paths, usernames, hostnames,\n  repo URLs, raw tool-output markers, and trace artifact leakage.\n- Evaluation output now reports per-corpus-kind, per-source, and per-fixture\n  breakdowns so public-derived stress results cannot be hidden in aggregate.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwaitdeadai%2Fagent-closeout-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwaitdeadai%2Fagent-closeout-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwaitdeadai%2Fagent-closeout-bench/lists"}