{"id":51626694,"url":"https://github.com/waitdeadai/faithbench-neg","last_synced_at":"2026-07-13T02:02:19.139Z","repository":{"id":361895020,"uuid":"1255769121","full_name":"waitdeadai/faithbench-neg","owner":"waitdeadai","description":"Per-failure-class adversarial-negatives benchmark + scorer for Lean 4 statement-autoformalization faithfulness (v0 scaffold)","archived":false,"fork":false,"pushed_at":"2026-06-01T17:05:34.000Z","size":65,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-01T19:09:49.513Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/waitdeadai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-01T06:45:57.000Z","updated_at":"2026-06-01T17:09:24.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/waitdeadai/faithbench-neg","commit_stats":null,"previous_names":["waitdeadai/faithbench-neg"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/waitdeadai/faithbench-neg","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waitdeadai%2Ffaithbench-neg","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waitdeadai%2Ffaithbench-neg/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waitdeadai%2Ffaithbench-neg/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waitdeadai%2Ffaithbench-neg/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/waitdeadai","download_url":"https://codeload.github.com/waitdeadai/faithbench-neg/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waitdeadai%2Ffaithbench-neg/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35407393,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-13T02:00:06.543Z","response_time":119,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-13T02:02:14.336Z","updated_at":"2026-07-13T02:02:19.116Z","avatar_url":"https://github.com/waitdeadai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# faithbench-neg\n\n**A held-out, per-failure-class benchmark of plausible-but-_unfaithful_ Lean 4 statement formalizations — plus a scorer that ranks any faithfulness checker by its _per-class_ catch-rate.**\n\nWhen an LLM autoformalizes \"determine all x such that …\" into a Lean statement, the statement can **type-check and still be the wrong theorem**: vacuous, answer-leaking, quantifier-swapped, premise-mistranslated, circular, or off-by-a-domain. Aggregate \"judge accuracy\" numbers hide _which_ of these a checker is blind to. `faithbench-neg` measures exactly that: every negative compiles, and the report is a **per-class catch-rate table**.\n\n\u003e **Honest status (read this first).** This repo is the *machine*: taxonomy, schema, scorer, baseline adapters, and a mutation skeleton — all runnable today. It is **not** yet a benchmark: the seed data is **illustrative placeholder** (`label_status: UNVERIFIED_EXAMPLE`, Lean **not** type-checked). The actual value — a held-out set of human-verified, compiling negatives — is **labeling work that requires Lean 4 + competition-math expertise** and has not been done here. The taxonomy + harness is the reusable asset; the labels are the moat.\n\n## Why this might matter (and the honest caveats)\n\nThis was selected by an adversarial research pass over the 2026 autoformalization frontier, and it survived as a **7/10 — \"a win on execution, not a wide moat.\"** Keeping that honesty up front:\n\n- **The gap is real but narrow.** No existing artifact ships a *named per-failure-class taxonomy with per-class catch-rates*. Closest neighbors and how we differ:\n  - **CriticLeanBench** (arXiv:2507.06181) — ships labeled incorrect formalizations, but with unstructured \"error annotations (where applicable)\", **no per-class taxonomy, no per-class catch-rates**.\n  - **ProofNetVerif** (EMNLP 2025) — human-vs-judge agreement in **aggregate**, not per failure class.\n  - **FormalEvolve** (arXiv:2603.19828) — AST rewriting for autoformalization **search diversity**, not adversarial-negative generation by failure class.\n- **Incumbent-absorption risk is the #1 kill-shot.** M-A-P or EPFL own labeled negatives + annotation pipelines and could fold per-class reporting into a dataset-card update quickly. The moat is being *first* and being the *canonical held-out suite people cite*.\n- **Labeling is slow, scarce, and IS the product.** A rushed/sloppy set has near-zero value.\n- **Short half-life.** Once public and scraped, usefulness may be ~one model generation. Anchor seeds on **fresh 2025–2026 competitions** to resist contamination.\n\nIf you can't commit real Lean+math labeling, the honest move is to **contribute the taxonomy upstream** rather than ship a thin set.\n\n## Install / run\n\nNo dependencies for the core (Python ≥3.10). `pytest` for tests.\n\n```bash\n# from the repo root\npython -m faithbench.cli validate data/seed\npython -m faithbench.cli score    data/seed --checkers type_check,mock_judge\npython -m faithbench.cli mutate    data/seed/example-0001.json\npython -m pytest -q\n```\n\n### The headline the scorer is built to produce\n\n`type_check` (compile-only) catches **0%** on every class — by construction, since all negatives compile. A real `llm_judge` (stub here) will catch some classes far better than others. That **per-class divergence** — not an aggregate number — is the deliverable. `v0` success = a table where at least one baseline is near-0 on at least one class.\n\n## What's here\n\n| Piece | State |\n|---|---|\n| `faithbench/core.py` | frozen taxonomy (v0.1.0) + item model + validation | ✅ runnable |\n| `faithbench/scoring.py` | per-class catch-rate + cry-wolf (FP) + table | ✅ runnable |\n| `faithbench/adapters/` | `type_check` + `mock_judge` (toy) run; `llm_judge`, `beq_plus` are **honest stubs** (raise, never fake) | ⚠️ partial |\n| `faithbench/mutate.py` | textual-placeholder skeleton; real version needs Lean AST via LeanInteract | ⚠️ skeleton |\n| `data/seed/` | 2 **UNVERIFIED_EXAMPLE** items — placeholders, Lean not type-checked | ❌ not real data |\n\n## Roadmap to a real `v0`\n\n1. Replace seed with **~50 human-verified items** (target 150–250 at v1), each: a fresh-competition NL problem (answer-determination class), one faithful Lean 4 statement that **type-checks against Mathlib**, and one labeled negative per applicable class (each must compile).\n2. Wire `llm_judge` (real API) and `beq_plus` (LeanInteract) adapters.\n3. Wire the mutation generator to a Lean AST + a type-check gate so humans only label compiling survivors.\n4. Publish the **per-class catch-rate leaderboard** — the thing no existing benchmark reports.\n\n## Deterministic lint suite (`faithbench.lint`) — and its honest ceiling\n\nA suite of **single-purpose, deterministic, fail-open** linters in the bash-hook\nidiom. No LLM, no network, no randomness — same input → same verdict. They are a\nfast **pre-filter**, not a faithfulness oracle.\n\n```bash\npython -m faithbench lint 'theorem t (x : ℝ) : True := by sorry'   # -\u003e vacuous\nbin/faithlint.sh '\u003cstatement\u003e'            # hook form; fail-open (exit 0)\nFAITHLINT_STRICT=1 bin/faithlint.sh '\u003cstatement\u003e'   # CI mode; exit 2 on a flaw\npython -m faithbench score data/seed --checkers type_check,faithlint,mock_judge\n```\n\n**What determinism can and cannot reach (measured on the seed, not asserted):**\n\n| | reference-free (lone statement / hook) | + reference-diff (gold known) | needs semantic judge |\n|---|---|---|---|\n| `vacuous` | ✅ goal is `True` / `False` hyp | ✅ | |\n| `conclusion_as_axiom` | ✅ hypothesis == goal | ✅ | |\n| `quantifier_swapped` | ❌ | ✅ ∀/∃ diff vs gold | |\n| `premise_mistranslated` | ❌ | ✅ binder-count diff | |\n| `domain_type_mismatch` | ❌ | ✅ binder-type diff | |\n| `answer_leaking` | ❌ | ❌ | **only here** |\n\nSo: **2 of 6 classes** are catchable from a lone statement (the hook case), **5 of 6**\nonce a gold reference exists (the autoformalizer-regression case), and `answer_leaking`\nis irreducibly semantic — it needs the LLM judge that the literature puts at a ~45%\nceiling. The deterministic suite's job is to catch the cheap cases at zero cost and\n**escalate the rest**, never to pretend it judged them. Two honesty constraints baked\nin: linters are **fail-open** (any error → exit 0), and the reference-free hook flags\nonly **high-precision** patterns (0% false-positive on the seed's faithful statements).\n\n\u003e Real-world caveat (from the adversarial review): the reference-free hook needs only\n\u003e a lone Lean statement, but the higher-recall reference-diff checks need a paired gold\n\u003e statement, which mathlib/blueprint projects don't systematically carry. The suite is\n\u003e most useful as a **regression gate on an autoformalizer that already has gold targets**,\n\u003e and as a cheap pre-filter elsewhere.\n\n## Universal: domains (not math-only)\n\nfaithbench is a **domain-pluggable framework**. The item model, scorer, and CLI\nare domain-agnostic; everything domain-specific lives behind one interface\n(`faithbench/domains/`): a failure-class taxonomy, a deterministic\n`structural(artifact, context)` \"cheap gate\", and a `reference_diff` against a\ngold artifact. Adding a domain = implement that interface and `register()`.\n\n```bash\npython -m faithbench domains                      # list registered domains\npython -m faithbench score data/seed              # lean_math (auto-detected)\npython -m faithbench score data/seed_tool_call    # tool_call\npython -m faithbench lint --domain tool_call --context tools.json '{\"tool\":\"get_weather\",\"arguments\":{\"unit\":\"kelvin\"}}'\n```\n\nThree domains ship, and they make the central point measurable: **how much of\nfaithfulness is deterministically checkable depends entirely on how strong the\ndomain's cheap gate is.**\n\n| domain | cheap gate | reference-free reach | irreducibly-semantic class | data |\n|---|---|---|---|---|\n| `lean_math` | Lean type-check | **2 / 6** classes | `answer_leaking` | needs human Lean+math labeling |\n| `tool_call` | JSON-schema validity (pure code) | **5 / 6** classes | `intent_drift` | **self-verifying** (machine-decidable) |\n| `code` | parse + run-against-tests (pure code) | **5 / 6** classes | `intent_drift` | **self-verifying** (by execution) |\n\n**`code` — the one that matters for LLM coding.** Artifact = a Python function;\ncheap gate = parse + execute against the spec's tests. It catches `syntax_error`,\n`wrong_signature`, `crashes`, `contract_violation`, and `forbidden_construct`\ndeterministically, and is blind to `intent_drift` — **code that passes every\nprovided test but is still wrong** (overfit / Goodhart), the exact failure that\nships bugs from green CI. That class is recoverable only with held-out probes\nagainst a gold oracle (`reference_diff`) or a judge. Runs as a fail-open hook:\n`FAITHLINT_DOMAIN=code FAITHLINT_CONTEXT=spec.json bin/faithlint.sh '\u003csource\u003e'`\n(in-process exec — trusted data only; sandbox untrusted candidates).\n\nThe `tool_call` domain (agent / function-call faithfulness) is the cleaner proof:\nbecause schema validity is decidable by code, its structural negatives need **no\nhuman labeling** — the seed is `machine_verified`. Every domain still has exactly\none class that is \"valid but means the wrong thing\" and needs a semantic judge —\n`answer_leaking` for math, `intent_drift` for tool calls. That is the honest,\nuniversal shape: **valid ≠ faithful; the deterministic suite catches the cheap\nviolations for free and escalates the one semantic class.**\n\n## Scaling (self-verifying data) + the semantic-judge layer\n\n**Data is generated and self-verified, not hand-labeled** — for the two\nmachine-decidable domains. `scripts/gen_code_seed.py` and\n`scripts/gen_tool_call_seed.py` synthesize items and then check each one against\nthe real domain (structural must catch the 5 structural classes, miss the\nsemantic one; `reference_diff` must catch the semantic one; faithful must stay\nclean) — a mislabeled item aborts the run. Current self-verified n: **code = 11\nper class, tool_call = 9 per class** (`lean_math` stays at n=1: it needs scarce\nhuman Lean labeling).\n\n**The judge layer closes the deterministic blind spot.** `LLMJudge` is wired to\nany model via env `FAITHBENCH_JUDGE_CMD` (a CLI reading the prompt on stdin) or an\ninjected callable; with neither it **raises rather than fabricates**. Layered\nresult on the `code` seed (judge here is a mock oracle in tests — real-model\naccuracy is the open empirical question, and the literature puts judges well\nbelow perfect):\n\n```\nfailure class        faithlint   llm_judge    n\nsyntax_error              100%        100%    11\nwrong_signature           100%        100%    11\ncrashes                   100%        100%    11\ncontract_violation        100%        100%    11\nforbidden_construct       100%        100%    11\nintent_drift                0%        100%    11   \u003c- structural BLIND; judge recovers it\nAGGREGATE                  83%        100%\ncry-wolf (FP)               0%          0%\n```\n\nThat is the whole thesis in one table: the deterministic gate catches the cheap\nclasses for free; exactly one semantic class per domain needs the judge.\n\nLicense: Apache-2.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwaitdeadai%2Ffaithbench-neg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwaitdeadai%2Ffaithbench-neg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwaitdeadai%2Ffaithbench-neg/lists"}