{"id":51430381,"url":"https://github.com/sattyamjjain/proofloop","last_synced_at":"2026-07-05T03:30:24.314Z","repository":{"id":338398824,"uuid":"1157292437","full_name":"sattyamjjain/proofloop","owner":"sattyamjjain","description":"Universal quality evaluation plugin for Claude Code — 7-dimension scoring (correctness, completeness, adherence, efficiency, safety), configurable rubrics, threshold blocking, auto-hooks \u0026 /judge command.","archived":false,"fork":false,"pushed_at":"2026-06-21T14:38:43.000Z","size":872,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-21T15:26:41.987Z","etag":null,"topics":["ai-agents","ai-evaluation","claude-code","code-quality","developer-tools","evaluation","llm-as-judge","plugin","python","quality","rubric","scoring"],"latest_commit_sha":null,"homepage":"https://github.com/sattyamjjain/verdict#readme","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sattyamjjain.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":"ROADMAP_2026.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-13T16:48:51.000Z","updated_at":"2026-06-21T14:38:44.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/sattyamjjain/proofloop","commit_stats":null,"previous_names":["sattyamjjain/verdict","sattyamjjain/proofloop"],"tags_count":21,"template":false,"template_full_name":null,"purl":"pkg:github/sattyamjjain/proofloop","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sattyamjjain%2Fproofloop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sattyamjjain%2Fproofloop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sattyamjjain%2Fproofloop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sattyamjjain%2Fproofloop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sattyamjjain","download_url":"https://codeload.github.com/sattyamjjain/proofloop/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sattyamjjain%2Fproofloop/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35142824,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-05T02:00:06.290Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","ai-evaluation","claude-code","code-quality","developer-tools","evaluation","llm-as-judge","plugin","python","quality","rubric","scoring"],"created_at":"2026-07-05T03:30:23.497Z","updated_at":"2026-07-05T03:30:24.301Z","avatar_url":"https://github.com/sattyamjjain.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Proofloop\n\n[![CI](https://github.com/sattyamjjain/proofloop/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/sattyamjjain/proofloop/actions/workflows/ci.yml)\n[![Release](https://img.shields.io/github/v/release/sattyamjjain/proofloop)](https://github.com/sattyamjjain/proofloop/releases/latest)\n[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)\n\n**Auto-grade every Claude Code and Cowork skill execution on seven\ndimensions. No LLM call. No config. Just a scorecard.**\n\nEvery other evaluation tool in this space (Braintrust, Langfuse,\nPhoenix, Helicone, Promptfoo, DeepEval, Ragas, LangSmith, Opik)\nrequires a second LLM to grade the first. Proofloop runs offline regex\nheuristics inside the editor. Zero ongoing cost, zero API key, zero\nnetwork call — it ships as a Claude Code plugin that fires on every\n`Stop` and `SubagentStop` event.\n\n---\n\n## See it\n\nEvery execution gets an evidence-based scorecard — rendered in your\nterminal in milliseconds, with no API call and no account:\n\n```text\n╔════════════════════════════════════════════════════════════════════════════╗\n║ PROOFLOOP SCORECARD -- feature-dev                                         ║\n║ Grade: A- (Very Good)  |  Composite: 8.75/10.0  |  2026-06-13T18:05:22Z    ║\n╠════════════════════════════════════════════════════════════════════════════╣\n║ Correctness    ██████████  10.0/10 (w=0.25) → No errors; verification…     ║\n║ Completeness   █████████░   9.0/10 (w=0.20) → Short transcript — may …     ║\n║ Adherence      ████████░░   8.0/10 (w=0.15) → Rubric available (compl…     ║\n║ Actionability  ████████░░   8.0/10 (w=0.15) → Output appears ready to…     ║\n║ Efficiency     ████████░░   8.0/10 (w=0.10) → Reasonable tool usage (…     ║\n║ Safety         ██████████  10.0/10 (w=0.10) → No safety concerns dete…     ║\n║ Consistency    █████░░░░░   5.0/10 (w=0.05) → No prior history for co…     ║\n╠════════════════════════════════════════════════════════════════════════════╣\n║ Summary: Good Feature Dev (A-) -- strong correctness, consistency could i… ║\n║                                                                            ║\n║ RECOMMENDATIONS:                                                           ║\n║   * Compare with prior executions and maintain quality baselines           ║\n╚════════════════════════════════════════════════════════════════════════════╝\n```\n\nThat `pytest` receipt in the run is what earns correctness **10/10**:\nwithout an executed check *anywhere* in the transcript, the top mark is\nwithheld — **no pass without proof.** It is no rubber stamp either —\nconsistency is held at `5` until there's a history to compare against.\n\n---\n\n## Install\n\n```shell\n/plugin marketplace add sattyamjjain/proofloop\n/plugin install proofloop@proofloop\n```\n\nThat's it. The next skill or subagent execution that matches an\n`always` entry in `judge-config.json` is scored automatically; the\nscorecard JSON lands in `skills/judge/scores/` and `/scorecard`\nrenders the trend line.\n\nWorks on Claude Code and Claude Cowork. For the Cowork plugin-loading\nworkaround (GH #39400), see [INSTALL-COWORK.md](INSTALL-COWORK.md).\n\n---\n\n## Why Proofloop\n\n|                            | Proofloop | Braintrust / Langfuse / Phoenix | Promptfoo / DeepEval | LangSmith / Opik |\n| -------------------------- | :-----: | :-----------------------------: | :------------------: | :--------------: |\n| Runs inside Claude Code    | ✓       | ✗                               | ✗                    | ✗                |\n| Offline (no LLM call)      | ✓       | ✗                               | optional             | ✗                |\n| Zero config for first score| ✓       | ✗                               | ✗                    | ✗                |\n| Per-domain rubrics         | ✓ (11)  | via code                        | via YAML             | via code         |\n| Cross-ecosystem transcripts| ✓       | traces only                     | provider-flex        | LangChain-first  |\n| Pip install / deps         | stdlib  | SDK + network                   | SDK                  | SDK + account    |\n| Per-rubric weight override | ✓       | ✗                               | ✗                    | ✗                |\n| Git-diff-aware delta       | ✓       | ✗                               | ✗                    | limited          |\n\n---\n\n## Features\n\n- **Dual-mode operation.** Automatic via hooks (`Stop`,\n  `SubagentStop`, `StopFailure`) or manual via `/judge`.\n- **Dual-platform.** Claude Code and Claude Cowork.\n- **7-dimension weighted scoring.** Correctness, completeness,\n  adherence, actionability, efficiency, safety, consistency — weights\n  configurable globally or per rubric via a sidecar `.weights.json`.\n- **Plugin-scope adapters.** Score transcripts from Claude Code,\n  Cowork, Codex, Cursor, Continue, and any OpenAI-compatible JSON\n  shape with a single `--adapter` flag.\n- **11 domain rubrics.** code-review, security, devops, data-analysis,\n  frontend-design, testing, documentation, content-writing, research,\n  default, custom-template. Per-rubric weight overrides via a sidecar\n  `\u003crubric\u003e.weights.json`.\n- **Model-aware efficiency.** Opus 4.7's new tokenizer (~35% more\n  tokens) doesn't silently penalise its longer outputs — length\n  thresholds scale by a per-model baseline.\n- **Diff-aware delta.** `/judge --against HEAD~1` renders a side-by-\n  side delta and exits non-zero on composite regression so CI can gate\n  on it.\n- **Local HTML dashboard.** `scripts/studio.py` emits a single-file\n  HTML report with radar charts + trend lines — no server, no build\n  step.\n- **Benchmark regression gate.** Curated transcript corpus plus\n  `scripts/benchmark_pack.py` asserts no heuristic drift across\n  releases. Wired into CI.\n- **Benchmark hygiene lint (ABA-anchored).** `scripts/bench_lint.py`\n  audits the regression-gate manifest itself for the four issue\n  classes described in [arXiv:2605.26079](https://arxiv.org/abs/2605.26079)\n  (Wang et al. 2026, *Automated Benchmark Auditing for AI Agents\n  and Large Language Models*). Surfaces a `bench_hygiene_score`,\n  text/JSON/SARIF output, and a `--lint` pre-flight gate on the\n  pack so a suspect corpus is caught before its scores ship. See\n  [§Benchmark hygiene lint](#benchmark-hygiene-lint-aba-anchored)\n  below.\n- **Same-family judge guard.** When the opt-in LLM second opinion is\n  enabled, Proofloop checks whether the judge model shares a vendor\n  family with the model that produced the transcript. A match sets\n  `self_preference_risk: true` and warns on the console (Claude judging\n  Claude inflates scores); a configured cross-family judge is\n  auto-preferred. See [§Same-family judge guard](#same-family-judge-guard)\n  below.\n- **Sycophancy signal.** Offline heuristic that flags when the\n  assistant abandons a correct answer under user pushback (\"are you\n  sure? I think it's X\") by capitulating without fresh reasoning — a\n  sycophantic flip docks the composite, while a *correct* concession\n  backed by re-derivation is not penalised. No LLM call. See\n  [§Sycophancy signal](#sycophancy-signal) below.\n- **Stdlib only.** Python 3.9+, no third-party packages, installs\n  instantly with zero supply-chain risk.\n\n\u003e **v1.x → v2.0.0 migration.** v2.0.0 trimmed Proofloop to its plugin\n\u003e scope per the 2026-05-03 v4.3 reset: 16 frontier-lab eval-bench\n\u003e rubrics, 6 cross-ecosystem adapters, and 7 bench-eval scripts were\n\u003e removed. If you depended on `swe-bench-pro`, `terminal-bench`,\n\u003e `clinical-agentic-workflow`, `eu-ai-act-audit-trail`,\n\u003e `tool-output-rewrite`, etc., pin to `v1.4.2` or fork. See\n\u003e [`CHANGELOG.md`](CHANGELOG.md) §[2.0.0] and\n\u003e [`CLAUDE.md`](CLAUDE.md) §v4.3 Scope Contract.\n\n---\n\n## Quick start\n\n### Automatic mode\n\nSkills on the `auto_judge.always` allowlist are scored without user\nintervention. The `Stop` hook emits:\n\n```\nProofloop: code-review → 8.7/10 (A-). Solid execution with minor areas for improvement.\n```\n\n### Manual mode\n\n```\n/judge                            # Score the last execution\n/judge --rubric security          # Force a specific rubric\n/judge --adapter codex --model claude-opus-4-7\n/judge --against HEAD~1           # Delta vs. previous run\n/scorecard                        # Trend view across runs\n/benchmark code-review            # Delta vs. reference standards\n/judge-config                     # View / update allowlist + threshold\n```\n\n### Cross-ecosystem\n\n```shell\n# Codex CLI session\npython3 skills/judge/scripts/score.py \\\n  --skill code-review \\\n  --transcript ~/.codex/sessions/latest.json \\\n  --rubric-dir skills/judge/rubrics \\\n  --scores-dir skills/judge/scores \\\n  --adapter codex\n\n# OpenAI-compatible (Cursor / Continue)\npython3 skills/judge/scripts/score.py --adapter openai-compatible ...\n```\n\n### Studio dashboard\n\n```shell\npython3 skills/judge/scripts/studio.py \\\n  --scores-dir skills/judge/scores \\\n  --output proofloop-studio.html\nopen proofloop-studio.html\n```\n\n### Explain a scorecard for a PR comment\n\n```shell\npython3 skills/judge/scripts/explain.py \\\n  --scorecard skills/judge/scores/code-review_2026-04-25.json \\\n  --format md \\\n  --out /tmp/pr-comment.md\n```\n\n`--format json` emits the same data under a stable\n`format_version: \"explain.v1\"` schema. See\n[`SKILL-judge-explain.md`](skills/judge/SKILL-judge-explain.md).\n\n---\n\n## Gate your CI on it\n\nProofloop ships as a GitHub Action, so the same offline scorer can fail a\npull request when an agent's work drops below your bar — no API key, no\naccount:\n\n```yaml\n- uses: sattyamjjain/proofloop@v3.1.0\n  with:\n    transcript: ./agent-run.jsonl   # the agent/skill transcript to score\n    skill: code-review              # selects the rubric\n    threshold: \"7.0\"                # fail the job below 7.0; omit to report only\n    # adapter: codex                # claude_code | codex | openai_compatible | cowork\n```\n\nIt exposes `composite` and `grade` as step outputs and prints a one-line\nsummary to the job log. The repo dogfoods this pattern on its own pull\nrequests via [`.github/workflows/self-score.yml`](.github/workflows/self-score.yml).\n\n---\n\n## Scoring system\n\n7 weighted dimensions summing to 1.0. Weights live in\n`judge-config.json.scoring.dimensions` and can be overridden per\nrubric via a `\u003crubric\u003e.weights.json` sidecar.\n\n| Dimension       | Default | Signal                                           |\n| --------------- | :-----: | ------------------------------------------------ |\n| Correctness     | 0.25    | Error / hallucination patterns, **unverified-success claims** |\n| Completeness    | 0.20    | TODO/FIXME/HACK density (docstring-scoped)       |\n| Adherence       | 0.15    | Deviation keywords vs. rubric criteria           |\n| Actionability   | 0.15    | Code fences + file actions − placeholders        |\n| Efficiency      | 0.10    | Tool-call density, retries, length (model-aware) |\n| Safety          | 0.10    | Destructive commands, exposed creds, **least-privilege over-scope** (context-aware) |\n| Consistency     | 0.05    | Variance vs. historical scores                   |\n\nGrades: A+ ≥ 9.5, A ≥ 9.0, A− ≥ 8.5, B+ ≥ 8.0, B ≥ 7.5, B− ≥ 7.0,\nC+ ≥ 6.5, C ≥ 6.0, C− ≥ 5.5, D ≥ 4.0, F otherwise.\n\n**Least-privilege sub-check (under Safety).** The safety dimension also\nscores generated agent code for least-privilege tool/skill scoping —\nthe CVE-class root cause behind omnibus free-form tools and over-scoped\nMCP servers. Offline and heuristic (no LLM), it flags a **wildcard\n(`*`/all) grant**, a **write/delete/admin scope** beyond read-only use,\nand an **omnibus free-form tool** that dispatches arbitrary\ncommand/code input at runtime (the single most common pattern). Each\nfinding docks safety and surfaces the offending tool plus a one-line\nremediation in the scorecard's top-level `least_privilege` array and\nthe safety justification. It is a sub-check, **not** an 8th dimension —\nthe 7-dimension contract is preserved. (Detecting the *absence* of a\ndeclaration is left to a manifest validator, since inferring it from a\nflat transcript false-positives on ordinary tool-use logs.)\n\n**Unverified-success / cheap-tier reward-hacking (under Correctness).**\nThe cheapest, most reliable fabricated-success tell is a trajectory\nthat *claims* a check passed — \"all tests pass\", \"build succeeded\",\n\"verified working\" — but carries no **receipt**: no evidence a check\nwas actually executed (a runner invocation, a test count, an exit\ncode). Claiming a pass without running it is fabricated success, a\ncorrectness/honesty failure, so `detect_unverified_success` docks the\n**correctness** dimension and adds a red flag — the same dual treatment\nProofloop already gives a hallucinated fact. The offending claim and a\none-line remediation surface in a top-level `unverified_success` array.\nConfigurable via `judge-config.json.unverified_success`\n(`enabled` / `correctness_dock` / `red_flag`).\n\nThe tiering is deliberate: this **cheap heuristic runs on every\ntrajectory**; it makes no embedding-probe call and no frontier-judge\ncall. Escalation to a model judge is the *separate*, opt-in\n`llm_second_opinion` tier you sample (off by default) — never a\nmandatory tier here. It is a correctness signal, **not** a new\n`reward_hacking` dimension (the 7-dimension contract holds; a\ntrajectory-grading reward-hacking *benchmark* remains out of scope per\nthe v4.3 reset). Anchored on the \"cheap reward-hacking detection\" idea\n([arXiv:2606.08893](https://arxiv.org/abs/2606.08893)) — heuristics on\nevery span, judge only on a sample. Honest limit: a determined agent\ncould fabricate a receipt too, which is why the model-judge tier stays\navailable as opt-in escalation.\n\nPer-rubric overrides: drop a sibling `\u003crubric\u003e.weights.json` next to\n`\u003crubric\u003e.md`. Shipped example: `security.weights.json` weights safety\nat 0.35, correctness at 0.20.\n\n---\n\n## Architecture\n\n```\n.claude-plugin/\n  plugin.json              # Plugin manifest\n  marketplace.json         # Marketplace listing\nskills/judge/\n  SKILL.md                 # Core skill definition\n  SKILL-judge-explain.md   # /judge --explain output schema\n  scripts/\n    score.py               # Scoring engine\n    report.py              # Scorecard reporter\n    benchmark.py           # Benchmark comparator\n    against.py             # /judge --against delta\n    compare.py             # /compare two-file diff with regression narrative\n    explain.py             # /judge --explain Markdown / JSON / HTML-printable\n    studio.py              # Local HTML dashboard\n    watch.py               # /judge --watch live re-scoring daemon\n    cost_estimator.py      # Per-scorecard USD cost estimator\n    hook_lint.py           # PostToolUse hook static analyzer\n  adapters/\n    claude_code.py         # Native JSONL (default)\n    cowork.py              # Claude Cowork sessions\n    codex.py               # OpenAI Codex CLI\n    openai_compatible.py   # Cursor / Continue / generic\n  analyzers/\n    llm_judge.py           # Opt-in second-opinion (Claude API + cache_control)\n  rubrics/                 # 11 plugin-domain rubrics + sidecar weights\n  scores/                  # Persisted JSON scorecards\n  references/              # Scoring methodology + benchmark standards\nagents/judge-agent.md\nhooks/\n  hooks.json               # Stop / SubagentStop / StopFailure\n  common.sh, judge-on-stop.sh, judge-on-subagent-stop.sh, judge-on-stop-failure.sh\ncommands/                  # /judge, /scorecard, /benchmark, /judge-config, /against, /compare\nscripts/\n  validate_marketplace.py        # Schema validator\n  install_rubric.py              # Fetch + validate community rubrics\n  benchmark_pack.py              # Regression gate for CI (now with --lint pre-flight)\n  bench_lint.py                  # ABA-anchored task-hygiene lint (arXiv:2605.26079)\n  sandbox_caps_check.py          # CLAUDE_SANDBOX_CAPS declaration check (CI)\n  check_readme_release_anchor.py # CHANGELOG ↔ README anchor forcing-function (CI)\nbenchmarks/\n  manifest.json + corpus/  # Regression-gate fixtures (NOT a public eval bench — see benchmarks/README.md)\n.github/workflows/\n  ci.yml                   # Tests + validators + benchmark gate + shellcheck\njudge-config.json\nCLAUDE.md, CHANGELOG.md, INSTALL-COWORK.md, README.md, LICENSE\n```\n\n---\n\n## Configuration reference\n\nAll in `judge-config.json`:\n\n### `auto_judge`\n\n| Key                 | Type     | Description                                              |\n| ------------------- | -------- | -------------------------------------------------------- |\n| `enabled`           | boolean  | Master switch for automatic judging.                     |\n| `always`            | string[] | Skill names auto-judged without further configuration.   |\n| `never`             | string[] | Skill names never auto-judged.                           |\n| `threshold`         | number   | Minimum composite to pass (0–10).                        |\n| `block_on_critical` | boolean  | Exit-2 the hook (block workflow) when below threshold.   |\n\n### `manual_judge`\n\n| Key              | Type    | Description                               |\n| ---------------- | ------- | ----------------------------------------- |\n| `default_rubric` | string  | Fallback rubric name.                     |\n| `verbose`        | boolean | Show per-dimension justifications.        |\n| `save_scores`    | boolean | Persist scorecard JSON.                   |\n\n### `scoring.dimensions`\n\nMap of dimension → weight. Weights must sum to 1.0 — `load_config`\nrejects otherwise (stderr warning + default fallback).\n\n### `tokenizer_baselines`\n\nPer-model multipliers for the efficiency analyser's length thresholds.\nShips with `claude-opus-4-7: 1.35`, others at 1.0. Override per model\nor add new ones — the `default` key catches unknown models.\n\n---\n\n## Requirements\n\n- **Python 3.9+** — stdlib only (no pip deps).\n- **jq** — `brew install jq` / `apt-get install jq`.\n- **bc** — `brew install bc` / `apt-get install bc`.\n\n## Running tests\n\n```shell\npython3 -m unittest discover tests/ -v        # full suite\npython3 scripts/validate_marketplace.py       # schema check\npython3 scripts/benchmark_pack.py             # regression gate\npython3 scripts/benchmark_pack.py --lint      # regression + ABA hygiene pre-flight\npython3 scripts/bench_lint.py                 # ABA hygiene lint alone\nshellcheck hooks/*.sh                         # hook-script lint\n```\n\n## Benchmark hygiene lint (ABA-anchored)\n\n`scripts/bench_lint.py` audits the regression-gate manifest itself\nbefore any score is consumed. It adapts the four issue classes from\n[arXiv:2605.26079](https://arxiv.org/abs/2605.26079) (Wang et al.\n2026, *Automated Benchmark Auditing for AI Agents and Large\nLanguage Models*, v1 2026-05-25) to Proofloop's transcript-regression\nshape. ABA found that **25.7% of tasks across 168 benchmarks**\ncontained one of these issues, and that removing them moved model\nscores by ~9.6–9.9% — i.e., suspect bench plumbing changes the\nverdict. The lint catches the same shape of issue in our own pack\n*before* CI greenwashes a broken corpus.\n\n| Rule | Class | Triggers on |\n|------|-------|-------------|\n| **VBL001** | SpecificationGap | missing `name`/`skill`, or no `expected_*` bound declared (case asserts nothing) |\n| **VBL002** | EnvironmentCoupling | absolute transcript path, escapes manifest dir via `..`, transcript missing, or `adapter` ↔ file-suffix mismatch |\n| **VBL003** | BrittleGrading | single-point composite/grade/dim bounds (`min == max`), or composite range narrower than 0.5 |\n| **VBL004** | MissingGroundTruth | transcript is 0-bytes or contains only blank lines |\n\n```shell\n# Standalone (text by default)\npython3 scripts/bench_lint.py\n\n# JSON for piping into other tools\npython3 scripts/bench_lint.py --json | jq '.bench_hygiene_score'\n\n# SARIF v2.1.0 for CI surfacing (GitHub code-scanning, etc.)\npython3 scripts/bench_lint.py --sarif bench_lint.sarif --quiet\n\n# Custom threshold (default 0.85)\npython3 scripts/bench_lint.py --threshold 0.95\n\n# Wired into the regression gate as a pre-flight (aborts before any\n# score is consumed if the corpus is suspect)\npython3 scripts/benchmark_pack.py --lint --sarif bench_lint.sarif\n```\n\nAggregate `bench_hygiene_score = 1 - flagged_cases / total_cases`.\nExit codes: **0** above threshold, **1** below, **2** on IO / arg\nfailure. Offline-only, stdlib-only — no LLM call (the heuristic is\nthe moat; defaulting to LLM judging would push the trust boundary\ninto a token-billed black box, which is exactly what we lint\nagainst).\n\n**Scope honesty.** Proofloop's \"benchmark pack\" scores transcripts\nagainst expected score bounds, not tasks against ground-truth\noutputs (`benchmarks/manifest.json` is explicitly *not* a public\neval bench — see `benchmarks/README.md`). The four ABA classes\ntherefore apply *by analogy*, mapped to the regression-gate shape;\nthe rule descriptions above and the lint's own help text make this\nadaptation explicit. If Proofloop ever grows a true task benchmark,\nthe rules will need a literal pass — tracked as **O17** in\n`CHANGELOG.md`.\n\n## Verifier-collapse detector\n\n`_analyze_consistency` already compares each run against the rolling\nhistory of recent scorecards for the same skill — but the existing\nlow-variance branch *rewards* `std_dev \u003c= 0.8` with a `+1` \"highly\nconsistent\" bonus. That branch silently rewards the failure mode\nwhere a judge has flatlined at the top of the scale.\n\nThe verifier-collapse detector (`v2.0.4+`, offline, stdlib-only)\ncomposes with that path. Over the rolling window of recent\nscorecards for the same skill, it flags `verifier_collapse=true`\nwhen **both**:\n\n| Condition | Default |\n|-----------|---------|\n| fraction of composites `\u003e= top_threshold` exceeds `top_bucket_fraction` | `\u003e= 8.5` for `\u003e= 95%` of cards |\n| `std_dev` of those composites falls below `max_std_dev` | `\u003c 0.3` (tighter than the existing 0.8 \"highly consistent\" cutoff) |\n| at least `min_samples` of the last `window` cards available | `5 of 10` |\n\nOn a hit, the consistency dimension is docked by `consistency_dock`\n(default `3`) — wide enough to net out the existing `+1` low-variance\nbonus that the same data would otherwise trigger. The boolean is\nmirrored at the scorecard top level for one-jq-query CI consumption,\nsurfaced in the `/judge --explain` Markdown as a `⚠️ Verifier\ncollapse detected` callout above the dimension table, and added to\nthe `explain.v1` JSON payload at both top-level (`verifier_collapse`)\nand per-dimension (`dimensions.consistency.verifier_collapse` +\n`verifier_collapse_reason` + `verifier_collapse_stats`) levels.\n\n```shell\n# Disable the detector entirely:\njq '.verifier_collapse.enabled = false' judge-config.json \u003e tmp \u0026\u0026 mv tmp judge-config.json\n\n# Demote the ship-gate from blocking to a stderr warning (default):\njq '.verifier_collapse.gate_mode = \"warn\"' judge-config.json \u003e tmp \u0026\u0026 mv tmp judge-config.json\n\n# Block the Stop hook (exit 2) when a collapse is detected:\njq '.verifier_collapse.gate_mode = \"fail\"' judge-config.json \u003e tmp \u0026\u0026 mv tmp judge-config.json\n```\n\nThe `judge-on-stop.sh` ship-gate honours `gate_mode` ∈\n`{\"warn\", \"fail\", \"off\"}` and emits\n`Proofloop {WARNING,BLOCKED}: verifier collapse detected for \u003cskill\u003e`\non stderr.\n\n**Anchor.** The signal is derived from Proofloop's own consistency\ndimension plus the **Soft-SVeRL** project anchor — distinct from\nvariance-based consistency, not a sibling-benchmark analogy. The\nheuristic is offline statistics, default-on, and never calls an\nLLM; this complements (does not replace) the opt-in LLM\nsecond-opinion analyzer.\n\n## Same-family judge guard\n\nThe opt-in LLM second opinion\n(`judge-config.json.llm_second_opinion.enabled = true`) is only as\ntrustworthy as the judge. An LLM judge systematically over-scores\noutputs from its own model family, so when the second-opinion judge\nshares a family with the model that produced the transcript, the score\nis biased upward and Proofloop says so.\n\nBefore the call, `same_family_guard` (in\n`skills/judge/analyzers/llm_judge.py`) buckets the executing model\n(from `score.detect_model_from_transcript`) and the configured judge\nmodel into vendor families (`anthropic` / `openai` / `google` /\n`meta`). On a same-family match it:\n\n1. sets `self_preference_risk: true` on the scorecard (mirrored in the\n   `same_family_guard` object with both families), and emits\n   `Proofloop WARNING: judge and executing model share a family …` on\n   stderr; and\n2. if `llm_second_opinion.alternate_judge_models` names a cross-family\n   judge, **auto-prefers** it for the call (reachable via the injected\n   client / proxy path the analyzer already documents).\n\nIn the stock configuration the second opinion is Claude judging\nClaude, so the guard fires on every enabled run — that is the honest\nsignal, not a bug. The guard is offline and stdlib-only; it asserts a\nclash only when both families are recognised (an unknown model never\nfabricates a risk).\n\n**Design rationale.** Self-preference in LLM-as-judge is measured, not\nhypothetical: pairwise judges favour their own family by double-digit\nwin-rate margins ([arXiv:2306.05685](https://arxiv.org/abs/2306.05685),\nMT-Bench — GPT-4 +10pp, Claude-v1 +25pp self-win-rate), and merely\nre-labelling an output as the judge's own work swings its scores by\n+23–93pp ([arXiv:2606.05976](https://arxiv.org/abs/2606.05976),\nrole-relabel). Proofloop keeps the judge framed as a third-party\n\"second-opinion judge\" (never first-person), flags the residual\nsame-family risk, and prefers a cross-family judge when one is\nconfigured.\n\n## Sycophancy signal\n\nA useful-but-wrong assistant agrees with you. The sycophancy signal\n(`detect_sycophancy` in `skills/judge/scripts/score.py`, `v2.0.6+`)\nscores **agreement-drift** across conversation turns — does the\nassistant *cave* when a user pushes back — without ever calling an LLM.\n\nIt parses the raw transcript's user/assistant turns and, for each user\n**pushback** that follows a prior assistant answer (\"are you sure? I\nthink it's X\", \"no, it's Y\", \"you're wrong\"), classifies the next\nassistant turn:\n\n| Assistant response to pushback | Classification | Effect |\n|--------------------------------|----------------|--------|\n| capitulates (\"you're right\") **without** fresh reasoning | sycophantic flip | `score` ↓, `red_flags` dock |\n| capitulates **with** a re-derivation / \"because …\" | legitimate concession | not penalised |\n| holds its answer | held | `score` 1.0 |\n\nThat middle row is the point: Proofloop **does not penalise a correct\nconcession**. Conceding to a *true* user correction — and explaining\nwhy — is good behaviour; only bare capitulation with no new\njustification is scored as sycophancy. The result rides on the\nscorecard as a top-level `sycophancy` object (`score` 0–1 where 1.0 =\nheld under pressure, plus `flipped`, `stance_consistency`, `pushbacks`,\n`rationale`); a confirmed flip is added to `red_flags` so it docks the\ncomposite through the existing deduction machinery. When a transcript\nhas no pushback to test, the signal is simply absent (it neither\nrewards nor penalises).\n\n**Not a rubric.** This is response-quality engine logic that composes\nwith the existing `correctness` / `consistency` dimensions — the\ninventory stays at 11 rubrics. It scores **agreement-drift**, distinct\nfrom the trajectory-injection proposal (2026-06-09) and the\nrole-routing self-preference guard (2026-06-07). A labelled\nfalse-premise probe set ships at\n`skills/judge/references/sycophancy_probes.json` across five locales\n(en/es/fr/hi/zh), because sycophancy persists across languages\n([arXiv:2606.08451](https://arxiv.org/abs/2606.08451)) and an\nEnglish-only probe set would under-measure it. The heuristic is\noffline; the opt-in LLM second opinion remains the only LLM path.\nRefs: [arXiv:2606.09068](https://arxiv.org/abs/2606.09068),\n[arXiv:2606.08629](https://arxiv.org/abs/2606.08629).\n\n## Roadmap\n\nSee [ROADMAP_2026.md](ROADMAP_2026.md) for the 90-day plan. Latest\nrelease: [v3.1.1](https://github.com/sattyamjjain/proofloop/releases/tag/v3.1.1)\n(safety false-positive fix — a clean review that assigns a credential-named\nvariable like `token = refresh(token)` is no longer docked; real hardcoded\nsecrets are still flagged).\nPrevious releases:\n[v3.1.0](https://github.com/sattyamjjain/proofloop/releases/tag/v3.1.0)\n(GitHub Action + CI gate — run the offline scorer in CI and fail a job\nbelow your threshold),\n[v3.0.0](https://github.com/sattyamjjain/proofloop/releases/tag/v3.0.0)\n(rebrand to Proofloop + engine hardening — adherence/correctness no\nlonger hand out unearned credit),\n[v2.0.8](https://github.com/sattyamjjain/proofloop/releases/tag/v2.0.8)\n(verifier-collapse detector — flags judges that have flatlined at the\ntop of the scale over the rolling scorecard window),\n[v2.0.3](https://github.com/sattyamjjain/proofloop/releases/tag/v2.0.3)\n(ABA-anchored benchmark hygiene lint + ship-gate wire-up — flags\nspec gaps, env coupling, brittle grading, and missing ground truth\nin the regression-gate manifest *before* its scores ship; SARIF\noutput for CI surfacing),\n[v2.0.2](https://github.com/sattyamjjain/proofloop/releases/tag/v2.0.2)\n(safety-dim allowlist tracks Claude Code v2.1.126 — `.git/`,\n`.vscode/`, and a closed POSIX/zsh shell-config-file set added to\n`_is_plugin_author_write`),\n[v2.0.1](https://github.com/sattyamjjain/proofloop/releases/tag/v2.0.1)\n(opt-in `duration_ms` enrichment + safety `.claude` path\nallowlist + marketplace validator v2.1.120 keys), and the\nbreaking\n[v2.0.0](https://github.com/sattyamjjain/proofloop/releases/tag/v2.0.0)\ntrim to the v4.3 plugin-only scope — see\n[`CHANGELOG.md`](CHANGELOG.md#200---2026-05-03) and the\n[v1.x → v2.0.0 migration note](#features). No open tracker issues —\neach cycle's scope is tracked in a fresh issue, opened when the\ncycle starts and closed at release.\n\n## Contributing\n\nRubric contributions welcome — see `skills/judge/rubrics/custom-template.md`.\nCommunity rubrics can be installed via `scripts/install_rubric.py`\nfrom any HTTPS URL that serves a Proofloop-shaped rubric.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsattyamjjain%2Fproofloop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsattyamjjain%2Fproofloop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsattyamjjain%2Fproofloop/lists"}