{"id":51575015,"url":"https://github.com/tjhavranek/paper-workshop","last_synced_at":"2026-07-10T23:30:55.260Z","repository":{"id":362827599,"uuid":"1260729280","full_name":"tjhavranek/paper-workshop","owner":"tjhavranek","description":"An adversarial AI expert workshop that stress-tests a research paper (rival-tradition referees argue; every comment quote-grounded and independently re-verified) and then rebuilds it: tracked-changes redline, clean version, your code re-run under a provenance wall, and a replication package. A Claude Code skill.","archived":false,"fork":false,"pushed_at":"2026-06-16T14:07:48.000Z","size":1043,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-16T15:27:58.784Z","etag":null,"topics":["adversarial-evaluation","claude","claude-code","multi-agent-debate","peer-review","reproducibility","research-audit","scientific-writing"],"latest_commit_sha":null,"homepage":null,"language":"Markdown","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tjhavranek.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-05T20:08:07.000Z","updated_at":"2026-06-16T14:09:35.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/tjhavranek/paper-workshop","commit_stats":null,"previous_names":["tjhavranek/paper-workshop"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tjhavranek/paper-workshop","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tjhavranek%2Fpaper-workshop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tjhavranek%2Fpaper-workshop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tjhavranek%2Fpaper-workshop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tjhavranek%2Fpaper-workshop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tjhavranek","download_url":"https://codeload.github.com/tjhavranek/paper-workshop/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tjhavranek%2Fpaper-workshop/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35346644,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adversarial-evaluation","claude","claude-code","multi-agent-debate","peer-review","reproducibility","research-audit","scientific-writing"],"created_at":"2026-07-10T23:30:54.680Z","updated_at":"2026-07-10T23:30:55.251Z","avatar_url":"https://github.com/tjhavranek.png","language":"Markdown","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CRUCIBLE: the `paper-workshop` skill\n\n[![ci](https://github.com/tjhavranek/paper-workshop/actions/workflows/ci.yml/badge.svg)](https://github.com/tjhavranek/paper-workshop/actions/workflows/ci.yml)\n[![DOI](https://zenodo.org/badge/1260729280.svg)](https://doi.org/10.5281/zenodo.20828996)\n\nImagine a panel of the world's leading experts, assembled for your exact paper, arguing it out\nfrom rival schools and then rebuilding it themselves, re-running your own code so every revised number is regenerated, not retyped.\n\n**An adversarial expert workshop that turns its critique into the edits that answer it: a\ntracked redline, a clean draft, and a replication package, before you submit.**\n\n*One tool, two names: `paper-workshop` is the repo you install and the phrase you type;\n**CRUCIBLE** is what you'll see in the report headers.*\n\nCRUCIBLE convenes a panel of AI referees built specifically for *your* manuscript and makes\nthem **argue with each other** from rival schools of thought, grounding every criticism in an\nexact quote (no invented objections, no confidence theater). Then, if you want, it\n**implements the agreed fixes**: a tracked-changes redline, a clean revised version, your\nanalysis code **re-run to regenerate every affected number**, and a reproducible replication package.\n\nTwo things are mechanically checked: every criticism is pinned to a real quote, and every revised number\nis produced by a real re-run of your own code. It is equally plain about the rest: effectiveness\nis not measured yet (no recall or false-positive numbers). We have run the workshop on more than ten\npapers in our own work, but only one is public (an accepted paper from the authors' own group; see\n[`examples/incentives-workshop`](examples/incentives-workshop)), where Act II is demonstrated\nend-to-end once; none of it is a controlled validation. Read the [limitations](LIMITATIONS.md)\nbefore you rely on it.\n\nA [Claude Code](https://claude.com/claude-code) skill. Claude-only. Runs on any paid plan.\n\n---\n\n## What makes it different\n\nGood AI paper-reviewers already exist: multi-agent review, grounded critique, and\ntopic-adaptive reviewers are all prior art, and dedicated citation-checkers triangulate more\nreference databases than CRUCIBLE's web-based check does. CRUCIBLE's bet is a pairing\nexisting tools rarely offer in one run: reviewers that genuinely **debate** from opposed\nobjective functions, and an integrated **rebuild** that re-runs your analysis and hands back\na corrected, reproducing manuscript.\n\n- **The referees argue.** Every contested choice in your paper is taken up by at least two\n  experts from *rival traditions* with *opposite jobs*: one tries to break it, one tries to\n  defend it. You see the real disagreement and the crux instead of a bland averaged verdict.\n- **The panel is built from your paper.** A scout reads your manuscript and assembles the\n  specific experts it needs (your identification strategy, your estimator, publication bias,\n  your benchmark, your proof), plus generalists who ask \"does this even matter?\" and \"would a\n  smart outsider follow it?\"\n- **Nothing is made up.** Every criticism cites an exact quote, checked by a deterministic\n  script rather than the model's memory, and the most load-bearing **cited works are fetched**\n  so the paper's claims about them are checked against the originals. What can't be verified is\n  flagged *needs author confirmation* and never asserted. No acceptance-probability numbers, ever.\n- **Every comment is re-checked from many angles** (Roundtable and up) by independent blind\n  verifiers (does the quote exist, does the criticism actually follow, is the severity\n  calibrated, does the proposed fix break something) before it reaches you; a comment whose\n  checks did not fully resolve arrives flagged *needs your confirmation*, not as a settled\n  must-fix. Desk Review, the lightest mode, is a single-pass read with inline chair checks\n  instead of the panel.\n- **It also argues FOR your paper.** Every full tribunal run also staffs a contribution rival\n  pair: an *overclaim prosecutor* hunting where your framing outruns the evidence, and a\n  *contribution maximizer* hunting the opposite failure, the bolder claim your own results\n  defensibly support but your paper never makes. The maximizer's candidates clear two\n  deterministic gates and arrive as a separate, non-blocking **Contribution Memo** of at most\n  3 items: suggestions for you to ratify or ignore, never must-fixes. At Workshop depth and\n  above a related-literature scout also looks past your own bibliography for overlooked work,\n  under a strict fetch-or-drop rule: a work it could not actually open is never cited as\n  evidence. The two gates' internals and what the scout does not certify are in\n  [`LIMITATIONS.md`](LIMITATIONS.md).\n- **It rebuilds the paper itself.** The opt-in second act (the **ATELIER**) turns the agreed\n  findings into a tracked-changes **redline** *and* a **clean revised version**, **re-runs\n  your own code** to regenerate the affected numbers, tables, and figures, and assembles a\n  **replication package**, under one hard rule: *no number enters your paper unless a real,\n  logged re-run produced it* (enforced by deterministic provenance and consistency checks).\n  Every change is mapped to the reviewer concern it answers.\n- **You stay in control.** It works on copies, never your originals. Anything that touches a\n  number, a sample, a claim, or a result waits for your sign-off. It never edits your only\n  copy, never submits, never releases data.\n\nIf you only want a fast referee-style critique, lighter tools (and CRUCIBLE's own **Desk\nReview** mode) do that. CRUCIBLE is for when you want the argument *and* the rebuild.\n\n**See it on a real paper.** [`examples/incentives-workshop`](examples/incentives-workshop) is CRUCIBLE\nrun end to end on an accepted JPE-Microeconomics meta-analysis: a topic-built referee panel that argues,\nthen a re-run of the authors' own Stata and R that regenerated the data byte-for-byte identical, with a\ndeterministic provenance proof. ([`examples/self-audit`](examples/self-audit) is the tool run on its own design.)\n\n**Limits.** **[`LIMITATIONS.md`](LIMITATIONS.md)** is the straight account of\nwhat is genuinely enforced and what is not proven yet: no measured recall or false-positive\nnumbers; same-model decorrelation is a design bet, not a proof; coverage means *attention*,\nnot correctness; and the Contribution Memo's selection is same-model judgment with no measured\nundersell-recall yet. Act II is built, [unit-tested](.github/workflows/ci.yml), and **demonstrated end-to-end once** on a\nreal accepted paper ([`examples/incentives-workshop`](examples/incentives-workshop)), a\ndemonstration and not independent validation; re-derive any regenerated number yourself. A\npre-release self-audit caught real overclaims and a quote-gate bug, both fixed\n([`examples/self-audit/`](examples/self-audit/)); a development pass, not validation.\n\n## Modes: pick your depth\n\n| Mode | What convenes | Experts | ≈ agents | Best for |\n|---|---|---|---|---|\n| **Desk Review** | one expert pass, no fleet | a few lenses | ~1–6 | a fast read; lightest setup |\n| **Roundtable** | a small adversarial panel | 6–8 | ~20–30 | a quick but real workshop |\n| **Workshop** *(default)* | the full adversarial workshop | 12–18 | ~45–65 | serious pre-submission review |\n| **Symposium** | a large fleet + close-readers | 25–40 | ~90–250 | top-venue preparation |\n| **Summit** | every subsystem, every sentence | 60–120+ | ~300–600 | the most exhaustive pass (opt-in) |\n\n*(Symposium/Summit also scale with paper length and are best run with dynamic workflows enabled; without them they fall back to Workshop depth.)*\n\n**Running it often is fine.** Desk Review and Roundtable are light (single-digit to ~30 agents)\nand the default Workshop is a few dozen — cheap enough to re-run as you revise a paper. Reserve\nSymposium/Summit for a major pre-submission pass.\n\n**Engine.** The workshop runs on subagents (helper Claude sessions your main session spawns),\nwhich work on every plan. When dynamic workflows are available (a Claude Code orchestration\nfeature: on by default on Max; on Pro, switch them on in `/config`), CRUCIBLE uses them to run\nthe same phases more efficiently; if they're off, Symposium and Summit run at Workshop depth\nand CRUCIBLE tells you rather than downgrading silently. Engine detection, model and context\ninheritance, and the 1M-context account caveat and its remedies live in the pre-flight,\n[`helpers/doctor.md`](helpers/doctor.md).\n\n**Running on Claude Fable 5 (mythos-class).** In the default mode the fleet inherits the\nsession model, so starting the session on Claude Fable 5 (Anthropic's mythos-class tier, the\nclass above Opus) lifts every seat, verifier, and chair to that capability with no\nconfiguration; `/model best` (Claude Code v2.1.170+) picks Fable where your plan has it and\nthe latest Opus where it does not. Anthropic's Fable 5 prompting guidance favors separate\nfresh-context verifier subagents over self-critique, which matches CRUCIBLE's verification\npanel: a design endorsement, not a measured gain. For biology- or security-flavored papers,\nstart on Opus deliberately, because a safety classifier can reroute Fable to Opus 4.8 mid-run;\nstarting on Opus keeps the run on one model. The run records which model actually served, and\nthe report discloses it. Fable's other June 2026 constraints (30-day input retention with no\nzero-data-retention, twice Opus pricing) are in [`helpers/doctor.md`](helpers/doctor.md) and\n[`helpers/safety_notes.md`](helpers/safety_notes.md). The workshop runs unchanged on Opus and\nSonnet; the methodology does not depend on the model tier.\n\n**Running on a usage-constrained plan: the economy register.** A default Workshop inherits the\nsession model into every agent, and on Fable that can exhaust a usage-capped plan's window\nmid-run; a locked-out run delivers zero findings. Saying \"economy\" casts the run in tiers\n(judgment seats at the Opus floor, mechanical phases on Sonnet, scout/chair/scribes at the\nsession model), with every deterministic rail unchanged, the cast recorded in `meta.json` and\nthe report header, and a never-upgrade clamp so economy can lower a run's cost but never raise\nit. The pre-flight ([`helpers/doctor.md`](helpers/doctor.md)) shows a cost preview and offers it\nbefore any Workshop-or-larger launch. The anchor so far is one real Workshop-band field run:\n3.70M subagent tokens in 55 minutes, well inside a Max-plan window, delivering a 60-finding\nverified ledger; one run's evidence, not blind validation (see [`LIMITATIONS.md`](LIMITATIONS.md)).\n\n## Install\n\n```bash\n# 1. Claude Code installed; Python 3.8+ on PATH (for the deterministic quote/number checks).\n# 2. Make the skills dir (it may not exist yet) and clone into it:\nmkdir -p ~/.claude/skills\ngit clone https://github.com/tjhavranek/paper-workshop ~/.claude/skills/paper-workshop\n# 3. Restart Claude Code, then confirm it loaded with a quick pass on any PDF:\n#      workshop my paper: some.pdf   desk review\n```\n\nWindows (PowerShell; if `python` isn't found, the `py -3` launcher or a conda Python works too):\n\n```powershell\nNew-Item -ItemType Directory -Force \"$env:USERPROFILE\\.claude\\skills\" | Out-Null\ngit clone https://github.com/tjhavranek/paper-workshop \"$env:USERPROFILE\\.claude\\skills\\paper-workshop\"\n# then restart Claude Code and confirm:  workshop my paper: some.pdf   desk review\n```\n\nFor the rebuild (Act II) you also need the interpreters your analysis uses (R / Python /\nStata), `latexmk` (LaTeX) or the bundled `docx` skill (Word), and `git`.\n\n## Use\n\nType these into a Claude Code session (run `claude` in a terminal to open one):\n\n```\nworkshop my paper: mypaper.pdf  desk review     # the lightest pass, one expert (good first run)\nworkshop my paper: mypaper.pdf                 # default Workshop mode, supportive register\nworkshop my paper: mypaper.pdf  roundtable     # a quick adversarial pass\nworkshop my paper: mypaper.pdf  summit brutal  # the most exhaustive pass, brutal register\nCRUCIBLE mypaper.pdf  symposium                 # the brand name works as a trigger too\n```\n\nCRUCIBLE runs the workshop and presents the report. Then it **stops and asks** whether to\nimplement the changes; if you say yes, it requests your source, data, and code and produces the\nredline, the clean version, and the replication package. *Register* (`supportive` / `brutal`)\nchanges only the tone of the write-up; the severity of a finding never changes.\n\n**Want it bolder?** Add `improvement` (for example `workshop my paper: mypaper.pdf symposium improvement`)\nto turn on an opt-in generative wing. Alongside the critique, the workshop then proposes substantive\nways to make the paper stronger: bolder defensible claims your own results support, analyses worth\nrunning, sharper framing. In the rebuild it drafts these as extra tracked changes you accept or\nreject. It is off by default, never enters the must-fix list, and is most aggressive in the heavier\nmodes (Symposium and Summit). Like the contribution memo, it is a grounded option set, not a\nvalidated verdict.\n\n## Lineage\n\nCRUCIBLE grew out of the authors'\n[`mad-research`](https://github.com/tjhavranek/mad-research) (a cross-model audit that produces\na memo) and\n[`research-audit-duel-protocol`](https://github.com/tjhavranek/research-audit-duel-protocol)\n(manual multi-model protocols). It inherits their discipline (a locked severity rubric,\nquote-and-locate grounding, a preserved minority report, no confidence scores, read-only\ntreatment of your files) and adds the topic-adapted debating fleet and the rebuild. It\ncomplements `mad-research` rather than replacing it: `mad-research` is a fast cross-model\n(Claude + Codex) audit memo, while CRUCIBLE is a deeper Claude-only fleet that can also rebuild\nthe paper. For an important project, run both in parallel and compare.\n(`mad-research`'s own small blinded comparison, n = 5 meta-analyses, found a Claude-only\nconfiguration ranked above its cross-model setup by an independent judge: illustrative, not\nproof. CRUCIBLE is Claude-only by design, and a single optional non-Claude \"what did we all\nmiss?\" pass is available for those who configure it.)\n\nA lighter sibling by the same author group,\n[`erc-ai-feedback`](https://github.com/tjhavranek/erc-ai-feedback), is a rubric-based pre-review\nfor ERC Starting and Consolidator grant proposals, run in a single chat session. It only flags\nproblems and leaves the drafting to the applicant, whereas CRUCIBLE argues a paper out and, if\nyou want, rebuilds it: one is built for proposals, the other for papers.\n\n## What it deliberately will not do\n\nNo fabricated citations, numbers, quotes, data, or results. No confidence or acceptance-odds\nnumbers. No silent edits; changes arrive as tracked redlines on copies for you to accept. No\nnumber in the revised paper that a logged re-run didn't produce. No automatic merge,\nsubmission, or data release. The author remains the author.\n\n## License \u0026 citation\n\nCC-BY-4.0 ([`LICENSE`](LICENSE)). If it helps your work, please cite:\n\n\u003e Havranek, T. \u0026 Irsova, Z. (2026). *CRUCIBLE (`paper-workshop`): an adversarial multi-expert\n\u003e workshop that stress-tests and rebuilds a research paper.* Zenodo. https://doi.org/10.5281/zenodo.20828996\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftjhavranek%2Fpaper-workshop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftjhavranek%2Fpaper-workshop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftjhavranek%2Fpaper-workshop/lists"}