{"id":51569434,"url":"https://github.com/artttj/quote-before-code","last_synced_at":"2026-07-10T18:30:28.349Z","repository":{"id":362853772,"uuid":"1261050645","full_name":"artttj/quote-before-code","owner":"artttj","description":"A Claude Code skill that forces agents to quote real code before writing new code. Backed by 930 controlled runs: 0/40 → 40/40 correctness.","archived":false,"fork":false,"pushed_at":"2026-06-07T04:36:24.000Z","size":215,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-07T05:14:26.224Z","etag":null,"topics":["ai-agent","claude-code","code-reuse","controlled-experiment","llm","prompt-engineering","skill","verification"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/artttj.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-06T07:11:07.000Z","updated_at":"2026-06-07T04:48:49.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/artttj/quote-before-code","commit_stats":null,"previous_names":["artttj/sysproof","artttj/quote-before-code"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/artttj/quote-before-code","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artttj%2Fquote-before-code","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artttj%2Fquote-before-code/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artttj%2Fquote-before-code/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artttj%2Fquote-before-code/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/artttj","download_url":"https://codeload.github.com/artttj/quote-before-code/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artttj%2Fquote-before-code/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35339931,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agent","claude-code","code-reuse","controlled-experiment","llm","prompt-engineering","skill","verification"],"created_at":"2026-07-10T18:30:27.529Z","updated_at":"2026-07-10T18:30:28.344Z","avatar_url":"https://github.com/artttj.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# quote-before-code\n\n[![version](https://img.shields.io/badge/version-v1.0.1-blue)](https://github.com/artttj/quote-before-code/releases) [![runs](https://img.shields.io/badge/runs-930-orange)](https://github.com/artttj/quote-before-code/blob/main/experiment/MASTER_SUMMARY.txt) [![p--value](https://img.shields.io/badge/p-1.9%C3%9710%E2%81%BB%C2%B3-green)](https://github.com/artttj/quote-before-code/blob/main/experiment/article_FINAL.md) [![license](https://img.shields.io/badge/license-MIT-gray)](https://github.com/artttj/quote-before-code/blob/main/LICENSE)\n\nA Claude Code skill that forces a coding agent to find and quote a real convention from your repo before it writes any code. One rule runs it:\n\n\u003e **Do not write code until you have quoted one real convention from the repo.**\n\nThis repo is two things: the skill, and the controlled experiment that decided its exact shape. The numbers below are why the skill reads the way it does.\n\n## The problem it fixes\n\nWhen a task looks easy, an agent writes the generic version from memory and never opens your repo. So it reinvents a helper that already exists, raises the wrong exception type, ignores a validation order, and breaks call sites that depended on the local convention. The requirement was sitting in the code. The agent was too sure of itself to go look.\n\n## What the experiment found\n\nI started with the opposite skill: a careful-sounding principle that told the agent to verify its work and to ground itself in the codebase. Then I tested whether it did anything. The study is 12 conditions, about 930 mechanically graded runs, three model tiers.\n\n- **Principle-style skills are inert.** \"Verify before done\" and \"ground your work in the codebase\" matched a bare control across every fair condition. On a weaker model the verbose skill was worse, because it pushed confident DONE declarations and suppressed honest hedging.\n- **A promising redesign died under power.** Front-loading \"research the codebase first\" led the control at n=10 (60% vs 45% correct). Scaled to 40 runs per arm the win vanished completely: 0/40 versus control 0/39, Fisher p = 1.0. In all 40 runs the agent never once opened the code it needed, even though the module was importable and the instruction was in every prompt.\n- **A procedure with a hard stop changed everything.** Swapping the principle for an executable step plus a blocking gate — \"locate the package, read it, quote a real convention, and do not write code until you have\" — took reuse of the existing code from **0% to 100%** and correctness on a latent-requirement task from **0/40 to 40/40** (p = 1.9 × 10⁻²³; pooled across tasks p = 7.8 × 10⁻¹⁹).\n- **Either ingredient alone is enough.** Ablation: the same instruction buried in a long multi-phase skill scored 0%, a focused principle scored 50%, and either a concrete how-to or the hard stop alone scored 100%. Dilution was part of why the original failed.\n- **The gate fixes discovery, not comprehension.** On the hardest task, the gate got the agent to read the right module 100% of the time but it still applied the rule correctly only 18% of the time. \"Didn't look\" and \"looked but misapplied\" are two different failures, and a hard stop only solves the first.\n\nThis lines up with the literature: intrinsic self-correction with no external signal does not reliably help and can hurt (Huang et al., ICLR 2024), and the gains in agentic coding come from scaffolding that makes the model fetch a real signal (SWE-agent, NeurIPS 2024). The skill is a prompt-level version of that scaffolding.\n\nThe takeaway, and the reason this skill exists: **write skills as an action plus a line the model cannot cross, not as a value.** \"Ground your work\" is noise. \"Do not write code until you have quoted a real convention\" is a behavior change you can measure at p below 10⁻¹⁸.\n\nThe full write-up is in [`experiment/article_FINAL.md`](experiment/article_FINAL.md). The condition-by-condition numbers are in [`experiment/MASTER_SUMMARY.txt`](experiment/MASTER_SUMMARY.txt).\n\n## Install\n\n```\n/plugin marketplace add artttj/quote-before-code\n/plugin install quote-before-code@quote-before-code\n```\n\nOr drop the skill in by hand:\n\n```bash\ngit clone https://github.com/artttj/quote-before-code.git\nmkdir -p ~/.claude/skills/quote-before-code\ncp quote-before-code/plugins/quote-before-code/skills/quote-before-code/SKILL.md ~/.claude/skills/quote-before-code/\n```\n\nIt triggers on its own when you change behavior in an existing codebase. Skip it for genuinely greenfield code with no surrounding repo to read.\n\n## What's inside\n\n```\nplugins/quote-before-code/skills/quote-before-code/SKILL.md   the skill: the Iron Law, the procedure, the hard stop\nexperiment/\n  article_FINAL.md        the write-up\n  MASTER_SUMMARY.txt      every condition, with stats\n  results_*.csv           the graded data per condition\n  *_runs.json             raw run records\n  harness/                arms, grader, reuse check, task loader\n  tasks/                  the 10 graded tasks\n  hostrepo/newsroom/      the host package agents were meant to reuse\n  analysis.py             Wilson intervals, Fisher tests, bootstraps\n```\n\n## Reproduce\n\n```bash\ncd experiment\npython3 -m venv .venv \u0026\u0026 source .venv/bin/activate\npip install -r requirements.txt\npython3 analysis.py        # re-derive the stats from the committed run data\n```\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fartttj%2Fquote-before-code","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fartttj%2Fquote-before-code","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fartttj%2Fquote-before-code/lists"}