{"id":51951704,"url":"https://github.com/dubsopenhub/shadow-score-spec","last_synced_at":"2026-07-29T06:01:28.311Z","repository":{"id":340293191,"uuid":"1165382179","full_name":"DUBSOpenHub/shadow-score-spec","owner":"DUBSOpenHub","description":"A framework-agnostic metric for measuring AI code generation quality. Sealed-envelope testing protocol + reference validators.","archived":false,"fork":false,"pushed_at":"2026-07-28T21:18:56.000Z","size":106,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-28T23:10:00.169Z","etag":null,"topics":["ai-agents","ai-quality","code-quality","gap-score","metrics","sealed-envelope","specification","testing-methodology"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DUBSOpenHub.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-02-24T05:27:05.000Z","updated_at":"2026-07-28T21:19:01.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/DUBSOpenHub/shadow-score-spec","commit_stats":null,"previous_names":["dubsopenhub/gap-score-spec","dubsopenhub/shadow-score-spec"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/DUBSOpenHub/shadow-score-spec","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DUBSOpenHub%2Fshadow-score-spec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DUBSOpenHub%2Fshadow-score-spec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DUBSOpenHub%2Fshadow-score-spec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DUBSOpenHub%2Fshadow-score-spec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DUBSOpenHub","download_url":"https://codeload.github.com/DUBSOpenHub/shadow-score-spec/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DUBSOpenHub%2Fshadow-score-spec/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36020486,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-29T02:00:04.910Z","response_time":95,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","ai-quality","code-quality","gap-score","metrics","sealed-envelope","specification","testing-methodology"],"created_at":"2026-07-29T06:01:27.067Z","updated_at":"2026-07-29T06:01:28.271Z","avatar_url":"https://github.com/DUBSOpenHub.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 📊 Shadow Score Spec\n\n[![Spec Version](https://img.shields.io/badge/spec-v2.0.0-blue.svg)](SPEC.md)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![Conformance Levels](https://img.shields.io/badge/conformance-L1%20%7C%20L2%20%7C%20L3%20%7C%20L4-green.svg)](SPEC.md#6-conformance-levels)\n\n**A framework-agnostic metric for measuring AI code generation quality.**\n\n---\n\n## The Problem\n\nAI coding tools write code and tests together. The tests always pass — but they only validate what the AI *thought* to check, not what the *specification actually requires*. There's no independent quality signal.\n\n## The Solution\n\n```\nShadow Score = (sealed_failures / sealed_total) × 100\n```\n\nGenerate acceptance tests from the spec **before code exists**. Hide them from the AI. Build the code. Then run both test suites. The delta is your **Shadow Score** — an adversarial, quantitative measure of implementation quality.\n\n## Interpretation Scale\n\n| Score | Level | Meaning |\n|-------|-------|---------|\n| 0% | ✅ Perfect | Implementation tests covered everything |\n| 1–15% | 🟢 Minor | Small blind spots — edge cases |\n| 16–30% | 🟡 Moderate | Meaningful gaps in coverage |\n| 31–50% | 🟠 Significant | Major gaps — review approach |\n| \u003e50% | 🔴 Critical | Fundamental quality issues |\n\n## Why \"Shadow Score\"?\n\n*Shadow Score measures what your AI missed when it couldn't see the tests.*\n**Shadow Score** sticks because it maps to something developers already know: *shadow testing* — running hidden checks alongside production to catch what the main path misses. That's exactly what sealed-envelope testing does. The AI builds code. Shadow tests — written before the code existed and hidden from the builder — judge it. Whatever fails is what the AI couldn't see.\n\n**0% = no shadows.** The implementation covered everything the spec required.  \n**60% = flying blind.** The AI missed more than half the acceptance criteria it never knew about.\n\n## Quick Start: Add Shadow Score in 5 Minutes\n\n### Option A: Use the reference validators\n\n```bash\n# Python\npython validators/shadow-score.py --sealed results-sealed.json --open results-open.json\n\n# Go\ncd validators \u0026\u0026 go build -o shadow-score-go . \u0026\u0026 cd ..\n./validators/shadow-score-go --sealed results-sealed.json --open results-open.json\n\n# Shell (minimal deps)\n./validators/shadow-score.sh results-sealed.txt results-open.txt\n```\n\n### Option B: Compute it yourself\n\n1. **Write sealed tests** from your spec (requirements → test cases, before code exists) — using a *different model family* than the one that will implement\n2. **Build the code** — the implementer never sees the sealed tests\n3. **Run both suites** — in a disposable workspace built from the implementer's commit\n4. **Compute**: `failed_sealed / total_sealed × 100`\n5. **Report**: Use the [JSON schema](validators/shadow-report-schema.json) or markdown format, including which families authored each side\n\nThat's it. Framework, language, and tooling don't matter — Shadow Score works anywhere.\n\n## The Sealed-Envelope Protocol\n\nShadow Score is computed using the **Sealed-Envelope Protocol** — a 4-phase testing methodology:\n\n```\n SPEC ──► SEAL GENERATION ──► IMPLEMENTATION ──► VALIDATION ──► HARDENING\n           (tests from spec,     (code + own        (run both      (fix from\n            hidden from          tests, never       suites in a   failure msgs\n            implementer,         sees sealed)       disposable    only — no\n            different model                        workspace)     test code)\n            family)\n```\n\n**The two critical rules:**\n\n1. **Isolation** — the implementer **never sees** the sealed tests. During hardening it receives only failure messages (test name, expected, actual), never test source. This forces root-cause fixes, not test-targeting hacks.\n2. **Independence** — the sealed tests are written by a **different model family** than the implementation. Isolation stops the builder seeing the tests; it does not stop it thinking like their author. Same-family agents share blind spots, so the scenario nobody tested is the scenario nobody handled.\n\nFull protocol details: [**SPEC.md §4**](SPEC.md#4-sealed-envelope-protocol)\n\n## Conformance Levels\n\n| Level | What's Required | Use Case |\n|-------|----------------|----------|\n| **L1** — Shadow Score | Compute + report Shadow Score | Retrofitting onto existing test suites |\n| **L2** — Sealed Envelope | L1 + test isolation + tamper hash | AI agent pipelines |\n| **L3** — Full Protocol | L2 + hardening loop + velocity tracking | Production autonomous builds |\n| **L4** — Adversarial Independence | L3 + cross-family authorship + disposable verify workspace + provenance in report | Published or gate-blocking scores |\n\n**Why L4 exists:** L1–L3 harden *information* isolation — they stop the builder from **seeing** the tests. They are all silently defeated by one configuration choice: pointing the seal author and the implementer at the same model. Same-family agents share blind spots, so the untested defect is also the unhandled defect. The sealed test is never written, the score reads 0%, and the bug ships green. That bias is directional — it always overclaims quality. See [**SPEC.md §3.6**](SPEC.md#36-authorship-independence).\n\n## Reference Implementations\n\nThe reference Level 4 implementation is **[Dark Factory](https://github.com/DUBSOpenHub/dark-factory)** — an autonomous agentic build system for the GitHub Copilot CLI with sealed-envelope testing, cross-family model independence enforced pre-dispatch, and seal plurality.\n\nThe reference Level 2 implementation is **[Terminal Stampede](https://github.com/DUBSOpenHub/terminal-stampede)** — a parallel agent runtime that shadow-scores each agent's work during merge using sealed tests with tamper-hash verification.\n\n## Worked Examples\n\n| Example | Shadow Score | What Happened |\n|---------|-----------|---------------|\n| [01 — Perfect Score](examples/01-perfect-score/) | 0% ✅ | All sealed tests passed |\n| [02 — Minor Gaps](examples/02-minor-gaps/) | 11.1% 🟢 | 2 edge cases missed |\n| [03 — Critical Gaps](examples/03-critical-gaps/) | 60% 🔴 | Only happy path tested |\n\n## Reporting Format\n\nShadow Reports can be produced in JSON (machine-readable) or Markdown (human-readable).\n\n**JSON schema:** [`validators/shadow-report-schema.json`](validators/shadow-report-schema.json)\n\n```json\n{\n  \"shadow_score_spec_version\": \"2.0.0\",\n  \"report\": {\n    \"shadow_score\": 11.1,\n    \"level\": \"minor\",\n    \"conformance_level\": 4,\n    \"independence\": \"strong\",\n    \"implementer_family\": \"anthropic\",\n    \"seal_author_families\": [\"openai\", \"google\"],\n    \"workspace_isolation\": \"strict\"\n  },\n  \"sealed_tests\": { \"total\": 18, \"passed\": 16, \"failed\": 2 },\n  \"failures\": [\n    {\n      \"test_name\": \"test_rejects_gpl_dependency\",\n      \"category\": \"security\",\n      \"expected\": \"exit code 2\",\n      \"actual\": \"exit code 0\",\n      \"message\": \"GPL dependency not blocked\"\n    }\n  ]\n}\n```\n\nA Shadow Score without provenance cannot be interpreted: 0% under `strong` independence and 0% under `weak` independence are different claims about the world, and only one of them is evidence.\n\nFull schema: [**SPEC.md §5**](SPEC.md#5-reporting-format)\n\n## Adopters\n\n| Project | Conformance | Description |\n|---------|-------------|-------------|\n| [Dark Factory](https://github.com/DUBSOpenHub/dark-factory) | Level 4 | Reference implementation — autonomous agentic build system with cross-family adversarial independence |\n| [Terminal Stampede](https://github.com/DUBSOpenHub/terminal-stampede) | Level 2 | Parallel agent runtime — shadow-scores each agent's work during merge via sealed tests |\n\n*Using Shadow Score? [Open a PR](https://github.com/DUBSOpenHub/shadow-score-spec/pulls) to add your project.*\n\n## Full Specification\n\n📄 **[Read the full spec →](SPEC.md)**\n\nCovers: definitions, formula, sealed-envelope protocol, reporting format, conformance levels, worked examples, and FAQ.\n\n## Contributing\n\nShadow Score is an open specification. Contributions welcome:\n\n- **Spec changes**: Open an issue to discuss before submitting a PR\n- **New validators**: PRs for additional language validators (Go, Rust, TypeScript) are welcome\n- **Adopter listings**: Add your project to the Adopters table\n\n## License\n\n[MIT](LICENSE) © 2026 DUBSOpenHub\n\n---\n\n🐙 Created with 💜 by [@DUBSOpenHub](https://github.com/DUBSOpenHub) with the [GitHub Copilot CLI](https://docs.github.com/copilot/concepts/agents/about-copilot-cli).\n\nLet's build! 🚀✨\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdubsopenhub%2Fshadow-score-spec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdubsopenhub%2Fshadow-score-spec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdubsopenhub%2Fshadow-score-spec/lists"}