{"id":51951708,"url":"https://github.com/dubsopenhub/fleet-scorecard-spec","last_synced_at":"2026-07-29T06:01:28.595Z","repository":{"id":356013377,"uuid":"1230638999","full_name":"DUBSOpenHub/fleet-scorecard-spec","owner":"DUBSOpenHub","description":"Portable protocol for judging agent-fleet CLI runs","archived":false,"fork":false,"pushed_at":"2026-05-06T16:22:21.000Z","size":29,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-07T22:21:29.410Z","etag":null,"topics":["agent-orchestration","copilot-cli","multi-agent","scorecard","spec"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DUBSOpenHub.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-05-06T07:20:39.000Z","updated_at":"2026-05-21T19:24:32.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/DUBSOpenHub/fleet-scorecard-spec","commit_stats":null,"previous_names":["dubsopenhub/fleet-scorecard-spec"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/DUBSOpenHub/fleet-scorecard-spec","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DUBSOpenHub%2Ffleet-scorecard-spec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DUBSOpenHub%2Ffleet-scorecard-spec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DUBSOpenHub%2Ffleet-scorecard-spec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DUBSOpenHub%2Ffleet-scorecard-spec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DUBSOpenHub","download_url":"https://codeload.github.com/DUBSOpenHub/fleet-scorecard-spec/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DUBSOpenHub%2Ffleet-scorecard-spec/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36020486,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-29T02:00:04.910Z","response_time":95,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent-orchestration","copilot-cli","multi-agent","scorecard","spec"],"created_at":"2026-07-29T06:01:27.131Z","updated_at":"2026-07-29T06:01:28.547Z","avatar_url":"https://github.com/DUBSOpenHub.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 📋 Fleet Scorecard Spec\n\n[![Spec Version](https://img.shields.io/badge/spec-v0.1.0-blue.svg)](SPEC.md)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![Conformance Levels](https://img.shields.io/badge/conformance-FSS--L1%20%7C%20FSS--L2%20%7C%20FSS--L3%20%7C%20FSS--L4-green.svg)](SPEC.md#6-conformance-levels)\n\n**A framework-agnostic protocol for judging whether an agent-fleet run was useful enough to repeat.**\n\n---\n\n## The Problem\n\nAgent fleets produce a lot of activity: commander bundles, worker logs,\ncollaboration ledgers, dashboards, diffs, tests, partials, and synthesized\nteardown artifacts. A run can look impressive while hiding the answer operators\nactually need:\n\n\u003e Was this fleet run worth repeating?\n\nWithout a standard scorecard, teams rerun by vibes, bury partial failures, and\nlose the best findings inside swarm noise.\n\n## The Solution\n\n```text\nFleet Score = change_clarity + winner_confidence + failure_accounting + repeat_decision\n```\n\nEach dimension is worth 25 points. Every conforming scorecard answers:\n\n1. What changed?\n2. What won?\n3. What failed?\n4. Would I run it again?\n\nFleet Scorecard is intentionally not cost accounting. Cost, model mix, duration,\nand token use can be optional telemetry, but they are not required scorecard\nquestions in v0.1.\n\n## Interpretation Scale\n\n| Score | Level | Meaning |\n|---:|---|---|\n| 85-100 | ✅ Strong | Repeatable pattern |\n| 70-84 | 🟢 Useful | Minor changes before repeating |\n| 50-69 | 🟡 Partial | Repeat only with changes |\n| 0-49 | 🔴 Weak | Redesign before repeating |\n\n## Why \"Fleet Scorecard\"?\n\nFleet Scorecard is the after-action report for agent orchestration.\n\nIt sticks because it maps to something operators already understand: a\nscorecard turns many signals into one accountable judgment. The fleet may be a\nswarm, stampede, conductor, hackathon, or manual collection of Copilot CLI\nsessions. The scorecard answers what happened and whether the pattern should be\nused again.\n\n## Quick Start: Add Fleet Scorecard in 5 Minutes\n\n### Option A: Use the reference validator\n\n```bash\npython validators/fleet-scorecard-validate.py examples/scorecard.json\n```\n\n### Option B: Produce the artifacts yourself\n\n1. **Write a run card** — run ID, mission, backend, source path.\n2. **Seal the rubric** — hash the four-question criteria before final scoring.\n3. **Collect evidence** — commander bundles, logs, ledgers, tests, diffs, Shadow Score.\n4. **Write the scorecard** — answer the four required questions.\n5. **Report the decision** — `run again`, `run again with changes`, `wait for completion`, or `do not rerun`.\n\nFramework, model provider, and orchestration backend do not matter.\n\n## The Fleet Scorecard Protocol\n\n```text\n RUN CARD ──► SEAL GENERATION ──► FLEET EXECUTION ──► EVIDENCE INDEX ──► SCORECARD\n               (rubric hash,       (agents,          (bundles,          (four questions,\n                before final        commanders,       logs, ledgers,      score, decision)\n                scoring)            workers)          tests, diffs)\n```\n\n**The critical rule:** a partial run stays partial. Heartbeats, preflights, and\ndashboards are useful telemetry, but they are not proof that the fleet produced\na complete outcome.\n\nFull protocol details: [**SPEC.md §4**](SPEC.md#4-fleet-scorecard-protocol)\n\n## Conformance Levels\n\n| Level | What's Required | Use Case |\n|---|---|---|\n| **FSS-L1** — Basic Scorecard | Four questions + status + rerun decision | Retrofitting onto any run |\n| **FSS-L2** — Sealed Rubric | L1 + rubric hash verification | Agent pipelines that need anti-vibes scoring |\n| **FSS-L3** — Evidence-linked | L2 + machine-readable evidence index | Auditable fleet retrospectives |\n| **FSS-L4** — Automated Lifecycle | L3 + automatic finalization/teardown emission | Production agent orchestration |\n\n## Reference Implementation\n\nThe reference Level 4 implementation is\n**[Agent Orchestra](https://github.com/DUBSOpenHub/agent-orchestra)** — a\nmulti-agent fleet conductor for the GitHub Copilot CLI. Agent Orchestra seals a\nFleet Scorecard rubric before commander launch and writes\n`.fleet-scorecards/\u003crun-id\u003e/scorecard.md` during teardown.\n\n## Worked Examples\n\n| Example | Conformance | What It Shows |\n|---|---|---|\n| [Basic Scorecard](examples/basic-scorecard.md) | FSS-L1 | Four-question scorecard for a partial run |\n| [Sealed Scorecard](examples/sealed-scorecard.md) | FSS-L2/FSS-L3 | Seal verification plus evidence links |\n| [Scorecard JSON](examples/scorecard.json) | FSS-L1 | Machine-readable answer format |\n| [Evidence Index](examples/evidence-index.json) | FSS-L3 | Source map for run claims |\n\n## Reporting Format\n\nFleet Scorecards can be produced in Markdown for humans and JSON for tools.\n\n**JSON schemas:** [`schemas/`](schemas/)\n\n```json\n{\n  \"schema_version\": \"fleet-scorecard-v0.1.0\",\n  \"run_id\": \"run-20260505-170318\",\n  \"status\": \"partial\",\n  \"fleet_score\": 78,\n  \"decision\": \"run again with changes\",\n  \"questions\": [\n    \"What changed?\",\n    \"What won?\",\n    \"What failed?\",\n    \"Would I run it again?\"\n  ],\n  \"answers\": {\n    \"what_changed\": \"The run produced commander bundles and a hardening backlog.\",\n    \"what_won\": \"Runtime hardening was the strongest consensus.\",\n    \"what_failed\": \"Two commanders were partial.\",\n    \"would_run_again\": \"Run again with a narrower mission.\"\n  }\n}\n```\n\nFull schema details: [**SPEC.md §5**](SPEC.md#5-reporting-format)\n\n## Adopters\n\n| Project | Conformance | Description |\n|---|---|---|\n| [Agent Orchestra](https://github.com/DUBSOpenHub/agent-orchestra) | FSS-L4 | Reference implementation — emits sealed Fleet Scorecards on teardown |\n\n*Using Fleet Scorecard? [Open a PR](https://github.com/DUBSOpenHub/fleet-scorecard-spec/pulls) to add your project.*\n\n## Full Specification\n\n📄 **[Read the full spec →](SPEC.md)**\n\nCovers: definitions, formula, protocol phases, reporting formats, conformance\nlevels, reference implementation, security guidance, and FAQ.\n\n## Contributing\n\nFleet Scorecard is an open specification. Contributions welcome:\n\n- **Spec changes**: Open an issue to discuss before submitting a PR.\n- **New validators**: PRs for additional language validators are welcome.\n- **Adopter listings**: Add your project to the Adopters table.\n\n## License\n\n[MIT](LICENSE) © 2026 DUBSOpenHub\n\nLet's orchestrate! 🎼\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdubsopenhub%2Ffleet-scorecard-spec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdubsopenhub%2Ffleet-scorecard-spec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdubsopenhub%2Ffleet-scorecard-spec/lists"}