{"id":51013008,"url":"https://github.com/alex-jb/council-diff","last_synced_at":"2026-06-21T06:01:35.253Z","repository":{"id":364328599,"uuid":"1264067934","full_name":"alex-jb/council-diff","owner":"alex-jb","description":"5-voice AI council for any decision · Brier-audited · TypeScript / Python","archived":false,"fork":false,"pushed_at":"2026-06-12T14:58:48.000Z","size":89,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-12T16:29:12.436Z","etag":null,"topics":["ai-agents","anthropic","brier-score","claude","decision-support","llm-evaluation","multi-agent","typescript"],"latest_commit_sha":null,"homepage":"https://www.vibexforge.com/council","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alex-jb.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-09T14:27:44.000Z","updated_at":"2026-06-12T14:58:53.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/alex-jb/council-diff","commit_stats":null,"previous_names":["alex-jb/council-diff"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/alex-jb/council-diff","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alex-jb%2Fcouncil-diff","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alex-jb%2Fcouncil-diff/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alex-jb%2Fcouncil-diff/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alex-jb%2Fcouncil-diff/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alex-jb","download_url":"https://codeload.github.com/alex-jb/council-diff/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alex-jb%2Fcouncil-diff/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34596047,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-21T02:00:05.568Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","anthropic","brier-score","claude","decision-support","llm-evaluation","multi-agent","typescript"],"created_at":"2026-06-21T06:01:34.453Z","updated_at":"2026-06-21T06:01:35.234Z","avatar_url":"https://github.com/alex-jb.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# council-diff\n\n[![npm version](https://img.shields.io/npm/v/council-diff.svg)](https://www.npmjs.com/package/council-diff)\n[![npm downloads](https://img.shields.io/npm/dm/council-diff.svg)](https://www.npmjs.com/package/council-diff)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n\n\u003e [English](README.md) · [中文](README.zh-CN.md)\n\n### Software 3.0 Reference Implementation · The OSS evaluation loop for multi-persona AI agents\n\n\u003e **\"Traditional software automates what you can specify. AI automates what you can verify.\"**\n\u003e Andrej Karpathy, Sequoia AI Ascent, April 20, 2026\n\n\u003e **\"Agentic engineers design specs, supervise plans, inspect diffs, write tests, create evaluation loops, manage permissions, isolate worktrees, and preserve quality.\"**\n\u003e Karpathy, same talk\n\ncouncil-diff is the smallest reproducible **evaluation loop** for multi-persona agents. Paste a decision, get 5 persona verdicts in parallel, watch them disagree, optionally let a Fable 5 Oracle adjudicate, then Brier-audit every voice when the outcome resolves. v0.3.0 shipped 6 months before Anthropic's \"advisor strategy\" beta. MIT licensed, on npm.\n\n## Why this is a Software 3.0 artifact\n\nKarpathy's framing splits software history into three eras:\n\n- **Software 1.0**: explicit code you write\n- **Software 2.0**: trained neural networks\n- **Software 3.0**: prompting an LLM interpreter\n\nSoftware 3.0 has no compiler error and no unit test that catches a hallucination. The job description shifts. The agentic engineer designs specs precise enough that ambiguity has nowhere to hide, then builds the **evaluation loop** that catches the model when it drifts.\n\ncouncil-diff maps onto that bullet directly:\n\n| Karpathy's job description | council-diff primitive |\n|---|---|\n| Design specs so precise ambiguity has nowhere to hide | 5 persona briefs per domain, each with explicit bias declarations |\n| Inspect diffs | `voice.verdict` + `voice.strength` + `voice.gap` per persona, side by side |\n| Write tests | `agreement_score` is the test. 1.0 unanimous, 0.0 split |\n| **Create evaluation loops** | **Brier audit module. Every voice scored at resolution, calibration tracked over 30 / 90 days** |\n| Manage permissions | `safeMode: true` forces zero-retention Sonnet 4.6, surfaces `data_retention` per call |\n| Preserve quality | Oracle layer. Fable 5 reads all 5 verdicts and adjudicates, Brier-audited separately |\n\nThe persona-vs-persona format is the spec. The agreement score is the test. The Brier audit is the evaluation loop. The Oracle is the supervisor. That is one OSS library, 5 verdicts per call, ~$0.03.\n\n## What it does\n\nSingle-LLM verdicts hide their own uncertainty. A 90% confident answer from one model and 5 specialists who disagree carry very different signals. council-diff exposes the disagreement.\n\nFor 6 built-in domains:\n\n- **founder**: YC Partner / VC Skeptic / Lawyer / Indie CFO / Pragmatic Spouse\n- **engineer**: Rust Maintainer / SRE Oncall / Recruiter / Junior Dev / CTO 5y Later\n- **investor**: Macro / Sector / PM / Growth VC / Activist Short\n- **career**: Mentor 20y / Recruiter / Peer Doing Well / CSO / Future You 5y\n- **product**: Real User / Competitor / Internal Dev / Garry-style / Naval-style\n- **quant**: Jane Street MD / Citadel / Two Sigma ML / Anthropic / HFT Engineer\n\nPlus `custom` for fully user-defined rosters.\n\n## Live case studies — quartet covering the full GO → KILL spectrum\n\nFour real `council.deliberate()` fires from 2026-06. Same engine. Four different verdict shapes. The shape of the question determined the shape of the verdict — and the agreement score is the calibration signal.\n\n| Case | Domain | Verdict | Agreement | Voice spread | Example |\n|---|---|---|---|---|---|\n| [Crypto payments on B2B SaaS](https://github.com/alex-jb/council-for-slack-2026/blob/main/docs/case-studies/crypto-payments-2026-06.md) | founder | **KILL** | **0.94** | 4 → 12 (8 pt, tightest) | `examples/founder-crypto-payments.ts` |\n| [Annual billing at 2 months free](https://github.com/alex-jb/council-for-slack-2026/blob/main/docs/case-studies/annual-billing-2026-06.md) | founder | GO | 0.89 | 72 → 88 (16 pt) | `examples/founder-annual-billing.ts` |\n| [GOOGL Q3 2026 — Druckenmiller vs Berkshire](./docs/case-studies/googl-q3-2026.md) | investor | WAIT | 0.78 | 38 → 72 (34 pt) | `examples/investor.ts` |\n| [Rust rewrite of Python inference router](https://github.com/alex-jb/council-for-slack-2026/blob/main/docs/case-studies/rust-rewrite-2026-06.md) | engineer | WAIT | 0.62 | 22 → 72 (50 pt, widest) | `examples/engineer-rust-rewrite.ts` |\n\nAgreement range: **0.62 → 0.94**. Voice spread range: **8 → 50 points**. Verdicts span GO / WAIT / KILL.\n\nReproduce any of these locally:\n\n```bash\nANTHROPIC_API_KEY=sk-... npx tsx examples/founder-crypto-payments.ts\n```\n\nEach fire ~$0.03, ~10s. **The verdicts were not chosen for variety** — the quartet was selected before firing for question shape diversity. The fact that the cleanest convergence (0.94 agreement, 8-pt spread) is on a KILL is itself the calibration claim: when every framing converges against, the council collapses tighter than for any single GO.\n\n## Install\n\n```bash\n# npm (TypeScript / Node)\nnpm install council-diff\n\n# skills.sh — agent-agnostic distribution, supports 71 AI agent platforms\n# (Amp, Antigravity, Antigravity CLI, Claude Code, Cline, Codex, Cursor,\n#  Deep Agents, Gemini CLI, GitHub Copilot, Kimi Code CLI, Open Code,\n#  Warp, Zed, +57 more)\nnpx skills add alex-jb/council-diff\n```\n\n## Quickstart\n\n```ts\nimport { CouncilDiff } from \"council-diff\";\n\nconst council = new CouncilDiff({ apiKey: process.env.ANTHROPIC_API_KEY });\n\nconst result = await council.deliberate({\n  domain: \"founder\",\n  decision: \"Should I raise a $1M seed or bootstrap?\",\n  context: \"B2B SaaS, $5K MRR, growing 20% MoM, solo founder, 12 months runway\",\n});\n\nconsole.log(result.recommendation);  // \"go\" | \"wait\" | \"kill\" | \"split\"\nconsole.log(result.agreement_score); // 0-1, how much voices agree\nconsole.log(result.consensus);       // 1-paragraph synthesis\n\nfor (const v of result.voices) {\n  console.log(`${v.voice_display} (${v.score}/100): ${v.verdict}`);\n  console.log(`  + ${v.strength}`);\n  console.log(`  - ${v.gap}`);\n}\n```\n\n## Oracle mode (Fable 5)\n\nFor hard calls, split councils, or anywhere you want a flagship-tier second opinion, opt into Oracle:\n\n```ts\nconst result = await council.deliberate({\n  domain: \"founder\",\n  decision: \"Ship hosted SaaS at $29/mo or stay OSS-only?\",\n  context: \"11-agent OSS stack, ~50 stars, 3 paying customers begging for managed\",\n  oracle: \"fable-5\",  // opt-in\n});\n\nconsole.log(result.recommendation);          // council vote\nconsole.log(result.oracle?.recommendation);  // Fable 5's vote\nconsole.log(result.oracle?.verdict);         // 2-3 sentences naming which voices it sided with\nconsole.log(result.oracle?.override_reason); // only set if Oracle disagrees with the council\n```\n\nCouncil deliberates first (Sonnet 4.6, 5 voices, ~$0.03). Then Fable 5 reads all 5 verdicts and the consensus, weighs them with the full Mythos-class reasoning budget, and either ratifies the council or overrides with reason. Adds ~$0.05-0.08 per call.\n\nUse it when the decision matters enough that paying for a second model with override authority is rational. Skip it for routine deliberations.\n\nTry it: `ANTHROPIC_API_KEY=... npm run example:oracle`\n\n### Data retention disclosure\n\nAnthropic enforces a **30-day server-side data retention policy on Mythos-class models** (Claude Fable 5, Opus 4.7-Mythos) per [their support article](https://support.claude.com/en/articles/15425996-data-retention-practices-for-mythos-class-models). The 5-voice base council uses Sonnet 4.6, which is **zero-retention** under standard enterprise terms.\n\nEvery Oracle response in v0.3.1+ includes the actual posture:\n\n```ts\nresult.oracle?.data_retention  // \"30day-mythos\" or \"zero\"\nresult.oracle?.downgraded      // true if safeMode forced the downgrade\n```\n\nIf your application has any privacy claim that conflicts with 30-day retention (mental-health journaling, \"on-device 零上传\" marketing copy, GDPR-sensitive PII, sealed business decisions), pass `safeMode: true` and Oracle silently downgrades to Sonnet 4.6:\n\n```ts\nconst council = new CouncilDiff({ safeMode: true });\n\nconst result = await council.deliberate({\n  domain: \"founder\",\n  decision: \"...\",\n  oracle: \"fable-5\",  // requested\n});\n\nresult.oracle?.model         // \"claude-sonnet-4-6\", actually ran\nresult.oracle?.downgraded    // true\nresult.oracle?.data_retention // \"zero\"\n```\n\nThis disclosure is not optional. council-diff's positioning is calibration honesty. Shipping a Mythos route without surfacing the retention boundary undermines the whole point.\n\n## Custom voices\n\n```ts\nconst result = await council.deliberate({\n  domain: \"custom\",\n  decision: \"Use Postgres or DynamoDB for this new service?\",\n  context: \"10K writes/sec peak, eventual consistency OK, team knows SQL well\",\n  custom_voices: [\n    { slug: \"dba\", display: \"Postgres DBA\", role_brief: \"Decades of OLTP. Bias: PG fits 95% of workloads.\" },\n    { slug: \"aws_se\", display: \"AWS Solutions Engineer\", role_brief: \"DynamoDB enthusiast. Bias: serverless \u003e self-managed.\" },\n    { slug: \"kafka_dev\", display: \"Kafka Streams Dev\", role_brief: \"Event-sourcing lens. Bias: write log + project to either.\" },\n    { slug: \"cost_eng\", display: \"Cost Engineer\", role_brief: \"Watches the bill. Bias: serverless costs 5x at scale.\" },\n    { slug: \"former_cto\", display: \"Former CTO with 3 migrations\", role_brief: \"Has done both migrations. Bias: stay where the team is fluent.\" },\n  ],\n});\n```\n\n## Output schema\n\n```ts\ninterface CouncilResult {\n  domain: CouncilDomain;\n  decision: string;\n  voices: {\n    voice: string;          // slug\n    voice_display: string;  // human-readable\n    score: number;          // 0-100, how strongly they support\n    verdict: string;        // 1-2 sentences\n    strength: string;       // strongest supporting signal\n    gap: string;            // biggest risk / counter\n  }[];\n  consensus: string;            // 1-paragraph synthesis (60-100 words)\n  agreement_score: number;      // 0-1, 1 = unanimous, 0 = split\n  recommendation: \"go\" | \"wait\" | \"kill\" | \"split\";\n  computed_at: string;          // ISO timestamp\n  oracle?: {                    // present only when oracle: \"fable-5\" was passed\n    model: string;              // e.g. \"claude-fable-5\"\n    recommendation: \"go\" | \"wait\" | \"kill\" | \"split\";\n    score: number;              // 0-100\n    verdict: string;            // 2-3 sentences\n    override_reason?: string;   // set when Oracle disagrees with council consensus\n    data_retention?: \"30day-mythos\" | \"zero\";\n    downgraded?: boolean;       // true if safeMode forced Sonnet 4.6\n  };\n}\n```\n\n## Cost\n\n- **Council only:** one Claude Sonnet 4.6 call per deliberation. ~$0.02-0.04 per call depending on context length.\n- **Council + Oracle (`oracle: \"fable-5\"`):** add one Claude Fable 5 call. ~$0.05-0.08 extra. Total ~$0.07-0.12 per Oracle deliberation.\n\n## Brier audit: the evaluation loop\n\nThis is the part that makes council-diff a Software 3.0 artifact rather than a chat prompt wrapper. Every deliberation can be logged at decision time and scored at resolution time. The Brier score (0 perfect, 1 maximally wrong, 0.25 random coin-flip) tells you whether the council is calibrated or just opinionated.\n\n```ts\nimport { addPrediction, resolvePrediction, brierScore, meanBrier } from \"council-diff/brier\";\n\n// At deliberation time:\nconst pred = addPrediction({\n  decision: result.decision,\n  domain: result.domain,\n  recommendation: result.recommendation,\n  agreement_score: result.agreement_score,\n  voice_scores: result.voices.map((v) =\u003e v.score),\n  resolve_by: \"2027-06-09\",  // 12mo from now\n});\n// Persist `pred` to your storage of choice (JSONL, SQLite, Postgres).\n\n// At resolution time (when the outcome is known):\nconst resolved = resolvePrediction(pred, { outcome: \"go-was-right\" });\nconst score = brierScore(resolved);  // 0 = perfect, 1 = max wrong, 0.25 = random\n\n// Aggregate over many resolutions:\nconst audit = meanBrier(allResolvedPreds);\nconsole.log(audit.edge_vs_random);  // positive = council adds calibration value\n```\n\nSee `src/brier.ts` for `predictedProbability` math + persistence-agnostic interface. Oracle calls are Brier-audited separately so you can see when the Oracle beats the council and when it underperforms.\n\n## How this differs from existing tools\n\n**Perplexity Model Council (Feb 2026)** is a closed UI feature that compares the *same question* across *different providers* (GPT-5.2 vs Claude 4.6 vs Gemini side-by-side). council-diff is an OSS library that compares *different personas* against *the same model*, with a Fable 5 Oracle adjudicating, and a Brier audit module that scores every voice over 30 / 90 days. Persona-of-the-judge instead of provider-of-the-judge. Closes the loop with reality.\n\n**Anthropic shipped \"advisor strategy\" (beta) on 2026-06-09.** Same pattern, 6 months later. Anthropic's June 2026 Skills release added an \"advisor strategy\" mode where agents consult an advisor model before deciding. That is literally what council-diff has shipped since v0.3.0: the 5-voice council consults, the Fable 5 Oracle adjudicates, the Brier audit closes the loop. The pattern was 6 months early and is still the only OSS implementation that ships the calibration layer underneath. Pairs with the official Skills standard cleanly. Drop `council-diff` into any `.claude/skills/` directory and it auto-loads.\n\n## Pattern source\n\n- [Karpathy, Sequoia AI Ascent, April 20, 2026](https://www.youtube.com/results?search_query=karpathy+sequoia+ai+ascent+software+3.0). Software 3.0 framing + agentic engineering job description.\n- [Perplexity Model Council UI](https://www.perplexity.ai/hub/blog/perplexity-model-council)\n- [Orallexa multi-agent debate](https://github.com/alex-jb/orallexa-ai-trading-agent)\n- Cohere Command A+ grounding citation pattern (sources cited inline as `[src:...]`)\n\n## Roadmap\n\n- [x] Brier audit math (v0.2)\n- [x] Fable 5 Oracle adjudication (v0.3)\n- [x] Data retention disclosure + safeMode (v0.3.1)\n- [x] Karpathy Software 3.0 positioning (v0.4)\n- [ ] **v0.5 TREX — execution-before-review** ([spec](docs/v0.5-trex-execution-spec.md)): Docker DinD sandbox runs candidate diff against repo tests, the 5 voices then review against *observed* test output, not just the diff. Kills the \"looks correct but breaks main\" failure mode.\n- [ ] Public Brier leaderboard at council.alex-jb.com\n- [ ] Streaming voice-by-voice output for UI\n- [ ] Python port (`pip install council-diff`)\n- [ ] CLI: `council \"should I quit my job\" --domain career`\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falex-jb%2Fcouncil-diff","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falex-jb%2Fcouncil-diff","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falex-jb%2Fcouncil-diff/lists"}