{"id":51572924,"url":"https://github.com/dshakes/firstpass","last_synced_at":"2026-07-10T21:02:36.044Z","repository":{"id":370373968,"uuid":"1293805949","full_name":"dshakes/firstpass","owner":"dshakes","description":"Route every LLM request to the cheapest model that provably passes your quality gate — with a signed, tamper-evident receipt for every decision. Proof over prediction.","archived":false,"fork":false,"pushed_at":"2026-07-09T02:58:47.000Z","size":211,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-09T03:06:09.935Z","etag":null,"topics":["agents","ai-agents","cost-optimization","inference","llm","llm-router","observability","proxy","reliability","rust"],"latest_commit_sha":null,"homepage":"https://dshakes.github.io/firstpass","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dshakes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-07-08T15:39:57.000Z","updated_at":"2026-07-09T02:58:22.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dshakes/firstpass","commit_stats":null,"previous_names":["dshakes/firstpass"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/dshakes/firstpass","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dshakes%2Ffirstpass","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dshakes%2Ffirstpass/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dshakes%2Ffirstpass/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dshakes%2Ffirstpass/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dshakes","download_url":"https://codeload.github.com/dshakes/firstpass/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dshakes%2Ffirstpass/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35343136,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agents","ai-agents","cost-optimization","inference","llm","llm-router","observability","proxy","reliability","rust"],"created_at":"2026-07-10T21:02:35.272Z","updated_at":"2026-07-10T21:02:36.038Z","avatar_url":"https://github.com/dshakes.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"assets/hero.svg\" alt=\"Firstpass — the cheapest model takes the first pass, proven not guessed\" width=\"840\"\u003e\n\n# Firstpass\n\n**Route every LLM request to the cheapest model that _provably_ passes your quality gate — and get a signed receipt for the decision.**\n\nProof over prediction. Built for agent fleets.\n\n[![CI](https://github.com/dshakes/firstpass/actions/workflows/ci.yml/badge.svg)](https://github.com/dshakes/firstpass/actions/workflows/ci.yml)\n[![license](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)\n[![spec](https://img.shields.io/badge/spec-v0.1-informational)](SPEC.md)\n[![site](https://img.shields.io/badge/site-live-19E3B1)](https://dshakes.github.io/firstpass)\n\n**[Website](https://dshakes.github.io/firstpass)** · [Quickstart](#quickstart) · [Install](#install) · [How it works](#how-it-works) · [Configure](#configuration) · [SPEC](SPEC.md)\n\n\u003c/div\u003e\n\n---\n\nFirstpass is a **drop-in, Anthropic-compatible proxy**. Point your agent's `base_url` at it and every request is routed to the cheapest model first, its **real output** is checked by a gate you define (tests, typecheck, schema, a judge), and it escalates one rung only when the gate fails — writing a tamper-evident audit trace for every decision.\n\n\u003e **Honestly scoped.** The proxy routes, gates, escalates, fails over, audits, and learns end-to-end over real HTTP — no test doubles in the plane, and the enforce path is [live-verified](#proof-not-adjectives) against real Anthropic. The [proof harness](#proof-not-adjectives) has a **200-task live result** below; still ahead: a coding-with-tests benchmark (to earn the conformal guarantee), a 30-day dogfood, and prebuilt binaries. See the [roadmap](#roadmap). Nothing here is claimed as measured that isn't.\n\n## Quickstart\n\nSee the whole loop in ~10 seconds — **no API keys**:\n\n```bash\ngit clone https://github.com/dshakes/firstpass \u0026\u0026 cd firstpass\ncargo run -p firstpass-proxy --example demo\n```\n\nIt stands up a mock upstream and the proxy, drives one real decision (cheap model fails the gate → escalates → passes), prints the [receipt](#the-receipt), submits feedback, and re-verifies the sealed chain.\n\n**Then use it in front of your own agent:**\n\n```bash\nfirstpass-proxy                                    # 1. run it (observe mode: logs, changes nothing)\nexport ANTHROPIC_BASE_URL=\"http://127.0.0.1:8080\"  # 2. point your agent at it — same wire format\n#   … use your agent normally; Firstpass records a receipt per call …\nunset ANTHROPIC_BASE_URL                            # 3. offboard anytime — one env var\n```\n\nTo turn on cheapest-first routing + gating, copy [`firstpass.example.toml`](firstpass.example.toml) and run in enforce mode — see [Configuration](#configuration).\n\n## Install\n\n| Method | Command |\n| --- | --- |\n| **curl \\| sh** | `curl --proto '=https' --tlsv1.2 -LsSf https://github.com/dshakes/firstpass/releases/latest/download/firstpass-proxy-installer.sh \\| sh` |\n| **Docker** | `docker run -p 8080:8080 -e FIRSTPASS_BIND=0.0.0.0:8080 ghcr.io/dshakes/firstpass:latest` |\n| **From source** | `cargo install --git https://github.com/dshakes/firstpass firstpass-proxy` |\n\n\u003e The `curl \\| sh` installer and prebuilt binaries (macOS · Linux · Windows, all checksummed) are produced by [cargo-dist](https://opensource.axo.dev/cargo-dist/) and attach to each [GitHub Release](https://github.com/dshakes/firstpass/releases). The container image is published to [GHCR](https://github.com/dshakes/firstpass/pkgs/container/firstpass) on every push to `main`. Homebrew is planned (needs a formula tap). `cargo install --git` and Docker work today.\n\n## Prediction vs. proof\n\n\u003cdiv align=\"center\"\u003e\u003cimg src=\"assets/compare.svg\" alt=\"Prediction guesses a model up front and never checks; Firstpass sends cheap, gates the real output, escalates only on failure, and emits a receipt\" width=\"880\"\u003e\u003c/div\u003e\n\nModel routers on the market route by **prediction** — a learned policy guesses which model will answer well, sends the request there, and never checks. Firstpass routes by **proof**: it runs a real gate on the real output, and the decision is a record you can audit — not a guess you hope was right.\n\n## How it works\n\n\u003cdiv align=\"center\"\u003e\u003cimg src=\"assets/how.svg\" alt=\"A request hits the cheapest rung, fails the gate, escalates one rung, passes, and is served — every decision logged to the audit trace\" width=\"840\"\u003e\u003c/div\u003e\n\n1. **Route** to the cheapest rung of a declarative ladder. BYOK — your keys pass through, redacted from every log.\n2. **Gate** the real output: inline (non-empty, JSON, [JSON-Schema](SPEC.md)) or subprocess plugins (your tests/linter/judge) that read the candidate on **stdin, never argv** — injection-resistant. Per-gate error budgets auto-disable a flaky gate.\n3. **Escalate** exactly one rung on gate failure — budget-capped; a provider 5xx fails over cross-provider. Serve the first output that passes, and record everything.\n\n## The receipt\n\nNot a dashboard number — a decision you can audit. Every call becomes a hash-chained JSON trace an external auditor can re-derive.\n\n\u003cdiv align=\"center\"\u003e\u003cimg src=\"assets/receipt.svg\" alt=\"Firstpass audit receipt: the cheap model failed the test gate, the next rung passed, served at a real saving versus always-top-tier\" width=\"620\"\u003e\u003c/div\u003e\n\n```jsonc\n{\n  \"trace_id\": \"0192f3a1-7c4e-7abc-9d21-4e8b1f0a2c33\",\n  \"prev_hash\": \"9f2c…a1b7\",                         // chains to the previous decision — tamper-evident\n  \"mode\": \"enforce\",\n  \"attempts\": [\n    { \"rung\": 0, \"model\": \"anthropic/claude-haiku-4-5\", \"cost_usd\": 0.0007,\n      \"gates\": [{ \"gate_id\": \"cargo-test\", \"verdict\": \"fail\", \"score\": 0.0 }],\n      \"verdict\": \"fail\" },                           // cheap model tried first — the test gate caught it\n    { \"rung\": 1, \"model\": \"anthropic/claude-sonnet-5\", \"cost_usd\": 0.0121,\n      \"gates\": [{ \"gate_id\": \"cargo-test\", \"verdict\": \"pass\", \"score\": 1.0 }],\n      \"verdict\": \"pass\" }                            // escalated one rung, proven to pass, served\n  ],\n  \"final\": { \"served_rung\": 1, \"total_cost_usd\": 0.0128,\n             \"counterfactual_baseline_usd\": 0.0630, \"savings_usd\": 0.0502 }\n}\n```\n\nPoint at any request and answer *why did this go to that model, and what did it cost* — and re-derive the hash chain yourself. Downstream outcomes flow back via [`POST /v1/feedback`](SPEC.md) onto a deferred-verdict side table that never alters the sealed record.\n\n## Configuration\n\nFirstpass is configured through the environment (12-factor) — run `firstpass-proxy --help` for the full reference:\n\n| Variable | Purpose | Default |\n| --- | --- | --- |\n| `FIRSTPASS_MODE` | `observe` \\| `enforce` | `observe` |\n| `FIRSTPASS_BIND` | listen address | `127.0.0.1:8080` |\n| `FIRSTPASS_CONFIG` | path to `firstpass.toml` (routes, ladders, gates) | — |\n| `FIRSTPASS_DB` | trace store path | `firstpass.db` |\n| `FIRSTPASS_UPSTREAM_ANTHROPIC` | upstream base URL | `https://api.anthropic.com` |\n| `FIRSTPASS_UPSTREAM_OPENAI` | upstream base URL | `https://api.openai.com` |\n\nRouting itself is declarative TOML — routes → mode, a cheapest-first model ladder, and the gates that must pass. Start from [`firstpass.example.toml`](firstpass.example.toml):\n\n```bash\ncp firstpass.example.toml firstpass.toml\nFIRSTPASS_MODE=enforce FIRSTPASS_CONFIG=./firstpass.toml firstpass-proxy\n```\n\n**Endpoints:** `POST /v1/messages` (drop-in) · `POST /v1/feedback` · `GET /v1/capabilities` · `GET /healthz`.\n\n## Proof, not adjectives\n\n\u003cdiv align=\"center\"\u003e\u003cimg src=\"assets/proof.svg\" alt=\"Proof-harness simulation: ~65% cheaper at equal-or-higher success, 0.16 vs 0.46 served-failure versus a predictive router, ≤10% conformal served-failure at 95% confidence, kill criterion reads PROCEED\" width=\"880\"\u003e\u003c/div\u003e\n\nThe harness (`cargo run -p firstpass-bench`) runs the full methodology — baselines, bootstrap confidence intervals, a split-conformal served-failure guarantee, and a **pre-registered kill criterion** that says *stop* if the thesis fails — behind real-backend trait seams. `--live` swaps the simulated backend for real providers (BYOK).\n\n**Live — real Anthropic, 200 graded verifiable tasks** (`--live`, difficulty-graded arithmetic; reproducible, deterministic):\n\n| policy | success | $/success | served-failure |\n|---|---|---|---|\n| always-cheap (Haiku) | 0.62 `[0.55, 0.69]` | $0.0001 | 0.38 |\n| always-top (Opus) | 0.98 `[0.96, 0.99]` | $0.0023 | 0.02 |\n| predictive router | 0.88 | $0.0007 | 0.12 |\n| **firstpass** | **1.00** `[1.00, 1.00]` | **$0.0003** | **0.00** |\n\nFirstpass served at **~85% lower $/success than always-top, at parity-or-better quality**, with **served-failure 0.00 vs the predictive router's 0.12** — verification catches what prediction serves blind. The cheap tier cleared 62% of tasks against a 13% break-even, so **the pre-registered kill criterion reads PROCEED**.\n\n**Reproduced with a real LLM-judge gate, not just a deterministic checker** (`FIRSTPASS_GATE=judge`, Sonnet grading each answer *without seeing the ground truth*, n=200): firstpass **1.00 success, ~84% cheaper than Opus, served-failure 0.00, PROCEED** — the win holds with the gate you'd actually deploy.\n\nStated plainly on the one thing this does **not** show: the **conformal served-failure guarantee stays degenerate**. Three live experiments pinned down why — on *self-checking* tasks the gate is either near-perfect (nothing to bound) or, if you weaken the judge, it false-rejects correct answers and the cost advantage flips. Earning the guarantee needs a domain where the *best practical* gate is still imperfect — a **coding-with-tests benchmark** (real test suites have coverage gaps), which requires a sandbox for untrusted code. That's the next milestone, honestly scoped, not a claim made today.\n\n## Roadmap\n\n- **M0 ✓** — proof harness: baselines, bootstrap CIs, conformal guarantee, pre-registered kill criterion.\n- **M1 ✓** — Rust proxy: Anthropic + OpenAI clients, observe **and** enforce, escalation, cross-provider failover, SQLite trace store — over real HTTP.\n- **M2 ✓** — gate framework: subprocess plugins, inline + schema gates, **native LLM-judge gate** (maker≠checker, candidate-as-data), error-budget auto-disable, feedback API + deferred verdicts.\n- **M2.5 ✓** — real-traffic proxy: **SSE streaming passthrough**, tool/multimodal-safe enforce; **`firstpass` CLI** (`up` / `doctor` / `trace`) + **MCP server**; live-provider proof harness (`--live`).\n- **M3 →** — 200-task live benchmark ✓ (cost/success proof at scale); still ahead: a **coding-with-tests benchmark** (a real *imperfect* gate, to earn the conformal guarantee), a 30-day self-host dogfood GA, and published binaries + Homebrew tap.\n- **Hosted GA →** — multi-tenant control plane, BYOK KMS envelope encryption, sandboxed gate execution — designed in [ADR 0001](docs/adr/0001-hosted-ga-architecture.md), gated behind dogfood GA.\n\n## Links\n\n[Website](https://dshakes.github.io/firstpass) · [SPEC](SPEC.md) · [Example config](firstpass.example.toml) · [Agent guide](AGENTS.md) · [llms.txt](llms.txt) · [License](LICENSE)\n\n\u003cdiv align=\"center\"\u003e\u003csub\u003eProof over prediction. No competitor products named — Firstpass competes on evidence, not adjectives.\u003c/sub\u003e\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdshakes%2Ffirstpass","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdshakes%2Ffirstpass","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdshakes%2Ffirstpass/lists"}