{"id":51520100,"url":"https://github.com/gordonmurray/proofgate","last_synced_at":"2026-07-08T15:01:46.599Z","repository":{"id":369690537,"uuid":"1290887095","full_name":"gordonmurray/proofgate","owner":"gordonmurray","description":"A reference AWS retrieval system, defined in Terraform, whose CI pipeline fails the build when retrieval quality (recall@10) or latency (p99) regress against a versioned golden set.","archived":false,"fork":false,"pushed_at":"2026-07-06T12:38:12.000Z","size":80,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-06T14:05:33.207Z","etag":null,"topics":["aws","ci-cd","ecs-fargate","evaluation","fastapi","information-retrieval","infrastructure-as-code","mlops","opentelemetry","retrieval-augmented-generation","terraform","vector-search"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gordonmurray.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-07-06T10:11:09.000Z","updated_at":"2026-07-06T12:37:43.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/gordonmurray/proofgate","commit_stats":null,"previous_names":["gordonmurray/neve"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/gordonmurray/proofgate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gordonmurray%2Fproofgate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gordonmurray%2Fproofgate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gordonmurray%2Fproofgate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gordonmurray%2Fproofgate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gordonmurray","download_url":"https://codeload.github.com/gordonmurray/proofgate/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gordonmurray%2Fproofgate/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35268549,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-08T02:00:06.796Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","ci-cd","ecs-fargate","evaluation","fastapi","information-retrieval","infrastructure-as-code","mlops","opentelemetry","retrieval-augmented-generation","terraform","vector-search"],"created_at":"2026-07-08T15:01:43.461Z","updated_at":"2026-07-08T15:01:46.587Z","avatar_url":"https://github.com/gordonmurray.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# proofgate\n\nproofgate is a reference deployment of a retrieval system on AWS, defined in\nTerraform and shipped through a gated CI pipeline. The pipeline includes an\nevaluation step that runs a versioned golden set against the deployed service and\nfails the build if retrieval quality or latency falls outside configured thresholds.\nA regression therefore cannot be promoted to production.\n\nThe motivation is simple. Most infrastructure is either up or down, and a health\ncheck is enough to decide whether to ship. A retrieval system can pass every health\ncheck and still return poor results, so proofgate adds a quality check to the\npipeline alongside the usual format, policy, build, and deploy steps.\n\nproofgate is also the harness used to develop and validate Firn, the S3-backed\nvector store on the request path.\n\n## Example\n\nThe evaluation step runs the golden set against a live endpoint and compares recall\nand latency to the thresholds. Running it against a deliberately broken corpus makes\nit fail, which blocks promotion:\n\n```\n=== proofgate eval gate — http://127.0.0.1:8080 ===\nrecall@10: 0.3800  (baseline 0.9, floor 0.8)\np99: 210.85ms  p50: 127.32ms  (200 reqs)\n  [FAIL] recall_floor: recall@10 0.3800 must be \u003e= floor 0.8\n  [FAIL] recall_drop_vs_baseline: drop 0.5200 vs baseline 0.9 must be \u003c= 0.05\n  [PASS] p99_latency_slo: p99 210.85ms must be \u003c= SLO 750.0ms\n\nEVAL GATE FAILED — promotion blocked.   # exit 1\n```\n\nThe service stays healthy throughout; only recall drops. A full walkthrough is in\n[docs/eval-gate-demo.md](docs/eval-gate-demo.md).\n\n## Architecture\n\nThe system is split into three Terraform modules.\n\nRuntime (the request path):\n\n```\nClient -\u003e ALB -\u003e ECS Fargate: retrieval-api\n   |- embed query  -\u003e local hashing embedder (default)  or  Bedrock (optional)\n   |- vector + FTS -\u003e Firn on S3 (foyer cache is internal to Firn)\n   |- rerank       -\u003e identity (default)  or  Bedrock (optional)\n -\u003e ranked response\n```\n\nThere is no separate cache tier. Caching is handled inside Firn (foyer), so the\ncache-hit rate is a property of the store rather than a bolted-on service.\n\nDelivery (GitHub Actions):\n\n```\n1 validate/fmt   ruff + terraform fmt/validate\n2 policy scan    checkov\n3 build          docker image\n4 apply          staging (guarded on AWS credentials)\n5 eval gate      recall@10 + p99 SLO\n6 promote        prod, only if the eval gate passes\n```\n\nObservability:\n\n```\nretrieval-api (OTel SDK) -\u003e OTel Collector -\u003e Amazon Managed Prometheus -\u003e Grafana\n                                           -\u003e Tempo or FirnTel\n```\n\nEval results are exported as metrics, so recall and latency can be tracked over time\nrather than only read from CI logs.\n\n## Repository layout\n\n```\nproofgate/\n  services/retrieval-api/   FastAPI orchestration service, instrumented with OpenTelemetry\n  eval/                     golden set, thresholds, baseline, and the evaluation harness\n  terraform/                three modules: runtime, delivery, observability\n  .github/workflows/        the CI pipeline\n  docs/                     design notes and the eval walkthrough\n```\n\n## Quickstart\n\nRequires Python 3.11 or newer. The Terraform and checkov commands run via Docker;\nsee [docs/DESIGN.md](docs/DESIGN.md) for the tooling setup.\n\n```bash\nmake install          # install retrieval-api and dev dependencies\nmake test             # unit tests (service and harness)\nmake lint             # ruff\n\n# Run the service and the eval gate against it:\nmake serve \u0026          # http://127.0.0.1:8080\nmake eval             # runs the golden set and reports pass/fail\n\n# Infrastructure gates:\nmake tf-validate\nmake policy-scan\n```\n\nQuery the running service:\n\n```bash\ncurl -s localhost:8080/search \\\n  -H 'content-type: application/json' \\\n  -d '{\"query\": \"what is the foyer cache\", \"k\": 3}' | jq\n```\n\n## The eval gate\n\n- **Golden set** ([`eval/golden.jsonl`](eval/golden.jsonl)): 50 queries against the\n  demo corpus, versioned and reviewed like code, one `{query, relevant_ids, notes}`\n  object per line.\n- **Metrics**: macro recall@10 against the golden set, and p99 latency from a fixed,\n  repeatable warm load run.\n- **Thresholds** ([`eval/thresholds.toml`](eval/thresholds.toml)): an absolute recall\n  floor, a maximum allowed drop from the committed\n  [`baseline.json`](eval/baseline.json), and the p99 SLO in milliseconds. The baseline\n  is only changed through a deliberate pull request; it is not ratcheted automatically\n  from a passing run.\n- **Mechanics**: the harness runs the golden set over the real HTTP path, writes\n  `eval-report.json`, and exits non-zero on any breach. The promote job depends on it.\n\n## Configuration\n\nThe same image runs in staging and production; behaviour comes from the environment.\n\n| Variable | Default | Purpose |\n| --- | --- | --- |\n| `PROOFGATE_CORPUS_PATH` | bundled `demo.jsonl` | Corpus to index at startup |\n| `PROOFGATE_EMBEDDER` | `local` | Embedder: `local` or `bedrock` |\n| `PROOFGATE_RERANKER` | `none` | Reranker: `none` or `bedrock` |\n| `PROOFGATE_ENV` | `local` | Environment label |\n| `OTEL_EXPORTER_OTLP_ENDPOINT` | *(unset)* | When set, spans are exported over OTLP |\n\nTwo choices are left configurable rather than fixed in code: Bedrock versus\nself-hosted inference, and Tempo versus FirnTel for traces.\n\n## Status\n\nThe current code runs retrieval-api against a local corpus, with the eval gate live\nand able to fail, the three Terraform modules validating, and the pipeline green in\nGitHub Actions. Bedrock embedding and reranking are configurable but not yet\nimplemented; the default embedder is the local one.\n\nDesign notes, conventions, and guardrails are in [docs/DESIGN.md](docs/DESIGN.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgordonmurray%2Fproofgate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgordonmurray%2Fproofgate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgordonmurray%2Fproofgate/lists"}