{"id":51336925,"url":"https://github.com/dpguthrie/braintrust-eval-gate","last_synced_at":"2026-07-02T03:32:36.926Z","repository":{"id":367618275,"uuid":"1281580930","full_name":"dpguthrie/braintrust-eval-gate","owner":"dpguthrie","description":"Example: gate GitHub PR merges on Braintrust eval thresholds via a required status check","archived":false,"fork":false,"pushed_at":"2026-06-26T18:14:44.000Z","size":11,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-26T20:10:46.685Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpguthrie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-26T17:45:56.000Z","updated_at":"2026-06-26T18:14:49.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dpguthrie/braintrust-eval-gate","commit_stats":null,"previous_names":["dpguthrie/braintrust-eval-gate"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/dpguthrie/braintrust-eval-gate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-eval-gate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-eval-gate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-eval-gate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-eval-gate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpguthrie","download_url":"https://codeload.github.com/dpguthrie/braintrust-eval-gate/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-eval-gate/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35032145,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-02T02:00:06.368Z","response_time":173,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-02T03:32:36.185Z","updated_at":"2026-07-02T03:32:36.895Z","avatar_url":"https://github.com/dpguthrie.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Braintrust eval gate — example\n\nA minimal, working example of using **Braintrust evals as a required GitHub status check**: a PR can't merge unless the eval clears defined quality thresholds — the same SDLC pattern as \"all unit tests pass\" or \"vulnerability scan clean,\" but for AI/eval performance.\n\nIt uses **only what the Braintrust SDK and GitHub Actions already provide** — no custom framework to install. The entire gate is one `Reporter` you define in your own eval file.\n\n---\n\n## How it works (the three facts that make this work)\n\n1. **A GitHub required status check is just a CI job that must exit `0`.** A GitHub Actions job surfaces as a check; if it exits non-zero, the check fails; branch protection can require it to be green before merge.\n2. **`braintrust eval` exits non-zero only when an eval *errors* — not when a score is *low*.** So out of the box, a regression to a terrible-but-non-erroring score still passes. That's the gap this example closes.\n3. **A custom `Reporter` controls the exit code — natively.** Braintrust's SDK lets you define a `Reporter` with `report_run(...)`; `braintrust eval` runs it and uses its return value as the process exit code. Return `False` → non-zero exit → job fails → required check blocks the merge. This is a documented SDK feature, not extra tooling.\n\n```\nPR opened ─▶ GitHub Action runs `braintrust eval`\n                 │\n                 ├─ runs your evals, computes scores + metrics\n                 ├─ your Reporter checks them against your thresholds\n                 └─ violation? report_run -\u003e False -\u003e exit 1\n                          │\n                          ▼\n              job \"eval-gate\" fails ─▶ required status check red ─▶ merge blocked\n```\n\n## Repo layout\n\n```\nbraintrust-eval-gate/\n├── evals/\n│   └── say_hi.eval.py              # the eval AND the gate Reporter (all in one file)\n├── .github/workflows/eval-gate.yml # CI job that runs `braintrust eval`\n└── requirements.txt                # braintrust, autoevals — nothing custom\n```\n\n## The gate is just a `Reporter`\n\nThis is the whole mechanism — copy it into your own eval file and edit the thresholds. It imports nothing but the Braintrust SDK:\n\n```python\nfrom braintrust import Eval, Reporter\nfrom autoevals import Levenshtein\n\nSCORE_FLOORS = {\"Levenshtein\": 0.60}      # minimum average score (0..1)\nMAX_SCORE_REGRESSION = 0.05               # max drop vs baseline (0.05 = 5pp)\nMETRIC_CEILINGS = {\"duration\": 5.0}       # absolute ceilings (e.g. seconds)\n\ndef report_eval(evaluator, result, **kwargs):\n    name = getattr(evaluator, \"eval_name\", \"eval\")\n    s_sum = result.summary\n    ok = True\n    for sname, s in s_sum.scores.items():\n        floor = SCORE_FLOORS.get(sname)\n        if floor is not None and s.score \u003c floor:\n            print(f\"::error::[{name}] {sname} {s.score:.3f} \u003c floor {floor}\"); ok = False\n        if s.diff is not None and s.diff \u003c -MAX_SCORE_REGRESSION:\n            print(f\"::error::[{name}] {sname} regressed {-s.diff*100:.1f}pp vs baseline\"); ok = False\n    for mname, m in s_sum.metrics.items():\n        ceiling = METRIC_CEILINGS.get(mname)\n        if ceiling is not None and m.metric \u003e ceiling:\n            print(f\"::error::[{name}] {mname} {m.metric:.2f}{m.unit} \u003e max {ceiling}{m.unit}\"); ok = False\n    print(s_sum)\n    return ok\n\ndef report_run(results, **kwargs):\n    return all(results)   # False here =\u003e non-zero exit =\u003e job fails =\u003e merge blocked\n\nReporter(\"eval-gate\", report_eval=report_eval, report_run=report_run)\n```\n\n- **`report_eval`** sees each evaluator's `summary.scores` (averages + `diff` vs baseline) and `summary.metrics` — apply whatever policy you want; it's plain Python.\n- **`report_run`** returns the overall pass/fail; its boolean *is* the exit code.\n- Define **one** `Reporter` and it applies to every `Eval` in the run automatically. To share it across many eval files, define it in any one file that's part of the run.\n\nSee [`evals/say_hi.eval.py`](evals/say_hi.eval.py) for the complete file (Reporter + a deterministic example `Eval`).\n\n## Run it locally\n\n```bash\npip install -r requirements.txt\nexport BRAINTRUST_API_KEY=...          # from https://www.braintrust.dev/app/settings/api-keys\nbraintrust eval evals\necho \"exit code: $?\"                    # 0 = passed the gate, 1 = blocked\n```\n\nThen **see it block**: raise a floor above the current score (e.g. `SCORE_FLOORS = {\"Levenshtein\": 0.99}`) and re-run — the command exits `1` and prints `::error::` annotations explaining why.\n\n## Wire it up as a required check on GitHub\n\n1. Push this repo to GitHub.\n2. Add a repo secret **`BRAINTRUST_API_KEY`** (Settings → Secrets and variables → Actions).\n3. Open a PR — the **`eval-gate`** job runs and posts a results comment.\n4. Make it blocking: **Settings → Branches → Branch protection rule** (or **Rulesets**) on `main` →\n   enable **\"Require status checks to pass before merging\"** → search for and select **`eval-gate`**.\n5. From now on, a PR whose eval drops below the policy can't be merged.\n\nThe workflow itself is just the official action:\n\n```yaml\n- uses: braintrustdata/eval-action@v1\n  with:\n    api_key: ${{ secrets.BRAINTRUST_API_KEY }}\n    runtime: python\n    paths: evals\n```\n\nThe action runs `braintrust eval`, which runs your `Reporter`. Nothing else is needed.\n\n## Two gotchas worth knowing\n\n- **A required check that never runs will hang the PR forever.** If you scope the workflow with `paths:` (only run when `agents/**` changes) but mark the check required globally, PRs that don't touch those paths wait on a check that never reports. Fixes: run on every PR (cheap), add a \"passthrough\" job that always reports, or use GitHub **Rulesets** with path conditions.\n- **The check name must be stable.** Branch protection matches by job name (`eval-gate` here). If you rename the job or change a matrix that alters the check name, the required check silently stops matching.\n\n## Adapting it to your project\n\n- Replace the example `Eval(...)` with your real one (Python or TypeScript — the `Reporter` API is identical in JS: `Reporter(\"name\", { reportEval, reportRun })`).\n- Edit `SCORE_FLOORS` / `MAX_SCORE_REGRESSION` / `METRIC_CEILINGS` to your bar. The scorer/metric names must match what your eval emits.\n- To enforce regression checks deterministically, set `base_experiment_name=` in your `Eval(...)` to your main-branch experiment so `diff` compares against a known baseline.\n\n## If your evals are too long/heavy to run inside CI\n\nThis example runs the eval **inside the GitHub runner**. For very long agentic evals (or ones needing private data/GPU), use the async pattern instead: create a pending check via GitHub's Checks API, run the eval off-runner, then PATCH the check run to `success`/`failure`.\n\n\u003e Note: this gate relies on the **SDK `braintrust eval` CLI** (what `braintrustdata/eval-action@v1` invokes). The `bt` CLI does not execute custom `Reporter`s, so the Reporter-based gate must run through `braintrust eval`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fbraintrust-eval-gate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpguthrie%2Fbraintrust-eval-gate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fbraintrust-eval-gate/lists"}