{"id":51336927,"url":"https://github.com/dpguthrie/braintrust-skills","last_synced_at":"2026-07-02T03:32:39.017Z","repository":{"id":353441745,"uuid":"1218657590","full_name":"dpguthrie/braintrust-skills","owner":"dpguthrie","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-02T04:14:41.000Z","size":137,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-02T06:20:14.882Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpguthrie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-23T05:00:09.000Z","updated_at":"2026-05-02T04:14:45.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dpguthrie/braintrust-skills","commit_stats":null,"previous_names":["dpguthrie/braintrust-flywheel-plugin","dpguthrie/braintrust-skills"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dpguthrie/braintrust-skills","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-skills","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-skills/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-skills/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-skills/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpguthrie","download_url":"https://codeload.github.com/dpguthrie/braintrust-skills/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-skills/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35032145,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-02T02:00:06.368Z","response_time":173,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-02T03:32:36.195Z","updated_at":"2026-07-02T03:32:38.865Z","avatar_url":"https://github.com/dpguthrie.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Braintrust Skills\n\nA collection of agent skills for working with [Braintrust](https://braintrust.dev) through the `bt` CLI and repository-local coding-agent workflows.\n\n## Available Skills\n\n| Skill | Purpose |\n|---|---|\n| `bt-flywheel` | Continuously improve Braintrust-backed AI agents by mining traces, updating measurement/datasets/code/instrumentation, running evals, and emitting portable exit handoffs. |\n| `bt-cost-optimizer` | Analyze Braintrust logs, scorers, Topics, Gateway/provider spend, datasets, and experiments to recommend safe cost optimizations. |\n\nInstall each skill by copying or installing the full directory under `skills/\u003cskill-name\u003e/`; references, scripts, and agent metadata are part of the skill.\n\n## Repository Layout\n\n```text\nskills/\u003cskill-name\u003e/        Installable skill bundles. SKILL.md is the canonical per-skill entrypoint.\nexamples/\u003cskill-name\u003e/      Copyable runner and integration examples.\nevals/\u003cskill-name\u003e/         Offline evals for validating a skill's behavior.\nscorers/\u003cskill-name\u003e/       Braintrust online scorers or support code for a skill.\n```\n\nDo not add `README.md` files inside individual skill directories by default. Keep agent-facing instructions in `SKILL.md`, detailed context in `references/`, deterministic helpers in `scripts/`, and install/navigation docs in this README or [`skills/README.md`](skills/README.md).\n\n## bt-flywheel\n\n### What it does\n\nThe flywheel guides you through a 5-phase improvement loop:\n\n1. **Orient** — Resolve project config, establish goal and baseline experiment\n2. **Discover** — Mine production traces broadly for errors, scores, search clusters, topics, latency/cost, behavior, and coverage gaps\n3. **Diagnose** — Route to what needs changing: measurement/scorer, dataset, agent code, instrumentation, or exit if healthy\n4. **Improve** — Apply the artifact-specific route: measurement, dataset, agent, or instrumentation\n5. **Verify \u0026 Decide** — Run smoke/full evals, compare to baseline, inspect regressions, route another loop or exit\n\nOn exit, the skill writes an adapter-neutral handoff into `bt-flywheel-summary.json`. It includes outcome, severity, blocking status, confidence, findings, changes, verification, structured links, local artifacts, and intent-based `next_steps`. The calling workflow maps those next steps to local review, CI, GitHub, Slack, Jira/Linear, app UI, release gates, webhooks, or no side effect.\n\nWorks in interactive dev sessions, CI pipelines, scheduled/cron contexts, post-deploy checks, incident follow-up, and other agent harnesses.\n\n### Agent-Agnostic Contract\n\n`bt-flywheel` is meant to be plugged into different coding agents and automation systems. The portable contract is:\n\n1. Make `skills/bt-flywheel/` available to the agent as a skill, instruction bundle, or checked-out reference directory.\n2. Give the agent repository access, `bt` CLI access, Braintrust credentials, and project context.\n3. Ask the agent to follow `skills/bt-flywheel/SKILL.md`.\n4. Expect `bt-flywheel-summary.json` and `bt-flywheel-narrative.md` on exit.\n5. Let the surrounding harness decide whether to open a PR, issue, Slack message, Jira/Linear ticket, release gate, app notification, or do nothing based on `outcome`, `blocking`, and `next_steps`.\n\nThe skill should not depend on a specific coding agent. Agent-specific files such as `.claude/skills/`, `.cursor/`, `AGENTS.md`, or CLI prompts are integration details.\n\n`bt-flywheel-summary.json` should validate against the schema bundled with the skill, e.g. [`skills/bt-flywheel/schemas/bt-flywheel-summary.schema.json`](skills/bt-flywheel/schemas/bt-flywheel-summary.schema.json) in this repo.\n\n### Support Matrix\n\n| Surface | Status | Notes |\n|---|---|---|\n| Core skill in `skills/bt-flywheel/` | Supported | Portable skill contract and Braintrust workflow |\n| Summary schema | Supported | `bt-flywheel-summary.json` output contract |\n| GitHub Actions examples | Maintained examples | Copy into caller repos; no reusable workflow contract |\n| Codex / Cursor / OpenCode examples | Templates | Use as starting points; adapt to each runner's current CLI/auth model |\n| Slack / Jira / Linear | Handoff only | The skill emits adapter-neutral `next_steps`; downstream harnesses map and execute them |\n| Webhooks | Handoff only | Use caller-owned configuration; never put raw webhook URLs in the handoff |\n| Online flywheel scorers | Best-effort portable | Assumes trace spans expose shell/edit/write events with names similar to `Bash`, `Terminal`, `Edit`, or `Write` |\n\n## bt-cost-optimizer\n\n`bt-cost-optimizer` helps a coding agent answer: \"What Braintrust usage is driving cost, what can the `bt` CLI prove from data, and how should we safely change logging, scoring, Topics, Gateway usage, datasets, or experiments?\"\n\nThe skill:\n\n- Uses `bt status`, `bt projects`, `bt sql`, and `bt view` to collect bounded evidence from Braintrust.\n- Uses `bt scorers` and `bt topics` to inspect scorer inventory and Topics status/config where available.\n- Runs a local analyzer over exported rows to rank high-byte fields, largest traces, scorer spans, LLM token usage, and `JSONAttachment` candidates.\n- Inspects local code for Braintrust logging, scorer, and Gateway patterns and maps sample findings back to instrumentation.\n- Produces `bt-cost-optimization-report.md` and optionally `bt-cost-optimization-summary.json`.\n\nThe skill distinguishes measured findings from advisory recommendations. `bt` can measure sampled rows, scorer spans, token totals, and Topics config/status; exact bill totals, negotiated pricing, retention policy, and Gateway cache/routing config may require billing/UI or code/config context.\n\n## Install Skills\n\nInstall the whole skill directory, not only `SKILL.md`; the `references/`, `scripts/`, and `agents/` files are part of each skill.\n\nFor Codex, use the standard skill installer and choose the skill path:\n\n```bash\npython3 ~/.codex/skills/.system/skill-installer/scripts/install-skill-from-github.py \\\n  --repo dpguthrie/braintrust-skills \\\n  --path skills/bt-flywheel\n\npython3 ~/.codex/skills/.system/skill-installer/scripts/install-skill-from-github.py \\\n  --repo dpguthrie/braintrust-skills \\\n  --path skills/bt-cost-optimizer\n```\n\nFor project-local CI or another agent harness, copy the full skill directory into the runner's skill path:\n\n```bash\nmkdir -p .agent-skills\ncurl -fsSL https://github.com/dpguthrie/braintrust-skills/archive/refs/heads/main.tar.gz \\\n  | tar -xz --strip-components=2 -C .agent-skills braintrust-skills-main/skills/bt-flywheel\n\ncurl -fsSL https://github.com/dpguthrie/braintrust-skills/archive/refs/heads/main.tar.gz \\\n  | tar -xz --strip-components=2 -C .agent-skills braintrust-skills-main/skills/bt-cost-optimizer\n```\n\nFor Claude Code (and most other agents), use `npx skills`:\n\n```bash\nnpx skills add dpguthrie/braintrust-skills@bt-flywheel -g -y\nnpx skills add dpguthrie/braintrust-skills@bt-cost-optimizer -g -y\n```\n\n## Usage\n\nOnce installed, invoke the skill directly if your agent supports skills:\n\n```\n/bt-flywheel\n/bt-cost-optimizer\n```\n\nOr ask any coding agent to run the skill from the skill path:\n\n\u003e \"Use `skills/bt-flywheel/SKILL.md` or `.agent-skills/bt-flywheel/SKILL.md` to improve my Braintrust-backed agent.\"\n\nFor ingest optimization:\n\n\u003e \"Use `skills/bt-cost-optimizer/SKILL.md` or `.agent-skills/bt-cost-optimizer/SKILL.md` to analyze my Braintrust usage costs and recommend safe optimizations.\"\n\n## Common Requirements\n\n- [`bt` CLI](https://github.com/braintrustdata/bt) installed and authenticated\n- A Braintrust project with logs, experiments, or datasets to inspect\n- (Optional) `.bt/config.json` configured via `bt setup` for zero-config project resolution\n\n---\n\n## bt-flywheel GitHub Actions\n\nThis repo includes example GitHub Actions workflows you can copy into your own repository. They install the skill and define the runner logic locally; they do not call a reusable workflow from this repo.\n\nCopy `examples/bt-flywheel/flywheel-caller.yml` to `.github/workflows/flywheel.yml` in your repo and customize the project-specific values, install command, prompt context, and staged paths.\n\nRequired secrets in your repo: `ANTHROPIC_API_KEY` (to run Claude Code), `BRAINTRUST_API_KEY`.\n\nIf your agent calls a third-party LLM directly (OpenAI, Gemini, etc.), include its key in the workflow environment or `.env` the workflow writes for eval invocations.\n\nSet staged paths explicitly in the workflow's change-detection step. Avoid `git add .` so generated summaries, logs, downloaded skills, and unrelated changes do not get committed accidentally.\n\nSee [`examples/bt-flywheel/flywheel-caller.yml`](examples/bt-flywheel/flywheel-caller.yml) for the full annotated Claude Code example. For other coding agents, use the portable templates in [`examples/bt-flywheel/integrations.md`](examples/bt-flywheel/integrations.md): the common parts are installing Braintrust, making `skills/bt-flywheel` available, invoking the agent, and consuming the two output artifacts.\n\n---\n\n## bt-flywheel Other Triggers\n\nThe same skill can be invoked from many harnesses:\n\n| Trigger | Typical handoff intent |\n|---|---|\n| Manual local dev session | `review_change`, `investigate`, or `no_action` |\n| Scheduled weekly improvement job | `review_change` if changes were made, `investigate` if follow-up is needed |\n| Post-deploy verification | `block_release`, `notify`, or `no_action` |\n| Braintrust score degradation alert | `investigate` with trace evidence |\n| New production topic cluster | `label_data`, `review_change`, or `investigate` |\n| PR comment command like `/flywheel` | `review_change` or `notify` |\n| Release gate | `block_release`, `rerun`, or `no_action` |\n| Incident retrospective | `investigate` with trace links and eval gaps |\n| Dataset refresh cadence | `review_change` plus validation eval |\n\n---\n\n## Flywheel Quality Scorers\n\nThe `scorers/bt-flywheel/` directory contains six Braintrust online scorers that evaluate the quality of the flywheel's own execution — i.e., whether the coding agent runner is following the flywheel methodology correctly.\n\nThese are not scorers for your downstream task agent. They score the flywheel coding-agent session itself, catching things like:\n\n| Scorer | What it catches |\n|---|---|\n| Evidence Before Change | Agent editing code without first running `bt sql` or `bt view` |\n| Smoke Test Discipline | Running a full eval without a smoke run first |\n| Run Efficiency | Duplicate Bash commands or unnecessary credential-seeking calls |\n| Narrative Specificity | Run summaries that are vague (\"improved performance\") instead of specific (exact deltas, trace links) |\n| Diagnostic Coherence | Code changes that aren't motivated by the actual findings |\n| Claimed vs Actual | Summary claiming changes that don't match the actual Edit/Write spans |\n\n### Deploying the scorers\n\nInstall dependencies and push once to register them in the Braintrust project where your coding-agent traces are logged:\n\n```bash\npip install -r scorers/bt-flywheel/requirements.txt\n\nBRAINTRUST_API_KEY=... \\\nBRAINTRUST_CC_PROJECT=my-agent-coding-agent \\\nFLYWHEEL_CODE_PATHS=\"src/|evals/|scorers\\.py\" \\\nbt functions push --language python \\\n  --requirements scorers/bt-flywheel/requirements.txt \\\n  --if-exists replace \\\n  scorers/bt-flywheel/flywheel_scorers.py\n```\n\nRe-run any time you want to push updated scorer logic.\n\n`FLYWHEEL_CODE_PATHS` scopes edit-tracking scorers to your source files. Leave it empty to match all Edit/Write spans.\n\nTrace assumption: the online scorers inspect span names for shell/edit/write events. They expect names similar to `Bash:`, `Terminal:`, `Edit:`, or `Write:`. If your coding agent logs different span names, adapt `scorers/bt-flywheel/_scoring.py` before relying on those scores.\n\nThe LLM-judge scorers (`Narrative Specificity`, `Diagnostic Coherence`) use `gpt-4o-mini` by default. Override with `FLYWHEEL_JUDGE_MODEL=\u003cmodel\u003e`.\n\n---\n\n## Offline Evals\n\nThe `evals/bt-flywheel/` directory contains two Braintrust offline evals for measuring the quality of the flywheel skill itself.\n\n### Why offline evals?\n\nThe online scorers (above) catch anti-patterns in individual live runs. The offline evals complement them by:\n\n- Testing the scorer functions against fixture data (regression safety net for scorer changes)\n- Validating that the LLM judge rubric correctly distinguishes good flywheel behavior from known failure modes\n- Providing a benchmark dataset of positive and negative examples you can extend as new failure modes are discovered\n\n### `evals/bt-flywheel/eval_scorers.py` — Scorer unit tests\n\nTests the four deterministic scorer functions (`Evidence Before Change`, `Smoke Test Discipline`, `Run Efficiency`, `Claimed vs Actual`) against 22 fixture span sequences. Each case asserts the computed score falls within an expected range.\n\n```bash\npip install -r evals/bt-flywheel/requirements.txt\n\nBRAINTRUST_API_KEY=... \\\nBRAINTRUST_EVAL_PROJECT=bt-flywheel \\\nbraintrust eval evals/bt-flywheel/eval_scorers.py\n```\n\n### `evals/bt-flywheel/eval_behavior.py` — Behavior quality evaluation\n\nTests whether the LLM judge correctly rates flywheel behavior against synthetic scenarios: positive examples, Act recommendation examples, and negative failure modes:\n\n| Tag | Scenario | Expected rating |\n|---|---|---|\n| `healthy-exit` | Healthy production → exits early, no changes | A/B |\n| `broken-scorer` | Bimodal distribution → updates scorer | A/B |\n| `dataset-gap` | New query patterns → adds examples | A/B |\n| `agent-bug-fixed` | Low scores on query type → targeted prompt fix | A/B |\n| `no-convergence` | 3 iterations, no improvement → graceful exit | A/B |\n| `act-pr` | Code change with passing evals → recommends PR | A/B |\n| `act-issue` | Human follow-up needed → recommends issue | A/B |\n| `act-webhook` | External release gate → recommends blocking webhook | A/B |\n| `act-none` | Healthy system → recommends no action | A/B |\n| `unnecessary-changes` | Healthy system → made changes anyway | C/D |\n| `wrong-diagnosis` | Bimodal scorer → tried to fix agent code | C/D |\n| `vague-summary` | Real issues found → summary has no specifics | C/D |\n| `ignored-regressions` | Metric improved but 5 regressions → marked done | C/D |\n| `incomplete-diagnosis` | Two issues found → only addressed one | C/D |\n\n```bash\nBRAINTRUST_API_KEY=... \\\nBRAINTRUST_EVAL_PROJECT=bt-flywheel \\\nFLYWHEEL_JUDGE_MODEL=gpt-4o \\\nbraintrust eval evals/bt-flywheel/eval_behavior.py\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fbraintrust-skills","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpguthrie%2Fbraintrust-skills","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fbraintrust-skills/lists"}