{"id":51336929,"url":"https://github.com/dpguthrie/braintrust-agent-evals","last_synced_at":"2026-07-02T03:32:39.110Z","repository":{"id":358691721,"uuid":"1241632857","full_name":"dpguthrie/braintrust-agent-evals","owner":"dpguthrie","description":"Harbor\u003c\u003eBraintrust evals for coding-agent tooling","archived":false,"fork":false,"pushed_at":"2026-05-18T17:22:23.000Z","size":103,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-18T17:36:06.543Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpguthrie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-17T16:20:48.000Z","updated_at":"2026-05-18T17:22:11.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dpguthrie/braintrust-agent-evals","commit_stats":null,"previous_names":["dpguthrie/braintrust-agent-evals"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/dpguthrie/braintrust-agent-evals","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-agent-evals","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-agent-evals/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-agent-evals/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-agent-evals/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpguthrie","download_url":"https://codeload.github.com/dpguthrie/braintrust-agent-evals/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fbraintrust-agent-evals/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35032145,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-02T02:00:06.368Z","response_time":173,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-02T03:32:36.389Z","updated_at":"2026-07-02T03:32:39.069Z","avatar_url":"https://github.com/dpguthrie.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Braintrust Harbor\n\nHarbor-first, Braintrust-backed evaluation helpers for coding-agent tooling.\n\nUse this package when you want to evaluate how coding agents interact with your\ndeveloper tools, CLIs, MCP servers, skills, repositories, or harness-specific\ninstructions.\n\n## What This Does\n\nThis package is the bridge between Harbor and Braintrust:\n\n- Harbor runs the agent trials in sandboxes.\n- Braintrust stores experiment rows, traces, scores, comparisons, and analysis.\n- `bt-harbor` runs one Harbor job or imports an existing Harbor job, then maps\n  each Harbor trial to one Braintrust experiment row.\n\nThe intended flow is:\n\n```text\nHarbor tasks + JobConfig\n  -\u003e harbor run\n  -\u003e Harbor job artifacts\n  -\u003e bt-harbor import\n  -\u003e Braintrust experiment\n```\n\n## Prerequisites\n\nYou need:\n\n- Python 3.11 or newer.\n- The Harbor CLI available on `PATH`.\n- A Harbor-supported sandbox backend, usually Docker for local examples.\n- Model provider API keys required by the Harbor agents you run.\n- `BRAINTRUST_API_KEY` only when using `--upload`. Local preview imports work\n  without uploading.\n\nInstall `braintrust-harbor` from PyPI:\n\n```bash\nuv tool install braintrust-harbor\n```\n\nCheck the CLIs:\n\n```bash\nbt-harbor --help\nharbor --version\n```\n\nYou can also run the latest published package without installing a persistent\ntool:\n\n```bash\nuvx braintrust-harbor --help\n```\n\nWhen developing from a checkout, install the local copy instead:\n\n```bash\nuv tool install -e .\n```\n\n## Quickstart\n\nStart with the prompt + repo + harness/model + tooling-version workflow:\n\n```bash\nbt-harbor init harness-model-demo --template harness-model-matrix\ncd harness-model-demo\n```\n\nEdit `eval-input.json` to choose:\n\n- `prompt`: the user request the agent should complete.\n- `repo.url` and `repo.ref`: optional repository to clone into the sandbox.\n- `agents`: Harbor harness/model pairs.\n- `tooling_versions`: labels and optional install commands for the tooling\n  versions you want to compare.\n\nGenerate a Harbor suite for one tooling version:\n\n```bash\npython scripts/materialize.py \\\n  --input eval-input.json \\\n  --tooling-version current \\\n  --out generated/current\n```\n\nRun Harbor and write a local Braintrust import preview:\n\n```bash\ncd generated/current\nbt-harbor run harbor-job.json \\\n  --project \"agent-tooling-demo\" \\\n  --suite-artifacts suite-artifacts.json \\\n  --metadata metadata.json \\\n  --scorer \"scorers:summary_present\" \\\n  --scorer \"scorers:used_demo_tool\" \\\n  --scorer \"scorers:no_harbor_exception\"\n```\n\nAdd `--upload` to publish the imported rows to Braintrust:\n\n```bash\nbt-harbor run harbor-job.json \\\n  --project \"agent-tooling-demo\" \\\n  --suite-artifacts suite-artifacts.json \\\n  --metadata metadata.json \\\n  --scorer \"scorers:summary_present\" \\\n  --scorer \"scorers:used_demo_tool\" \\\n  --scorer \"scorers:no_harbor_exception\" \\\n  --upload\n```\n\n## Core Concepts\n\nHarbor concepts:\n\n- A task is a directory with instructions, an environment, and verifier tests.\n- A trial is one agent attempt at a task.\n- A job is a collection of trials across tasks, datasets, agents, and models.\n\nBraintrust concepts:\n\n- An experiment is a comparable eval run.\n- Each row represents one eval case. In this package, one Harbor trial becomes\n  one Braintrust row.\n- Scores are numeric checks added by Harbor rewards or Braintrust-compatible\n  scorers.\n- Traces preserve the trial trajectory, tool calls, command logs, and scorer\n  spans when those artifacts are available.\n\nCurrent tracing is an offline import from completed Harbor job artifacts, not a\nlive hook into Harbor's eval lifecycle. See [Tracing](docs/tracing.md) for the\nexact contract and limitations.\n\nThis package does not replace Harbor's `JobConfig`, agent adapters, sandbox\nmanagement, or task format. It also does not impose your product's scoring\npolicy. Suites should keep product-specific prompts, fixtures, verifiers, and\nscorers in their own task directories.\n\n## Common Workflows\n\nRun a Harbor job and import it:\n\n```bash\nbt-harbor run harbor-job.json \\\n  --project \"agent-tooling-demo\" \\\n  --suite-artifacts suite-artifacts.json\n```\n\nImport a job that another system already ran:\n\n```bash\nbt-harbor import jobs/nightly-agent-tooling-run \\\n  --project \"agent-tooling-demo\" \\\n  --suite-artifacts suite-artifacts.json \\\n  --upload\n```\n\nAdd suite scorers:\n\n```bash\nbt-harbor import jobs/nightly-agent-tooling-run \\\n  --project \"agent-tooling-demo\" \\\n  --suite-artifacts suite-artifacts.json \\\n  --scorer \"my_suite.scorers:summary_present\" \\\n  --scorer \"my_suite.scorers:tool_order_quality\" \\\n  --upload\n```\n\nRun on a schedule:\n\n1. Keep the Harbor execution in CI, cron, or your scheduler.\n2. Install the latest version of your tooling in the task environment.\n3. Run `bt-harbor run ... --upload`, or run Harbor separately and then\n   `bt-harbor import ... --upload`.\n4. Compare Braintrust experiments by metadata such as `tooling_version`,\n   `agent`, `model`, `scenario`, and `variant`.\n\n## Examples\n\nSee [examples/README.md](examples/README.md) for the full examples guide.\n\n- [harness-model-matrix](examples/harness-model-matrix/README.md):\n  best first example for evaluating one prompt across optional repos,\n  harness/model pairs, and tooling versions.\n- [minimal-cli-tool](examples/minimal-cli-tool/README.md): smallest runnable\n  template for a toy CLI, command logging, artifacts, and a verifier.\n- [skill-vs-no-skill](examples/skill-vs-no-skill/README.md): task layout\n  pattern for comparing a skill, MCP server, rules file, or guide against a\n  control variant.\n- [import-existing-harbor-job](examples/import-existing-harbor-job/README.md):\n  import-only pattern for teams that already run Harbor elsewhere.\n\n## Customization Checklist\n\nFor a real suite, customize these pieces:\n\n- `harbor-job.json`: agents, models, concurrency, datasets, attempts, and\n  environment backend.\n- `tasks/*/instruction.md`: the user-facing task prompt.\n- `tasks/*/environment/`: install your CLI, MCP server, skill, repo, or service.\n- `tasks/*/tests/`: verify the agent produced the required behavior and reward.\n- `.agent-tooling-eval.json`: row input, expected output, and metadata labels.\n- `suite-artifacts.json`: extra trial artifacts to attach to Braintrust rows.\n- `scorers.py`: Braintrust-compatible scorers for trajectory and output quality.\n\nUseful metadata dimensions are `tooling_version`, `scenario`, `variant`,\n`agent`, `model`, `repo_ref`, and any feature flag names you are comparing.\n\n## External Documentation\n\nHarbor:\n\n- [Core concepts](https://www.harborframework.com/docs/core-concepts)\n- [Task structure](https://www.harborframework.com/docs/tasks)\n- [Run jobs](https://www.harborframework.com/docs/run-jobs)\n- [Agents](https://www.harborframework.com/docs/agents)\n\nBraintrust:\n\n- [Evaluate systematically](https://www.braintrust.dev/docs/evaluate)\n- [Run experiments](https://www.braintrust.dev/docs/evaluate/run-evaluations)\n- [Interpret evaluation results](https://www.braintrust.dev/docs/evaluate/interpret-results)\n- [Examine traces](https://www.braintrust.dev/docs/observe/examine-traces)\n\nProject docs:\n\n- [Architecture](docs/architecture.md)\n- [Tracing](docs/tracing.md)\n- [Scorers](docs/scorers.md)\n- [Compatibility](docs/compatibility.md)\n- [Releasing](docs/releasing.md)\n\n## Library API\n\n```python\nfrom braintrust_harbor import (\n    HarborBatchConfig,\n    SuiteArtifactConfig,\n    import_harbor_job_to_braintrust,\n    run_harbor_batch,\n)\n```\n\nThe recommended integration point is still Harbor's native `JobConfig`:\n\n```text\nHarbor JobConfig -\u003e one Harbor run -\u003e one Braintrust experiment\n```\n\nIf your suite needs extra dimensions such as \"with skill\" vs. \"without skill\",\ngenerate ordinary Harbor task directories or datasets for those variants, then\npoint the Harbor job config at them.\n\n## Status\n\nThis is an alpha extraction. The Harbor job import path and Braintrust upload\npath are usable, but the CLI and normalized trace/event helpers should be\ntreated as evolving APIs until a stable release.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fbraintrust-agent-evals","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpguthrie%2Fbraintrust-agent-evals","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fbraintrust-agent-evals/lists"}