{"id":51216881,"url":"https://github.com/testdouble/skills-test-harness","last_synced_at":"2026-06-28T04:31:26.101Z","repository":{"id":358132084,"uuid":"1212552954","full_name":"testdouble/skills-test-harness","owner":"testdouble","description":"A test harness for skills files","archived":false,"fork":false,"pushed_at":"2026-05-15T20:51:19.000Z","size":905,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-15T23:42:28.238Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/testdouble.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-16T13:47:37.000Z","updated_at":"2026-05-15T20:51:22.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/testdouble/skills-test-harness","commit_stats":null,"previous_names":["testdouble/skills-test-harness"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/testdouble/skills-test-harness","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/testdouble%2Fskills-test-harness","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/testdouble%2Fskills-test-harness/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/testdouble%2Fskills-test-harness/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/testdouble%2Fskills-test-harness/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/testdouble","download_url":"https://codeload.github.com/testdouble/skills-test-harness/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/testdouble%2Fskills-test-harness/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34877471,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-28T02:00:05.809Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-28T04:31:25.271Z","updated_at":"2026-06-28T04:31:26.088Z","avatar_url":"https://github.com/testdouble.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Test Harness\n\nA test harness for evaluating Claude Code skills and agents. It measures two\nthings you can improve independently:\n\n- **Trigger accuracy** — does Claude call your skill (or delegate to your\n  agent) when it *should*, and stay quiet when it *shouldn't*?\n- **Effectiveness** — once invoked, does the output meet quality criteria you\n  define? An LLM judge scores it against a rubric.\n\nYou write evals, run them inside a Test Sandbox, and track trigger accuracy,\noutput quality, and cost over time through a dashboard and analytics. **Skills**\nand **agents** each have their own trigger-accuracy and effectiveness paths —\npick the one that matches what you're improving.\n\n## Prerequisites\n\n- **Docker Sandboxes (`sbx`)** — the harness runs Claude Code inside Docker\n  Sandboxes via the standalone [`sbx`](https://docs.docker.com/reference/cli/sbx/) CLI. Install `sbx` using [Docker's\n  instructions](https://docs.docker.com/ai/sandboxes/), then run `sbx login` before creating the harness sandbox.\n- **Bun** — the CLI and web app are built with Bun. Install from\n  [bun.sh](https://bun.sh).\n\n## Setup\n\nAll commands run from the **repository root**.\n\n1. **Build the harness.** This installs dependencies and compiles the `harness`\n   and `harness-web` binaries:\n\n   ```bash\n   make build\n   ```\n\n2. **Create the Test Sandbox and log in.** Authenticate the Sandbox CLI, then\n   create a persistent sandbox and open Claude Code so you can authenticate:\n\n   ```bash\n   sbx login\n   ./harness sandbox-setup\n   ```\n\n   Complete the login in the Claude TUI. If you aren't prompted, run `/login`.\n   When setup finishes, exit with `/exit`.\n\nYou're ready. Now choose what you want to measure.\n\n## What do you want to measure?\n\nPick the path that matches your goal — each is a short, end-to-end guide.\n\n**Improving a custom skill**\n\n1. [Know *when* Claude should call my skill](docs/getting-started/skill-trigger-accuracy.md) — trigger accuracy\n2. [Make my skill better *at its job*](docs/getting-started/skill-effectiveness.md) — effectiveness\n\n**Improving a custom agent**\n\n1. [Know *when* Claude should delegate to my agent](docs/getting-started/agent-trigger-accuracy.md) — trigger accuracy\n2. [Make my agent better *at its job*](docs/getting-started/agent-effectiveness.md) — effectiveness\n\nAfter your first run, [view results in the dashboard](docs/getting-started/viewing-results.md)\nand [query trends with analytics](docs/getting-started/analytics.md).\n\n---\n\n## Reference\n\nEverything above is all you need for first-time use. The links below are\nreference material — reach for them when a guide points you here.\n\n### Getting Started Guides\n\n- [Skill Trigger Accuracy](docs/getting-started/skill-trigger-accuracy.md) — measure and improve when Claude calls your skill\n- [Skill Effectiveness](docs/getting-started/skill-effectiveness.md) — measure and improve skill output quality with LLM-judge rubrics\n- [Agent Trigger Accuracy](docs/getting-started/agent-trigger-accuracy.md) — measure and improve when Claude delegates to your agent\n- [Agent Effectiveness](docs/getting-started/agent-effectiveness.md) — measure and improve agent output quality with LLM-judge rubrics\n- [Viewing Results](docs/getting-started/viewing-results.md) — using the harness-web dashboard\n- [Analytics](docs/getting-started/analytics.md) — importing data and CLI queries\n\n### Full Workflow Guides\n\n- [Building SCIL Evals](docs/scil-evals-guide.md) — manual test authoring and the Skill Call Improvement Loop\n- [Building Rubric Evals](docs/rubric-evals-guide.md) — manual rubric authoring and iterating on LLM-judge quality criteria\n\n### Configuration Reference\n\n- [Test Suite Reference](docs/test-suite-reference.md) — full `tests.json` field reference: test types, expectation types, validation\n- [Test Scaffolding](docs/test-scaffolding.md) — how scaffolds provide project context inside the Test Sandbox\n\n### Eval Authoring Skills\n\nClaude Code skills that generate eval suites for you:\n\n- [Building Skill Eval Scaffolds](docs/build-skill-eval-scaffold.md) — `/build-skill-eval-scaffold`: analysis, signal planning, scaffold generation\n- [Building Agent Eval Scaffolds](docs/build-agent-eval-scaffold.md) — `/build-agent-eval-scaffold`: the agent equivalent\n- [Writing Skill-Call Evals](docs/write-scil-evals.md) — `/write-scil-evals`: prompt categories, output format\n- [Writing Agent-Call Evals](docs/write-acil-evals.md) — `/write-acil-evals`: the agent equivalent\n- [Writing Skill Eval Rubrics](docs/write-skill-eval-rubric.md) — `/write-skill-eval-rubric`: criteria categories, output format\n- [Writing Agent Eval Rubrics](docs/write-agent-eval-rubric.md) — `/write-agent-eval-rubric`: the agent equivalent\n- [Script Extraction](docs/script-extraction.md) — `/script-extraction`: hardening skills by extracting mechanical steps into shell scripts\n\n### Deep Dives\n\n- [Skill Call Improvement Loop](docs/skill-call-improvement-loop.md) — SCIL mechanics: holdout splits, scoring, improvement prompt, CLI flags\n- [Agent Call Improvement Loop](docs/agent-call-improvement-loop.md) — ACIL mechanics: agent detection, temp plugin isolation, holdout splits, scoring\n- [LLM Judge Evaluation](docs/llm-judge.md) — judge mechanics: prompt construction, scoring, output format, error handling\n- [Parquet Schema](docs/parquet-schema.md) — field reference for analytics Parquet files\n\n### Architecture\n\n- [Test Harness Architecture](docs/test-harness-architecture.md) — system architecture, package boundaries, data flow, and dependency graph\n- [Sandbox Integration](docs/sandbox-integration.md) — Test Sandbox architecture, API, lifecycle, and consumer patterns\n- [Project Discovery](docs/project-discovery.md) — generated project attributes: languages, frameworks, tooling, commands\n\n### Package Documentation (contributor)\n\n- [CLI](docs/cli.md) — Yargs command layer and path resolution\n- [Execution](docs/execution.md) — test-run pipeline, test-eval, SCIL/ACIL orchestration, error hierarchy\n- [Data](docs/data.md) — shared data layer: types, config parsing, JSONL I/O, DuckDB analytics, SCIL utilities\n- [Evals](docs/evals.md) — evaluation engine: boolean evals, LLM judge scoring, rubric parsing, orchestrator\n- [Claude Integration](docs/claude-integration.md) — Claude CLI wrapper API, argument construction, sandbox delegation\n- [Sandbox Integration Package](docs/sandbox-integration-package.md) — Test Sandbox API: full public interface, error handling, testing patterns\n- [Web](docs/web.md) — Hono API server, React SPA, test run and SCIL views, per-test analytics\n- [Bun Helpers](docs/bun-helpers.md) — cross-runtime path resolution utilities\n- [Test Fixtures](docs/test-fixtures.md) — shared fixture data, loadFixtures utility, analytics JSONL scenarios\n\n## Maintenance and Support\n\n- **Maintenance horizon:** Indefinitely maintained, best-effort. No SLA.\n- **Project type:** Personal project, with some Test Double support.\n- **How to report issues:** GitHub Issues, with best-effort response within 2 weeks.\n\nHan is an open source product of [Test Double](https://testdouble.com), and maintained by the following people:\n\n- [River Lynn Bailey](https://github.com/mxriverlynn): Creator, and primary maintainer\n\n## LEGAL NOTICES\n\nCopyright 2026 [Test Double, Inc](https://testdouble.com). Distributed under the [MIT license](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftestdouble%2Fskills-test-harness","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftestdouble%2Fskills-test-harness","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftestdouble%2Fskills-test-harness/lists"}