{"id":51020646,"url":"https://github.com/tushariitr-19/assay","last_synced_at":"2026-06-28T21:00:45.296Z","repository":{"id":364808732,"uuid":"1268900186","full_name":"tushariitr-19/assay","owner":"tushariitr-19","description":"Framework-agnostic evaluation harness for Go — test your MCP servers and AI agents with scored, CI-ready checks.","archived":false,"fork":false,"pushed_at":"2026-06-14T14:40:45.000Z","size":10,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-14T16:44:35.389Z","etag":null,"topics":["adk","agent-eval","ai-agents","cli","evaluation","golang","llm-eval","mcp","model-context-protocol","testing"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tushariitr-19.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-14T04:17:02.000Z","updated_at":"2026-06-14T14:40:49.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/tushariitr-19/assay","commit_stats":null,"previous_names":["tushariitr-19/assay"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/tushariitr-19/assay","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tushariitr-19%2Fassay","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tushariitr-19%2Fassay/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tushariitr-19%2Fassay/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tushariitr-19%2Fassay/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tushariitr-19","download_url":"https://codeload.github.com/tushariitr-19/assay/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tushariitr-19%2Fassay/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34903523,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-28T02:00:05.809Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adk","agent-eval","ai-agents","cli","evaluation","golang","llm-eval","mcp","model-context-protocol","testing"],"created_at":"2026-06-21T16:00:26.006Z","updated_at":"2026-06-28T21:00:45.290Z","avatar_url":"https://github.com/tushariitr-19.png","language":"Go","funding_links":[],"categories":["Testing","Template Engines"],"sub_categories":["Testing Frameworks"],"readme":"# assay\n\n[![Go Version](https://img.shields.io/badge/Go-1.25+-00ADD8?style=flat\u0026logo=go)](https://golang.org)\n[![Go Reference](https://pkg.go.dev/badge/github.com/tushariitr-19/assay.svg)](https://pkg.go.dev/github.com/tushariitr-19/assay)\n[![Go Report Card](https://goreportcard.com/badge/github.com/tushariitr-19/assay)](https://goreportcard.com/report/github.com/tushariitr-19/assay)\n[![codecov](https://codecov.io/gh/tushariitr-19/assay/branch/main/graph/badge.svg)](https://codecov.io/gh/tushariitr-19/assay)\n[![CI](https://github.com/tushariitr-19/assay/workflows/CI/badge.svg)](https://github.com/tushariitr-19/assay/actions)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n\nA framework-agnostic evaluation library and CLI for Go. Point `assay` at an **MCP server** or an **AI agent**, give it a suite of checks in YAML, and it scores whether the right tools were called, with the right arguments, fast enough, and producing the expected output — then fails your CI when something regresses.\n\nBuilt on the official [Go MCP SDK](https://github.com/modelcontextprotocol/go-sdk) and tested against [Google ADK for Go](https://github.com/google/adk-go).\n\n---\n\n## Why assay?\n\nThe Go ecosystem has agent frameworks (ADK, eino, trpc-agent-go) and a first-class [MCP SDK](https://github.com/modelcontextprotocol/go-sdk) — but no standalone way to *evaluate* what you build with them. The evaluators that exist are **bundled into a single framework** and only test that framework's agents. If you ship an MCP server, there's nothing in Go that answers the everyday question: *does my server still expose the tools it claims, and do they still work?*\n\n`assay` is that missing piece — a deterministic, framework-agnostic test harness for tool-using systems:\n\n- **Framework-agnostic** — the core depends on nothing but the standard library. Anything that satisfies a tiny interface can be evaluated: MCP servers, ADK agents, or your own.\n- **Deterministic** — MCP server checks need no LLM and no API keys. They run free, in CI, on every commit.\n- **Minimal integration** — test an MCP server with zero Go code (a binary + a YAML). Test an agent with two lines.\n- **CI-ready** — every run returns a proper exit code, so a failing suite breaks the build.\n- **Open source** — MIT licensed.\n- **Production grade** — keyless-testable core, structured reports (console + JSON), clean multi-module architecture.\n\n---\n\n## Architecture\n\n![Architecture](docs/architecture/architecture.svg)\n\n`assay` is built as **two parallel tracks over one shared core.**\n\nThe **core** (the root `assay` package) is completely agnostic — it imports no framework SDK. It defines two contracts:\n\n- `Agent` — anything with `Run(ctx, input) (Result, error)`. This is the *agent track*: give it a prompt, see which tools it chose and what it returned.\n- `ToolServer` — anything with `ListTools(ctx)` and `CallTool(ctx, name, args)`. This is the *MCP track*: probe a server's contract directly, no LLM involved.\n\n**Adapters** live in their own modules so their heavy dependency trees never reach the core:\n\n- `adk/` wraps a Google ADK agent and satisfies `Agent`.\n- `mcp/` wraps the official MCP SDK client and satisfies `ToolServer`.\n\nBoth tracks feed the **same shared machinery** — the runner, the scoring, the console/JSON reporters, the YAML suite loader, and the CLI with its exit codes. Adding support for a new framework (LangChain-Go, a vanilla LLM loop, another MCP transport) means writing one thin adapter against an existing interface — nothing in the core changes.\n\n\u003e The dependency arrow only ever points one way: adapters depend on the core's interfaces, never the reverse. If the core's `go.mod` ever needs to import an SDK, the boundary has been violated.\n\n---\n\n## What it evaluates\n\n### MCP server checks (deterministic, no API key)\n\n| Check | Description |\n|-------|-------------|\n| `tools_list` | Asserts the server exposes the expected set of tools |\n| `tool_call` | Calls a tool with given args and asserts on the outcome — `expect_no_error`, `result_contains`, `max_latency` |\n\n### Agent metrics\n\n| Metric | Description |\n|--------|-------------|\n| `tool_correctness` | Did the agent call the expected tools (optionally ordered, with argument matching)? |\n| `output_contains` | Does the response contain the expected substrings? |\n| `output_regex` | Does the response match the expected patterns? |\n| `max_latency` | Did the run complete within the latency ceiling? |\n\nEach check/metric returns a score in `[0,1]`; a suite passes when the aggregate meets its threshold.\n\n---\n\n## Quickstart — test an MCP server (zero Go code)\n\nBuild your server to a binary, write a `checks.yaml`, and run `assay`. No integration code required.\n\n```yaml\n# checks.yaml\nthreshold: 1.0\nchecks:\n  - name: lists_all_tools\n    type: tools_list\n    expect_tools: [get_visa_bulletin, check_priority_date, explain_term]\n\n  - name: explain_term_works\n    type: tool_call\n    tool: explain_term\n    args: { term: \"priority date\" }\n    expect_no_error: true\n    max_latency: 10s\n```\n\n```bash\nassay mcp --server \"./immigration-mcp-server\" --suite checks.yaml\n```\n\n```text\nCASE                      SCORE  RESULT\ntools_list                1.00   ✓\n  tools_list              1.00   3/3 expected tools exposed\ntool_call:explain_term    1.00   ✓\n  tool_call:explain_term  1.00   2/2 assertions passed\nSUITE SCORE: 1.00   threshold 1.00   PASS\n```\n\nExit code is `0` on pass, `1` on failure — drop it straight into a CI job.\n\n---\n\n## Quickstart — test an agent (a few lines of Go)\n\nAgents are constructed in code, so you wrap your agent in the matching adapter, register it, and hand off to the assay CLI. Here, an ADK agent:\n\n```go\npackage main\n\nimport (\n    \"github.com/tushariitr-19/assay/adk\"\n    \"github.com/tushariitr-19/assay/cli/app\"\n)\n\nfunc main() {\n    adapter, _ := adk.New(rootAgent) // wrap your ADK agent\n    app.SetAgent(adapter)            // register it\n    app.Main()                       // hand off to the assay CLI\n}\n```\n\nBuild it and run the agent track through the CLI:\n\n```bash\ngo build -o myeval .\n./myeval agent --suite cases.yaml\n```\n\n```yaml\n# cases.yaml\nthreshold: 0.80\ncases:\n  - name: search_recent_papers\n    input: \"Find recent papers on retrieval-augmented generation\"\n    expect_tools: [TopicSearchAgent]\n    output_contains: [\"RAG\"]\n    max_latency: 60s\n```\n\n```text\nCASE                  SCORE  RESULT\nsearch_recent_papers  1.00   ✓\n  tool_correctness    1.00   1/1 expected tools called\n  output_contains     1.00   1/1 substrings present\n  max_latency         1.00   17527ms \u003c= 60000ms\nSUITE SCORE: 1.00   threshold 0.80   PASS\n```\n\nA runnable version lives in [`examples/research-agent/`](examples/research-agent), dogfooded against [research-agent-adk](https://github.com/tushariitr-19/research-agent-adk).\n\n**Bring your own agent:** if you're not on ADK, implement the one-method `Agent` interface yourself — everything else (metrics, runner, reports, CLI) works unchanged.\n\n```go\ntype MyAgent struct{ /* ... */ }\n\nfunc (a MyAgent) Run(ctx context.Context, input string) (assay.Result, error) {\n    // call your agent, return Output + the tool calls it made\n}\n```\n\nYou can also call the runner directly from a `_test.go` instead of building a CLI:\n\n```go\nadapter, _ := adk.New(rootAgent)\nsuite, _ := assay.LoadSuite(\"cases.yaml\")\nreport := assay.Runner{}.Run(ctx, adapter, suite)\nreport.WriteText(os.Stdout)\n```\n\n---\n\n## Prerequisites\n\n- Go 1.25+\n- For MCP server checks: **no API keys** — `assay` only talks to your server.\n- For the agent example: whatever your agent needs (e.g. a Google AI Studio key for the ADK example).\n\n---\n\n## Install\n\n```bash\n# Library — to embed in Go tests or programs\ngo get github.com/tushariitr-19/assay\n\n# CLI — to test MCP servers from the command line\ngo install github.com/tushariitr-19/assay/cli/cmd/assay@latest\n```\n\nThe installed `assay` binary tests **MCP servers** out of the box (`assay mcp ...`). To test an **agent**, build your own binary that registers it via `app.SetAgent` — see the agent quickstart above — since agents are constructed in code.\n\n---\n\n## Project Structure\n\n```\nassay/\n├── agent.go             ← Agent interface, Result, ToolCall\n├── toolserver.go        ← ToolServer interface, ToolOutcome\n├── metric.go            ← agent metrics (tool_correctness, output_contains, …)\n├── check.go             ← MCP checks (tools_list, tool_call)\n├── runner.go            ← agent runner\n├── server_runner.go     ← MCP server runner\n├── reporter.go          ← console + JSON reports (shared by both tracks)\n├── suite_yaml.go        ← agent suite loader\n├── server_yaml.go       ← MCP suite loader\n├── config/              ← run configuration\n├── logger/              ← structured logging via zap\n├── internal/\n│   ├── fakeagent/       ← keyless agent for tests\n│   └── faketoolserver/  ← keyless server for tests\n├── adk/                 ← ADK adapter (own module)\n├── mcp/                 ← MCP adapter (own module)\n├── cli/                 ← the assay binary (own module)\n│   ├── app/             ← importable CLI: SetAgent, Main\n│   └── cmd/assay/       ← the assay binary entry point\n├── examples/\n│   └── research-agent/  ← ADK dogfood example\n├── tests/               ← integration tests\n└── docs/                ← architecture diagram + screenshots\n```\n\nThis is a **multi-module repository**: the root is the agnostic core library; `adk`, `mcp`, and `cli` are separate modules so their SDK dependencies stay out of the core.\n\n---\n\n## Makefile Targets\n\n```bash\nmake build             # Build the assay CLI binary\nmake test              # Unit tests (keyless, no network)\nmake test-integration  # Integration tests (spawns real servers / agents)\nmake test-all          # All tests\nmake lint              # go vet + gofmt check\nmake clean             # Remove build artifacts\n```\n\n---\n\n## Design Decisions\n\n**Agnostic core, thin adapters** — the core knows two interfaces and nothing about ADK or MCP. Framework specifics live in dedicated modules, so the published library stays dependency-light and any new framework is a small additive adapter.\n\n**Keyless, deterministic core** — every metric and check is unit-tested against in-memory fakes (`fakeagent`, `faketoolserver`) with no network or credentials, so the whole core runs green in CI for free.\n\n**MCP testing is contract testing, not LLM testing** — an MCP server doesn't decide anything; it provides tools. So `assay` probes it directly (`tools/list`, `tools/call`) rather than putting a model in the loop. Deterministic, cheap, and exactly what \"does my server work?\" means.\n\n**Two tracks, one report** — the agent runner and the server runner produce the identical report structure, so the console/JSON reporters and CI exit codes are shared, untouched, across both.\n\n**Transport vs. tool errors are distinct** — a crashed connection surfaces as a Go error; a tool that ran but reported failure surfaces as `IsError`, scored by the `expect_no_error` check. The two are never conflated.\n\n---\n\n## Roadmap\n\n- [x] MCP server evaluation — `tools_list` and `tool_call` checks\n- [x] Agent evaluation — tool correctness, output contains/regex, latency\n- [x] Official MCP SDK adapter (stdio transport)\n- [x] Google ADK adapter\n- [x] Unified `assay` CLI with CI exit codes\n- [x] Console + JSON reports, YAML suites\n- [x] `assay agent` subcommand via register-and-build (app.SetAgent + Main)\n- [ ] LLM-as-judge metric for response quality (behind a pluggable `Judge` interface)\n- [ ] Nested trajectory granularity — score leaf tool calls inside sub-agents, not just root routing\n- [ ] MCP `resources` and `prompts` checks\n- [ ] MCP HTTP transport (in addition to stdio)\n- [ ] More adapters — LangChain-Go, vanilla LLM loops\n- [ ] Synthetic test-case generation\n\n---\n\n## Contributing\n\nPRs welcome. To support a new framework, you only write an adapter:\n\n1. Create a new module (e.g. `langchain/`) with its own `go.mod`.\n2. Implement either `assay.Agent` (for agents) or `assay.ToolServer` (for tool servers).\n3. Translate the framework's output into `assay.Result` / `assay.ToolOutcome`.\n4. Add an example under `examples/` and an integration test under `tests/`.\n\nThe core never needs to change.\n\n---\n\n## License\n\nMIT — see [LICENSE](LICENSE) for details.\n\n## Acknowledgements\n\n- [Model Context Protocol Go SDK](https://github.com/modelcontextprotocol/go-sdk) — the official Go MCP SDK.\n- [Google ADK](https://github.com/google/adk-go) — Agent Development Kit for Go.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftushariitr-19%2Fassay","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftushariitr-19%2Fassay","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftushariitr-19%2Fassay/lists"}