{"id":49250355,"url":"https://github.com/blackwell-systems/mcp-assert","last_synced_at":"2026-05-10T04:22:14.027Z","repository":{"id":353310197,"uuid":"1218716134","full_name":"blackwell-systems/mcp-assert","owner":"blackwell-systems","description":"Deterministic correctness testing for MCP servers. Assert your tools return the right results, not just any results. No LLM-as-judge.","archived":false,"fork":false,"pushed_at":"2026-04-23T10:12:13.000Z","size":54,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-23T11:32:51.701Z","etag":null,"topics":["ai-agents","assertions","ci","deterministic-testing","developer-tools","evals","evaluation","golang","language-server-protocol","mcp","mcp-server","model-context-protocol","testing"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blackwell-systems.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-23T06:32:08.000Z","updated_at":"2026-04-23T10:12:39.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/blackwell-systems/mcp-assert","commit_stats":null,"previous_names":["blackwell-systems/mcp-assert"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/blackwell-systems/mcp-assert","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackwell-systems%2Fmcp-assert","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackwell-systems%2Fmcp-assert/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackwell-systems%2Fmcp-assert/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackwell-systems%2Fmcp-assert/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blackwell-systems","download_url":"https://codeload.github.com/blackwell-systems/mcp-assert/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackwell-systems%2Fmcp-assert/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32245151,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-24T13:21:15.438Z","status":"ssl_error","status_checked_at":"2026-04-24T13:21:15.005Z","response_time":64,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","assertions","ci","deterministic-testing","developer-tools","evals","evaluation","golang","language-server-protocol","mcp","mcp-server","model-context-protocol","testing"],"created_at":"2026-04-25T00:01:42.038Z","updated_at":"2026-05-10T04:22:14.017Z","avatar_url":"https://github.com/blackwell-systems.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/social-preview.png\" alt=\"mcp-assert\" width=\"600\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/blackwell-systems\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/blackwell-systems/blackwell-docs-theme/main/badge-trademark.svg\" alt=\"Blackwell Systems\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://go.dev/\"\u003e\u003cimg src=\"https://img.shields.io/badge/go-1.23+-blue.svg\" alt=\"Go\"\u003e\u003c/a\u003e\n  \u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-MIT-green.svg\" alt=\"License: MIT\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/blackwell-systems/mcp-assert\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/blackwell-systems/mcp-assert/main/assets/badge-passing.svg?v=3\" alt=\"mcp-assert: passing\" height=\"20\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/blackwell-systems/mcp-assert\"\u003e\u003cimg src=\"https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/blackwell-systems/mcp-assert/main/assets/downloads-badge.json\" alt=\"Downloads\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n**Test your MCP server against the real protocol. No mocks. No imports. No language lock-in.**\n\nmcp-assert connects to your server exactly like Claude, Cursor, or any MCP client would: real stdio/SSE/HTTP transport, full initialize handshake, actual tool calls. It checks responses against expectations you define in YAML. If it passes mcp-assert, it works with every MCP client.\n\n\u003e [!WARNING]\n\u003e We scanned 58 MCP servers and found **32 real bugs** across 13 servers. The most common failure: tools crash instead of returning errors agents can recover from. See the [scorecard](https://blackwell-systems.github.io/mcp-assert/scorecard/).\n\n```\nYour YAML        ──→  mcp-assert  ──→  MCP Server\n(inputs + assertions)    (client)        (any language)\n                            │\n                        Pass / Fail\n```\n\n### Your server can't tell the difference\n\nmcp-assert speaks the full MCP protocol: initialize handshake, `tools/list` discovery, `tools/call` with real arguments. It finds bugs that unit tests miss because it tests over the wire, not in-process.\n\n### Adopted in production\n\n- **[Wyre Technology](https://github.com/wyre-technology)**: 25 MCP servers tested via shared baseline workflow using `mcp-assert-action`\n- **[Ant Group (AntV)](https://github.com/antvis/mcp-server-chart)**: integrated into CI within 3 days of launch\n- **[Vera](https://github.com/aallan/vera)**: recommended test harness on project roadmap ([#529](https://github.com/aallan/vera/issues/529))\n- **Fix PRs merged**: Google, Grafana, LangChain, official MCP SDKs\n\nThe testing standard for MCP, like pytest for Python or Jest for JavaScript.\n\nAdd it to any MCP server project in one line:\n\n```yaml\n- uses: blackwell-systems/mcp-assert-action@v1\n  with:\n    suite: evals/\n```\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/demo.gif\" alt=\"mcp-assert demo\" width=\"720\"\u003e\n\u003c/p\u003e\n\n\u003e [!NOTE]\n\u003e LLMs are for subjective outputs. Assertions are for deterministic ones. Most MCP tools are deterministic. mcp-assert covers them.\n\n## Install\n\n```bash\n# npm (no Go required)\nnpx @blackwell-systems/mcp-assert\n\n# pip (no Go required)\npip install mcp-assert\n\n# Go\ngo install github.com/blackwell-systems/mcp-assert/cmd/mcp-assert@latest\n\n# Homebrew\nbrew install blackwell-systems/tap/mcp-assert\n\n# Docker\ndocker run blackwellsystems/mcp-assert audit --server \"npx my-server\"\n\n# Snap (Linux)\nsudo snap install mcp-assert --classic\n\n# Scoop (Windows)\nscoop bucket add blackwell-systems https://github.com/blackwell-systems/scoop-bucket\nscoop install mcp-assert\n\n# Winget (Windows)\nwinget install BlackwellSystems.mcp-assert\n\n# curl | sh (macOS / Linux)\ncurl -fsSL https://raw.githubusercontent.com/blackwell-systems/mcp-assert/main/install.sh | sh\n```\n\n## Quick Start\n\n### Audit any MCP server in seconds. No setup.\n\nPoint it at any server:\n\n```bash\nmcp-assert audit --server \"npx my-mcp-server\"\n```\n\n```\n  Server: my-server\n  Transport: stdio\n  Score: 83%\n\n  ✓ read_query      1ms  responds, returns content\n  ✗ create_table    0ms  internal error: panic: nil pointer...\n  ✓ list_tables     1ms  responds, returns content\n\n  3 tools tested, 2 healthy, 1 crashed\n```\n\n\u003e [!TIP]\n\u003e The audit connects, discovers every tool via `tools/list`, calls each one with schema-generated inputs, and reports which tools crash vs. handle errors properly. No YAML needed. To go deeper, generate assertion files and customize them:\n\n\n```bash\n# Audit + generate starter YAML for CI\nmcp-assert audit --server \"npx my-mcp-server\" --output evals/\n\n# Edit the generated YAMLs: add expected content, setup steps, multi-step flows\n\n# Run in CI with regression detection\nmcp-assert ci --suite evals/ --threshold 95\n```\n\n### Write assertions from scratch\n\n```bash\n# Scaffold your first assertion\nmcp-assert init evals                   # Or: init evals --server \"my-server\" for auto-generation\n\n# Run it\nmcp-assert run --suite evals/ --fixture evals/fixtures\n```\n\nSee the [Getting Started guide](https://blackwell-systems.github.io/mcp-assert/getting-started/) for a full walkthrough.\n\n### Already using Vitest, Jest, Bun, PHPUnit, or pytest?\n\n```bash\n# Vitest\nnpm install -D vitest-mcp-assert\n```\n\n```ts\n// mcp.test.ts\nimport { describeMcpSuite } from 'vitest-mcp-assert'\ndescribeMcpSuite('mcp server', 'evals/')\n```\n\n```bash\n# pytest\npip install pytest-mcp-assert\npytest --mcp-suite evals/\n```\n\n\u003e [!IMPORTANT]\n\u003e Same YAML files work across the CLI, Vitest, Jest, Bun, PHPUnit, pytest, and Go test. No migration needed. Write once, run anywhere.\n\n## Everything you can do\n\n| Command | What it does | Setup required |\n|---------|-------------|----------------|\n| `audit --server \"...\"` | Scan any server, classify every tool as healthy/crashed/timed out | None |\n| `fuzz --server \"...\"` | Throw adversarial inputs at every tool, find crashes and hangs | None |\n| `init --server \"...\"` | Generate a complete test suite from tools/list + capture snapshots | None |\n| `run --suite evals/` | Run YAML assertions, report pass/fail | YAML files |\n| `ci --suite evals/` | Run with thresholds, baselines, JUnit XML, GitHub Step Summary | YAML files |\n| `coverage --suite evals/ --server \"...\"` | Report which tools have assertions and which don't | YAML files |\n| `snapshot --suite evals/ --update` | Capture responses as golden files for regression detection | YAML files |\n| `watch --suite evals/` | Re-run on YAML changes, show diffs when status flips | YAML files |\n| `matrix --languages go:gopls,ts:tsserver` | Same suite across multiple language servers | YAML files |\n| `intercept --server \"...\" --trajectory t.yaml` | Proxy between agent and server, capture live tool call trace | Trajectory YAML |\n| `lint --server \"...\"` | Check tool schemas for missing descriptions, untyped params, and agent usability issues | None |\n\nStart with `audit` (zero setup), then `fuzz` (adversarial testing), then `init` (generates everything), then customize the YAML for your specific assertions.\n\n## Zero-Effort Coverage\n\n```bash\n# Generate stub assertions for every tool the server exposes\nmcp-assert generate --server \"my-mcp-server\" --output evals/ --fixture ./fixtures\n\n# Capture actual outputs as snapshots\nmcp-assert snapshot --suite evals/ --server \"my-mcp-server\" --update\n\n# Assert nothing changed\nmcp-assert run --suite evals/ --server \"my-mcp-server\"\n```\n\n## How It Differs From LLM-as-Judge Frameworks\n\nFor deterministic tools, mcp-assert is the better fit. For subjective outputs, LLM-as-judge frameworks remain the right choice. Use both if your server mixes tool types.\n\n| Dimension | LLM-as-judge eval frameworks | mcp-assert |\n|---|---|---|\n| Best for | Subjective outputs (prose, creative content) | Deterministic outputs (data, state, validation) |\n| Grading | Language model scoring (flexible, costly) | Assertion-based (exact, free) |\n| Speed | Seconds per test (LLM round-trip) | Milliseconds per test (no LLM) |\n| CI cost | API calls on every run | Zero external dependencies |\n| Reliability | Not measured | pass@k / pass^k per assertion |\n| Regression | Not supported | Baseline comparison, fail on backslide |\n| Multi-language | Not supported | Same assertion across N language servers |\n\n## Why not just write tests?\n\nYou'd need MCP protocol bootstrapping, a server-agnostic runner (your Go tests can't test your TypeScript server), and eval features (regression detection, Docker isolation, JUnit output). mcp-assert handles all of that. One YAML file, any server, any language.\n\n## CI Integration\n\nUse the [mcp-assert GitHub Action](https://github.com/blackwell-systems/mcp-assert-action) for zero-setup CI:\n\n```yaml\n- uses: blackwell-systems/mcp-assert-action@v1\n  with:\n    suite: evals/\n    threshold: 95\n```\n\nDownloads the binary, runs assertions, uploads JUnit XML results, writes GitHub Step Summary. No Go toolchain required on your runners.\n\nOr run directly:\n\n```bash\nmcp-assert ci --suite evals/ --threshold 95 --junit results.xml\n```\n\nSee the [CI Integration guide](https://blackwell-systems.github.io/mcp-assert/ci-integration/) for JUnit XML, markdown summaries, badges, and regression detection.\n\n## pytest Integration\n\nRun mcp-assert assertions as pytest test items:\n\n```bash\npip install pytest-mcp-assert\npytest --mcp-suite evals/\n```\n\nEach YAML file becomes a pytest Item with pass/fail/skip semantics. Configure via `pyproject.toml`:\n\n```toml\n[tool.pytest.ini_options]\nmcp_suite = \"evals/\"\nmcp_fixture = \"fixtures/\"\n```\n\nThen just run `pytest`. See `pytest-plugin/README.md` for all options.\n\n## Vitest Integration\n\nRun mcp-assert assertions as Vitest tests:\n\n```bash\nnpm install -D vitest-mcp-assert\n```\n\nAuto-discover all YAML files in a directory:\n\n```ts\n// mcp.test.ts\nimport { describeMcpSuite } from 'vitest-mcp-assert'\ndescribeMcpSuite('mcp server', 'evals/')\n```\n\nOr run individual assertions:\n\n```ts\nimport { test } from 'vitest'\nimport { runMcpAssert } from 'vitest-mcp-assert'\ntest('echo tool', () =\u003e runMcpAssert('evals/echo.yaml'))\n```\n\nSame YAML files work across Vitest, pytest, and the CLI. See `vitest-plugin/README.md` for all options.\n\n## Documentation\n\nFull documentation is available at [blackwell-systems.github.io/mcp-assert](https://blackwell-systems.github.io/mcp-assert):\n\n- [Getting Started](https://blackwell-systems.github.io/mcp-assert/getting-started/): install, scaffold, first run\n- [Writing Assertions](https://blackwell-systems.github.io/mcp-assert/writing-assertions/): YAML format, all 18 assertion types + 4 trajectory types, 8 block types, 6 test framework plugins (pytest, Vitest, Jest, Bun, PHPUnit, Go test), setup steps, capture, fixtures\n- [CLI Reference](https://blackwell-systems.github.io/mcp-assert/cli/): full command reference with flags and examples\n- [Examples](https://blackwell-systems.github.io/mcp-assert/examples/): 65 example suites across 8 languages (606 assertions)\n- [CI Integration](https://blackwell-systems.github.io/mcp-assert/ci-integration/): GitHub Action, JUnit XML, regression detection\n- [Badge](https://blackwell-systems.github.io/mcp-assert/badge/): add the \"Works with mcp-assert\" badge to your server README\n- [Architecture](https://blackwell-systems.github.io/mcp-assert/architecture/): internals and design decisions\n- [Roadmap](https://blackwell-systems.github.io/mcp-assert/roadmap/): what's shipped and what's next\n- [Scorecard](https://blackwell-systems.github.io/mcp-assert/scorecard/): 32 bugs found across 13 servers, 9 fix PRs submitted, 58 servers scanned\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/download-stats.svg?v=2\" alt=\"Download stats\" width=\"320\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/blackwell-systems/mcp-assert\"\u003e\n    \u003cpicture\u003e\n      \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"assets/star-cta.png\"\u003e\n      \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"assets/star-cta-light.png\"\u003e\n      \u003cimg src=\"assets/star-cta-light.png\" alt=\"Star mcp-assert on GitHub\" width=\"600\"\u003e\n    \u003c/picture\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackwell-systems%2Fmcp-assert","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblackwell-systems%2Fmcp-assert","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackwell-systems%2Fmcp-assert/lists"}