{"id":51555059,"url":"https://github.com/markndg/mcp-probe","last_synced_at":"2026-07-29T04:00:25.267Z","repository":{"id":357887412,"uuid":"1238145475","full_name":"markndg/mcp-probe","owner":"markndg","description":"Contract testing and conformance checks for Model Context Protocol (MCP) servers","archived":false,"fork":false,"pushed_at":"2026-05-14T17:03:09.000Z","size":77,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-14T18:44:54.145Z","etag":null,"topics":["ai-agents","anthropic","ci-cd","claude","cli","conformance-testing","developer-tools","integration-testing","llm","mcp","mcp-conformance","mcp-server","mcp-testing","model-context-protocol","rust","testing"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/markndg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-13T21:21:25.000Z","updated_at":"2026-05-14T17:02:32.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/markndg/mcp-probe","commit_stats":null,"previous_names":["markndg/mcp-probe"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/markndg/mcp-probe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markndg%2Fmcp-probe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markndg%2Fmcp-probe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markndg%2Fmcp-probe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markndg%2Fmcp-probe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/markndg","download_url":"https://codeload.github.com/markndg/mcp-probe/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markndg%2Fmcp-probe/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36016154,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-29T02:00:04.910Z","response_time":95,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","anthropic","ci-cd","claude","cli","conformance-testing","developer-tools","integration-testing","llm","mcp","mcp-conformance","mcp-server","mcp-testing","model-context-protocol","rust","testing"],"created_at":"2026-07-10T04:00:29.938Z","updated_at":"2026-07-29T04:00:25.249Z","avatar_url":"https://github.com/markndg.png","language":"Rust","funding_links":[],"categories":["SDKs"],"sub_categories":["Rust"],"readme":"# mcp-probe\n\n**pytest for [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) servers.**\n\nSpawn your server, complete a real MCP handshake over stdio, assert on JSON-RPC responses, and ship **JUnit** and **SARIF** to CI — in one command.\n\n```bash\nmcp-probe conformance --command python3 --server-arg examples/echo-server/server.py\n```\n\n---\n\n## Why mcp-probe?\n\nMCP is a wire protocol. Production agents call `initialize`, negotiate capabilities, then hit `tools/list`, `resources/list`, and `tools/call`. Most teams still validate that by hand.\n\n**mcp-probe** gives you:\n\n- A **declarative suite format** (JSON scenarios + steps)\n- A **built-in conformance pack** that respects optional capabilities (no false failures for servers without prompts/resources)\n- **CI-ready reports** (JUnit + SARIF + stable JSON)\n- A **GitHub Action** you can drop into any workflow\n\n---\n\n## Quickstart (\u003c 2 minutes)\n\nFrom a clone of this repository:\n\n```bash\ncargo build --release -p mcp-probe\nexport PATH=\"$PWD/target/release:$PATH\"\n\n# Built-in conformance against the in-repo example server (stdlib Python, no npm)\nmcp-probe conformance \\\n  --command python3 \\\n  --server-arg examples/echo-server/server.py\n\n# Your own suite\nmcp-probe run --config examples/suites/passing.suite.json\n```\n\nExample human output:\n\n```text\nmcp-probe 0.2.0\n  suite             : examples/suites/passing.suite.json\n  server command    : python3 examples/echo-server/server.py\n  protocol (request): 2024-11-05\n  protocol (server) : 2024-11-05\n  server capabilities: resources, tools\n\n  PASS  echo tool round-trips text\n  SKIP  prompts/list is skipped on a tools-only server\n        reason: server did not advertise `prompts` capability\n\nsummary: OK — 4 passed, 0 failed, 1 skipped (5 total)\n```\n\nTo see a **failing** run (useful when tuning assertions):\n\n```bash\nmcp-probe run --config examples/suites/failing.suite.json\necho $?   # 1\n```\n\nMore detail: [`examples/README.md`](./examples/README.md).\n\n---\n\n## GitHub Actions (one snippet)\n\n```yaml\n- uses: markndg/mcp-probe@v0.2.0\n  with:\n    server-command: python3\n    server-args: examples/echo-server/server.py\n    version: v0.2.0\n```\n\nOr run your own suite:\n\n```yaml\n- uses: markndg/mcp-probe@v0.2.0\n  with:\n    suite: tests/mcp/smoke.suite.json\n    version: source   # build from checkout; use v0.2.0 in production\n```\n\nThe action writes JUnit + SARIF under `mcp-probe-results/` and can upload SARIF to the **Code Scanning** tab. Full example: [`docs/example-workflow.yml`](./docs/example-workflow.yml).\n\n---\n\n## Install\n\n| Method | Command |\n|--------|---------|\n| **From source** | `cargo build --release -p mcp-probe` then add `target/release` to `PATH` |\n| **GitHub Action** | `uses: markndg/mcp-probe@v0.2.0` (see [`action.yml`](./action.yml)) |\n| **Python wrapper** | `pip install -e .` — shells out to the Rust binary ([`mcp_probe`](./mcp_probe/)) |\n\nRelease binaries: [GitHub Releases](https://github.com/markndg/mcp-probe/releases) (tag `v*`).\n\n---\n\n## Commands\n\n| Command | Purpose |\n|---------|---------|\n| `mcp-probe conformance` | Built-in capability-aware conformance pack |\n| `mcp-probe run --config \u003cfile\u003e` | Run a custom suite |\n| `mcp-probe validate-suite \u003cfile\u003e` | Static validation (no server) |\n| `mcp-probe init [--command …]` | Scaffold a starter suite |\n| `mcp-probe inspect --command …` | List tools/resources/prompts; optional `--write-suite` |\n| `mcp-probe record` | Run a suite + mandatory NDJSON trace |\n| `mcp-probe fuzz` | Crash smoke on first scenario step (not protocol fuzzing) |\n\nCommon flags: `--timeout-ms`, `--protocol-version` (default `2024-11-05`), `--junit`, `--sarif`, `--report`, `--format human|json`.\n\n```bash\nmcp-probe run --help\n```\n\n---\n\n## Suite format (short)\n\nSee **[`docs/SUITE_SCHEMA.md`](./docs/SUITE_SCHEMA.md)** for the full reference.\n\n```json\n{\n  \"version\": 2,\n  \"server\": { \"command\": \"my-mcp-server\", \"args\": [] },\n  \"scenarios\": [{\n    \"name\": \"lists tools\",\n    \"skip_unless_any_capability\": [\"tools\"],\n    \"steps\": [{\n      \"send\": { \"method\": \"tools/list\", \"params\": {} },\n      \"expect\": { \"result\": { \"tools\": [{ \"name\": \"search\" }] } }\n    }]\n  }]\n}\n```\n\n**Expectations** apply to the JSON-RPC `result` object: subset match, strict match, inline JSON Schema, or `result_schema_path` relative to the suite file.\n\n---\n\n## Conformance \u0026 capabilities\n\nThe built-in pack checks `tools/list`, `resources/list`, and `prompts/list` with JSON Schema. Each scenario is gated on the matching **`initialize.capabilities`** key:\n\n- Server advertises only `tools` → resources and prompts scenarios are **skipped**, run still **passes**.\n- Server advertises all three → all scenarios run.\n\nThis matches the MCP spec: resources and prompts are optional surfaces.\n\n---\n\n## Reports\n\n| Output | Flag | Consumer |\n|--------|------|----------|\n| Human summary | default stdout | Developers, CI logs |\n| JSON | `--report` or `--format json` | Scripts, Python wrapper |\n| JUnit XML | `--junit` | GitHub Actions, Jenkins, GitLab |\n| SARIF | `--sarif` | GitHub Code Scanning (per-scenario rule IDs) |\n\nSkipped scenarios appear in JUnit as `\u003cskipped message=\"…\"/\u003e` and in SARIF as informational results.\n\n---\n\n## Python\n\n```python\nfrom pathlib import Path\nfrom mcp_probe.runner import run_suite, run_conformance\n\nr = run_suite(Path(\"examples/suites/passing.suite.json\"), junit_path=Path(\"out/junit.xml\"))\nassert r.ok, r.report\n\nr = run_conformance(command=\"python3\", server_args=[\"examples/echo-server/server.py\"])\n```\n\nPytest plugin: `pytest -p mcp_probe.pytest_plugin` with fixtures `mcp_probe_bin`, `mcp_run_suite`, `mcp_run_conformance`.\n\n---\n\n## What works today\n\n| Feature | Notes |\n|---------|--------|\n| Stdio transport | Primary, production-tested path |\n| HTTP transport | Experimental JSON-RPC POST + `Mcp-Session-Id` |\n| Handshake | `initialize` + `notifications/initialized` |\n| Assertions | Subset / strict JSON, JSON Schema, JSON Pointer, expected RPC errors |\n| Conformance | Built-in + embedded `default` JSON pack (`--pack`) |\n| Capability skips | `skip_unless_any_capability` on scenarios |\n| CI | GitHub Action, JUnit, SARIF |\n\n**Non-goals:** Streamable HTTP/SSE, full protocol formal verification, protocol-aware framing fuzzing (`fuzz` is a crash smoke tool only).\n\n---\n\n## Development\n\n```bash\ncargo fmt \u0026\u0026 cargo clippy --all-targets -- -D warnings \u0026\u0026 cargo test --workspace\n```\n\n- [`CONTRIBUTING.md`](./CONTRIBUTING.md) — how to send PRs and add compatibility rows\n- [`COMPATIBILITY.md`](./COMPATIBILITY.md) — servers tested so far\n- [`CHANGELOG.md`](./CHANGELOG.md) — release notes\n- [`SECURITY.md`](./SECURITY.md) — vulnerability reporting\n- [`docs/RELEASE_CHECKLIST.md`](./docs/RELEASE_CHECKLIST.md) — maintainer release steps\n\n---\n\n## Architecture\n\n- **`mcp-probe-core`** — transport, suite model, runner, conformance, JUnit/SARIF/human reports, validation, discovery\n- **`mcp-probe`** — CLI (`clap`)\n- **`mcp_probe`** — Python subprocess wrapper\n\nEach scenario uses a **fresh server process** by default (`session: per_scenario`) for deterministic CI.\n\n---\n\n## License\n\nMIT — see [LICENSE](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarkndg%2Fmcp-probe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarkndg%2Fmcp-probe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarkndg%2Fmcp-probe/lists"}