https://github.com/markndg/mcp-probe
Contract testing and conformance checks for Model Context Protocol (MCP) servers
https://github.com/markndg/mcp-probe
ai-agents anthropic ci-cd claude cli conformance-testing developer-tools integration-testing llm mcp mcp-conformance mcp-server mcp-testing model-context-protocol rust testing
Last synced: 8 days ago
JSON representation
Contract testing and conformance checks for Model Context Protocol (MCP) servers
- Host: GitHub
- URL: https://github.com/markndg/mcp-probe
- Owner: markndg
- License: mit
- Created: 2026-05-13T21:21:25.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2026-05-14T17:03:09.000Z (3 months ago)
- Last Synced: 2026-05-14T18:44:54.145Z (3 months ago)
- Topics: ai-agents, anthropic, ci-cd, claude, cli, conformance-testing, developer-tools, integration-testing, llm, mcp, mcp-conformance, mcp-server, mcp-testing, model-context-protocol, rust, testing
- Language: Rust
- Homepage:
- Size: 75.2 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-mcp-devtools - markndg/mcp-probe - Rust CLI that validates conformance, schema correctness, and capability negotiation against any MCP server. (SDKs / Rust)
README
# mcp-probe
**pytest for [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) servers.**
Spawn your server, complete a real MCP handshake over stdio, assert on JSON-RPC responses, and ship **JUnit** and **SARIF** to CI — in one command.
```bash
mcp-probe conformance --command python3 --server-arg examples/echo-server/server.py
```
---
## Why mcp-probe?
MCP is a wire protocol. Production agents call `initialize`, negotiate capabilities, then hit `tools/list`, `resources/list`, and `tools/call`. Most teams still validate that by hand.
**mcp-probe** gives you:
- A **declarative suite format** (JSON scenarios + steps)
- A **built-in conformance pack** that respects optional capabilities (no false failures for servers without prompts/resources)
- **CI-ready reports** (JUnit + SARIF + stable JSON)
- A **GitHub Action** you can drop into any workflow
---
## Quickstart (< 2 minutes)
From a clone of this repository:
```bash
cargo build --release -p mcp-probe
export PATH="$PWD/target/release:$PATH"
# Built-in conformance against the in-repo example server (stdlib Python, no npm)
mcp-probe conformance \
--command python3 \
--server-arg examples/echo-server/server.py
# Your own suite
mcp-probe run --config examples/suites/passing.suite.json
```
Example human output:
```text
mcp-probe 0.2.0
suite : examples/suites/passing.suite.json
server command : python3 examples/echo-server/server.py
protocol (request): 2024-11-05
protocol (server) : 2024-11-05
server capabilities: resources, tools
PASS echo tool round-trips text
SKIP prompts/list is skipped on a tools-only server
reason: server did not advertise `prompts` capability
summary: OK — 4 passed, 0 failed, 1 skipped (5 total)
```
To see a **failing** run (useful when tuning assertions):
```bash
mcp-probe run --config examples/suites/failing.suite.json
echo $? # 1
```
More detail: [`examples/README.md`](./examples/README.md).
---
## GitHub Actions (one snippet)
```yaml
- uses: markndg/mcp-probe@v0.2.0
with:
server-command: python3
server-args: examples/echo-server/server.py
version: v0.2.0
```
Or run your own suite:
```yaml
- uses: markndg/mcp-probe@v0.2.0
with:
suite: tests/mcp/smoke.suite.json
version: source # build from checkout; use v0.2.0 in production
```
The action writes JUnit + SARIF under `mcp-probe-results/` and can upload SARIF to the **Code Scanning** tab. Full example: [`docs/example-workflow.yml`](./docs/example-workflow.yml).
---
## Install
| Method | Command |
|--------|---------|
| **From source** | `cargo build --release -p mcp-probe` then add `target/release` to `PATH` |
| **GitHub Action** | `uses: markndg/mcp-probe@v0.2.0` (see [`action.yml`](./action.yml)) |
| **Python wrapper** | `pip install -e .` — shells out to the Rust binary ([`mcp_probe`](./mcp_probe/)) |
Release binaries: [GitHub Releases](https://github.com/markndg/mcp-probe/releases) (tag `v*`).
---
## Commands
| Command | Purpose |
|---------|---------|
| `mcp-probe conformance` | Built-in capability-aware conformance pack |
| `mcp-probe run --config ` | Run a custom suite |
| `mcp-probe validate-suite ` | Static validation (no server) |
| `mcp-probe init [--command …]` | Scaffold a starter suite |
| `mcp-probe inspect --command …` | List tools/resources/prompts; optional `--write-suite` |
| `mcp-probe record` | Run a suite + mandatory NDJSON trace |
| `mcp-probe fuzz` | Crash smoke on first scenario step (not protocol fuzzing) |
Common flags: `--timeout-ms`, `--protocol-version` (default `2024-11-05`), `--junit`, `--sarif`, `--report`, `--format human|json`.
```bash
mcp-probe run --help
```
---
## Suite format (short)
See **[`docs/SUITE_SCHEMA.md`](./docs/SUITE_SCHEMA.md)** for the full reference.
```json
{
"version": 2,
"server": { "command": "my-mcp-server", "args": [] },
"scenarios": [{
"name": "lists tools",
"skip_unless_any_capability": ["tools"],
"steps": [{
"send": { "method": "tools/list", "params": {} },
"expect": { "result": { "tools": [{ "name": "search" }] } }
}]
}]
}
```
**Expectations** apply to the JSON-RPC `result` object: subset match, strict match, inline JSON Schema, or `result_schema_path` relative to the suite file.
---
## Conformance & capabilities
The built-in pack checks `tools/list`, `resources/list`, and `prompts/list` with JSON Schema. Each scenario is gated on the matching **`initialize.capabilities`** key:
- Server advertises only `tools` → resources and prompts scenarios are **skipped**, run still **passes**.
- Server advertises all three → all scenarios run.
This matches the MCP spec: resources and prompts are optional surfaces.
---
## Reports
| Output | Flag | Consumer |
|--------|------|----------|
| Human summary | default stdout | Developers, CI logs |
| JSON | `--report` or `--format json` | Scripts, Python wrapper |
| JUnit XML | `--junit` | GitHub Actions, Jenkins, GitLab |
| SARIF | `--sarif` | GitHub Code Scanning (per-scenario rule IDs) |
Skipped scenarios appear in JUnit as `` and in SARIF as informational results.
---
## Python
```python
from pathlib import Path
from mcp_probe.runner import run_suite, run_conformance
r = run_suite(Path("examples/suites/passing.suite.json"), junit_path=Path("out/junit.xml"))
assert r.ok, r.report
r = run_conformance(command="python3", server_args=["examples/echo-server/server.py"])
```
Pytest plugin: `pytest -p mcp_probe.pytest_plugin` with fixtures `mcp_probe_bin`, `mcp_run_suite`, `mcp_run_conformance`.
---
## What works today
| Feature | Notes |
|---------|--------|
| Stdio transport | Primary, production-tested path |
| HTTP transport | Experimental JSON-RPC POST + `Mcp-Session-Id` |
| Handshake | `initialize` + `notifications/initialized` |
| Assertions | Subset / strict JSON, JSON Schema, JSON Pointer, expected RPC errors |
| Conformance | Built-in + embedded `default` JSON pack (`--pack`) |
| Capability skips | `skip_unless_any_capability` on scenarios |
| CI | GitHub Action, JUnit, SARIF |
**Non-goals:** Streamable HTTP/SSE, full protocol formal verification, protocol-aware framing fuzzing (`fuzz` is a crash smoke tool only).
---
## Development
```bash
cargo fmt && cargo clippy --all-targets -- -D warnings && cargo test --workspace
```
- [`CONTRIBUTING.md`](./CONTRIBUTING.md) — how to send PRs and add compatibility rows
- [`COMPATIBILITY.md`](./COMPATIBILITY.md) — servers tested so far
- [`CHANGELOG.md`](./CHANGELOG.md) — release notes
- [`SECURITY.md`](./SECURITY.md) — vulnerability reporting
- [`docs/RELEASE_CHECKLIST.md`](./docs/RELEASE_CHECKLIST.md) — maintainer release steps
---
## Architecture
- **`mcp-probe-core`** — transport, suite model, runner, conformance, JUnit/SARIF/human reports, validation, discovery
- **`mcp-probe`** — CLI (`clap`)
- **`mcp_probe`** — Python subprocess wrapper
Each scenario uses a **fresh server process** by default (`session: per_scenario`) for deterministic CI.
---
## License
MIT — see [LICENSE](./LICENSE).