{"id":32777190,"url":"https://github.com/modelcontextprotocol/conformance","last_synced_at":"2026-02-05T18:01:18.908Z","repository":{"id":322308366,"uuid":"1017407368","full_name":"modelcontextprotocol/conformance","owner":"modelcontextprotocol","description":"Conformance Tests for MCP","archived":false,"fork":false,"pushed_at":"2026-01-23T18:33:28.000Z","size":604,"stargazers_count":31,"open_issues_count":14,"forks_count":17,"subscribers_count":5,"default_branch":"main","last_synced_at":"2026-01-25T12:07:06.317Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/modelcontextprotocol.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-10T13:39:08.000Z","updated_at":"2026-01-23T18:33:31.000Z","dependencies_parsed_at":"2025-12-15T21:13:43.701Z","dependency_job_id":null,"html_url":"https://github.com/modelcontextprotocol/conformance","commit_stats":null,"previous_names":["modelcontextprotocol/conformance"],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/modelcontextprotocol/conformance","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modelcontextprotocol%2Fconformance","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modelcontextprotocol%2Fconformance/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modelcontextprotocol%2Fconformance/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modelcontextprotocol%2Fconformance/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/modelcontextprotocol","download_url":"https://codeload.github.com/modelcontextprotocol/conformance/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modelcontextprotocol%2Fconformance/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29128621,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-05T17:12:17.649Z","status":"ssl_error","status_checked_at":"2026-02-05T17:11:23.670Z","response_time":65,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-11-04T14:00:28.221Z","updated_at":"2026-02-05T18:01:18.899Z","avatar_url":"https://github.com/modelcontextprotocol.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MCP Conformance Test Framework\n\nA framework for testing MCP (Model Context Protocol) client and server implementations against the specification.\n\n\u003e [!WARNING]\n\u003e This repository is a work in progress and is unstable. Join the conversation in the #conformance-testing-wg in the MCP Contributors discord.\n\n**For SDK maintainers:** See [SDK Integration Guide](./SDK_INTEGRATION.md) for a streamlined guide on integrating conformance tests into your SDK repository.\n\n## Quick Start\n\n### Testing Clients\n\n```bash\n# Using the everything-client (recommended)\nnpx @modelcontextprotocol/conformance client --command \"tsx examples/clients/typescript/everything-client.ts\" --scenario initialize\n\n# Run an entire suite of tests\nnpx @modelcontextprotocol/conformance client --command \"tsx examples/clients/typescript/everything-client.ts\" --suite auth\n```\n\n### Testing Servers\n\n```bash\n# Run all server scenarios (default)\nnpx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp\n\n# Run a single scenario\nnpx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --scenario server-initialize\n```\n\n### List Available Scenarios\n\n```bash\nnpx @modelcontextprotocol/conformance list\n```\n\n## Overview\n\nThe conformance test framework validates MCP implementations by:\n\n**For Clients:**\n\n1. Starting a test server for the specified scenario\n2. Running the client implementation with the test server URL\n3. Capturing MCP protocol interactions\n4. Running conformance checks against the specification\n5. Generating detailed test results\n\n**For Servers:**\n\n1. Connecting to the running server as an MCP client\n2. Sending test requests and capturing responses\n3. Running conformance checks against server behavior\n4. Generating detailed test results\n\n## Usage\n\n### Client Testing\n\n```bash\nnpx @modelcontextprotocol/conformance client --command \"\u003cclient-command\u003e\" --scenario \u003cscenario-name\u003e [options]\n```\n\n**Options:**\n\n- `--command` - The command to run your MCP client (can include flags)\n- `--scenario` - The test scenario to run (e.g., \"initialize\")\n- `--suite` - Run a suite of tests in parallel (e.g., \"auth\")\n- `--expected-failures \u003cpath\u003e` - Path to YAML baseline file of known failures (see [Expected Failures](#expected-failures))\n- `--timeout` - Timeout in milliseconds (default: 30000)\n- `--verbose` - Show verbose output\n\nThe framework appends `\u003cserver-url\u003e` as an argument to your command and sets the `MCP_CONFORMANCE_SCENARIO` environment variable to the scenario name. For scenarios that require additional context (e.g., client credentials), the `MCP_CONFORMANCE_CONTEXT` environment variable contains a JSON object with scenario-specific data.\n\n### Server Testing\n\n```bash\nnpx @modelcontextprotocol/conformance server --url \u003curl\u003e [--scenario \u003cscenario\u003e]\n```\n\n**Options:**\n\n- `--url` - URL of the server to test\n- `--scenario \u003cscenario\u003e` - Test scenario to run (e.g., \"server-initialize\"). Runs all available scenarios by default\n- `--suite \u003csuite\u003e` - Suite to run: \"active\" (default), \"all\", or \"pending\"\n- `--expected-failures \u003cpath\u003e` - Path to YAML baseline file of known failures (see [Expected Failures](#expected-failures))\n- `--verbose` - Show verbose output\n\n## Test Results\n\n**Client Testing** - Results are saved to `results/\u003cscenario\u003e-\u003ctimestamp\u003e/`:\n\n- `checks.json` - Array of conformance check results with pass/fail status\n- `stdout.txt` - Client stdout output\n- `stderr.txt` - Client stderr output\n\n**Server Testing** - Results are saved to `results/server-\u003cscenario\u003e-\u003ctimestamp\u003e/`:\n\n- `checks.json` - Array of conformance check results with pass/fail status\n\n## Expected Failures\n\nSDKs that don't yet pass all conformance tests can specify a baseline of known failures. This allows running conformance tests in CI without failing, while still catching regressions.\n\nCreate a YAML file listing expected failures by mode:\n\n```yaml\n# conformance-baseline.yml\nserver:\n  - tools-call-with-progress\n  - resources-subscribe\nclient:\n  - sse-retry\n```\n\nThen pass it to the CLI:\n\n```bash\nnpx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --expected-failures ./conformance-baseline.yml\n```\n\n**Exit code behavior:**\n\n| Scenario Result | In Baseline? | Outcome                                   |\n| --------------- | ------------ | ----------------------------------------- |\n| Fails           | Yes          | Exit 0 — expected failure                 |\n| Fails           | No           | Exit 1 — unexpected regression            |\n| Passes          | Yes          | Exit 1 — stale baseline, remove the entry |\n| Passes          | No           | Exit 0 — normal pass                      |\n\nThis ensures:\n\n- CI passes when only known failures occur\n- CI fails on new regressions (unexpected failures)\n- CI fails when a fix lands but the baseline isn't updated (stale entries)\n\n## GitHub Action\n\nThis repo provides a composite GitHub Action so SDK repos don't need to write their own conformance scripts.\n\n### Server Testing\n\n```yaml\nsteps:\n  - uses: actions/checkout@v4\n\n  # Start your server (SDK-specific)\n  - run: |\n      my-server --port 3001 \u0026\n      timeout 15 bash -c 'until curl -s http://localhost:3001/mcp; do sleep 0.5; done'\n\n  - uses: modelcontextprotocol/conformance@v0.1.11\n    with:\n      mode: server\n      url: http://localhost:3001/mcp\n      expected-failures: ./conformance-baseline.yml # optional\n```\n\n### Client Testing\n\n```yaml\nsteps:\n  - uses: actions/checkout@v4\n\n  - uses: modelcontextprotocol/conformance@v0.1.11\n    with:\n      mode: client\n      command: 'python tests/conformance/client.py'\n      expected-failures: ./conformance-baseline.yml # optional\n```\n\n### Action Inputs\n\n| Input               | Required    | Description                                     |\n| ------------------- | ----------- | ----------------------------------------------- |\n| `mode`              | Yes         | `server` or `client`                            |\n| `url`               | Server mode | URL of the server to test                       |\n| `command`           | Client mode | Command to run the client under test            |\n| `expected-failures` | No          | Path to YAML baseline file                      |\n| `suite`             | No          | Test suite to run                               |\n| `scenario`          | No          | Run a single scenario by name                   |\n| `timeout`           | No          | Timeout in ms for client tests (default: 30000) |\n| `verbose`           | No          | Show verbose output (default: false)            |\n| `node-version`      | No          | Node.js version (default: 20)                   |\n\n## Example Clients\n\n- `examples/clients/typescript/everything-client.ts` - Single client that handles all scenarios based on scenario name (recommended)\n- `examples/clients/typescript/test1.ts` - Simple MCP client (for reference)\n- `examples/clients/typescript/auth-test.ts` - Well-behaved OAuth client (for reference)\n\n## Available Scenarios\n\n### Client Scenarios\n\n- **initialize** - Tests MCP client initialization handshake\n  - Validates protocol version\n  - Validates clientInfo (name and version)\n  - Validates server response handling\n- **tools-call** - Tests tool invocation\n- **auth/basic-dcr** - Tests OAuth Dynamic Client Registration flow\n- **auth/basic-metadata-var1** - Tests OAuth with authorization metadata\n\n### Server Scenarios\n\nRun `npx @modelcontextprotocol/conformance list --server` to see all available server scenarios, including:\n\n- **server-initialize** - Tests server initialization and capabilities\n- **tools-list** - Tests tool listing endpoint\n- **tools-call-\\*** - Various tool invocation scenarios\n- **resources-\\*** - Resource management scenarios\n- **prompts-\\*** - Prompt management scenarios\n\n## Architecture\n\nSee `src/runner/DESIGN.md` for detailed architecture documentation.\n\n### Key Components\n\n- **Runner** (`src/runner/`) - Orchestrates test execution and result generation\n  - `client.ts` - Client testing implementation\n  - `server.ts` - Server testing implementation\n  - `utils.ts` - Shared utilities\n  - `index.ts` - Public API exports\n- **CLI** (`src/index.ts`) - Command-line interface using Commander.js\n- **Scenarios** (`src/scenarios/`) - Test scenarios with expected behaviors\n- **Checks** (`src/checks/`) - Conformance validation functions\n- **Types** (`src/types.ts`) - Shared type definitions\n\n## Adding New Scenarios\n\n1. Create a new directory in `src/scenarios/\u003cscenario-name\u003e/`\n2. Implement the `Scenario` interface with `start()`, `stop()`, and `getChecks()`\n3. Register the scenario in `src/scenarios/index.ts`\n\nSee `src/scenarios/initialize/` for a reference implementation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodelcontextprotocol%2Fconformance","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmodelcontextprotocol%2Fconformance","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodelcontextprotocol%2Fconformance/lists"}