{"id":50332191,"url":"https://github.com/ericchansen/mcp-tester","last_synced_at":"2026-05-29T10:30:46.961Z","repository":{"id":355959387,"uuid":"1230408541","full_name":"ericchansen/mcp-tester","owner":"ericchansen","description":"Instrumented MCP server + driver + analyzer for testing client conformance to list_changed notifications","archived":false,"fork":false,"pushed_at":"2026-05-06T01:41:01.000Z","size":31,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-06T03:26:02.823Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ericchansen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-06T01:09:25.000Z","updated_at":"2026-05-06T01:41:04.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ericchansen/mcp-tester","commit_stats":null,"previous_names":["ericchansen/mcp-tester"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ericchansen/mcp-tester","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchansen%2Fmcp-tester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchansen%2Fmcp-tester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchansen%2Fmcp-tester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchansen%2Fmcp-tester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ericchansen","download_url":"https://codeload.github.com/ericchansen/mcp-tester/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchansen%2Fmcp-tester/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33648529,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-29T02:00:06.066Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-29T10:30:46.086Z","updated_at":"2026-05-29T10:30:46.956Z","avatar_url":"https://github.com/ericchansen.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mcp-tester\n\nInstrumented MCP server for testing client conformance to `*/list_changed` notifications and other protocol features.\n\n## Why\n\n[PR mcaps-microsoft/msx-mcp#321](https://github.com/mcaps-microsoft/msx-mcp/pull/321) proposed dynamic tool surfacing via `notifications/tools/list_changed`. The PR was closed with the claim that \"Copilot CLI caches `tools/list` at session start and ignores `notifications/tools/list_changed`\" -- but no instrumented experiment was ever run to prove or disprove this.\n\nThis repo provides the apparatus to test that claim -- and any MCP client's conformance to `list_changed` -- with real evidence.\n\n## What It Does\n\n1. **Probe MCP Server** (`src/server.ts`) -- A self-instrumenting MCP server that logs every JSON-RPC message on its stdio pipe. It exposes alpha-state tools, resources, and prompts, plus trigger tools that swap them to beta-state and send the corresponding `*/list_changed` notification.\n\n2. **Automated Client Driver** (`src/driver.ts`) -- An SDK-based MCP `Client` that drives the server through the full mutation cycle. This is the **control**: it proves the protocol works when both endpoints honor the spec. Any client that diverges from this behavior is provably non-conformant.\n\n3. **Log Analyzer** (`src/analyze.ts`) -- Reads the JSONL log and emits a markdown evidence table answering specific questions: Did the client receive the notification? Did it re-fetch? Can it call the new item? Can it still call the removed item?\n\n4. **Human Protocol** (`PROTOCOL.md`) -- Step-by-step instructions for running the same test against Copilot CLI or any other MCP client.\n\n## Quick Start\n\n```bash\ngit clone https://github.com/ericchansen/mcp-tester.git\ncd mcp-tester\nnpm install\nnpm run build\n```\n\n### Run the automated control (SDK client)\n\n```bash\nnpm run driver -- --domain all\n```\n\nThis spawns the probe server, runs the full mutation protocol for tools, resources, and prompts, and reports results. You should see all three domains pass with full conformance.\n\n### Analyze a log\n\n```bash\nnpm run analyze -- path/to/log.jsonl --domain tools\n```\n\n## Testing Your Own MCP Client\n\nSee [PROTOCOL.md](PROTOCOL.md) for step-by-step instructions on testing Copilot CLI or any other MCP client against this probe.\n\n## Repo Layout\n\n```\nmcp-tester/\n|-- README.md          # This file\n|-- PROTOCOL.md        # Human-driven test procedure\n|-- LICENSE            # MIT\n|-- package.json\n|-- tsconfig.json\n|-- src/\n|   |-- server.ts      # Probe MCP server (stdio)\n|   |-- logger.ts      # JSONL stdio tap + log writer\n|   |-- analyze.ts     # CLI: parses log -\u003e markdown evidence table\n|   +-- driver.ts      # CLI: spawns server, drives it via SDK Client\n+-- results/\n    +-- copilot-cli/\n        +-- .gitkeep   # Committed experiment results go here\n```\n\n## How the Probe Works\n\n### Initial State (Alpha)\n\n| Domain    | Alpha Item                   |\n|-----------|------------------------------|\n| Tools     | `mcptester_probe_alpha`      |\n| Resources | `probe://alpha`              |\n| Prompts   | `mcptester_alpha_prompt`     |\n\nPlus five permanent tools: `mcptester_mutate_tools`, `mcptester_mutate_resources`, `mcptester_mutate_prompts`, `mcptester_status`.\n\n### After Mutation (Beta)\n\nWhen a `mcptester_mutate_*` tool is called:\n\n1. The alpha item is removed (via SDK `handle.remove()`)\n2. The beta item is registered\n3. The SDK auto-sends the corresponding `notifications/*/list_changed`\n4. `debouncedNotificationMethods` coalesces remove + register into a single notification\n\n| Domain    | Beta Item                    |\n|-----------|------------------------------|\n| Tools     | `mcptester_probe_beta`       |\n| Resources | `probe://beta`               |\n| Prompts   | `mcptester_beta_prompt`      |\n\n### Logging\n\nEvery JSON-RPC message (inbound and outbound) is logged to a JSONL file. The logger taps `process.stdin` and `process.stdout` via Transform streams installed before the SDK transport, so it sees every byte without interfering with the protocol.\n\n## Limitations\n\n- **stdio transport only** -- no HTTP/SSE testing yet\n- **Three `list_changed` flows only** -- future tests planned for sampling, elicitation, roots, progress, completions\n- **Single-server testing** -- tests one MCP server connection at a time\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fericchansen%2Fmcp-tester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fericchansen%2Fmcp-tester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fericchansen%2Fmcp-tester/lists"}