{"id":30801049,"url":"https://github.com/loadmill/test-mcp","last_synced_at":"2026-01-20T17:29:24.612Z","repository":{"id":312865525,"uuid":"1040858836","full_name":"loadmill/test-mcp","owner":"loadmill","description":"Automated testing tool for MCP servers and agents","archived":false,"fork":false,"pushed_at":"2025-09-02T13:13:29.000Z","size":123,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-09-02T14:32:18.253Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/loadmill.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-19T15:57:18.000Z","updated_at":"2025-09-02T13:19:25.000Z","dependencies_parsed_at":"2025-09-02T14:34:07.714Z","dependency_job_id":"ab01f521-1972-4315-a54a-c7e3ac3bbcff","html_url":"https://github.com/loadmill/test-mcp","commit_stats":null,"previous_names":["loadmill/test-mcp"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/loadmill/test-mcp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loadmill%2Ftest-mcp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loadmill%2Ftest-mcp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loadmill%2Ftest-mcp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loadmill%2Ftest-mcp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/loadmill","download_url":"https://codeload.github.com/loadmill/test-mcp/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loadmill%2Ftest-mcp/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273819824,"owners_count":25174116,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-05T02:00:09.113Z","response_time":402,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-05T20:49:23.281Z","updated_at":"2026-01-20T17:29:24.597Z","avatar_url":"https://github.com/loadmill.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 📦 test-mcp\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://www.npmjs.com/package/@loadmill/test-mcp\"\u003e\u003cimg src=\"https://img.shields.io/npm/v/@loadmill/test-mcp?color=green\" alt=\"npm version\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://discord.gg/BHAVZUFrWX\"\u003e\u003cimg src=\"https://img.shields.io/discord/1412375815236091906?logo=discord\u0026label=discord\" alt=\"Discord\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#whats-test-mcp\"\u003eWhat's test-mcp?\u003c/a\u003e •\n  \u003ca href=\"#installation\"\u003eInstallation\u003c/a\u003e •\n  \u003ca href=\"#getting-started\"\u003eGetting Started\u003c/a\u003e •\n  \u003ca href=\"#configuration--test-format\"\u003eConfiguration \u0026 Test Format\u003c/a\u003e •\n  \u003ca href=\"#how-to-run\"\u003eHow to Run\u003c/a\u003e •\n  \u003ca href=\"#cli-flags\"\u003eCLI Flags\u003c/a\u003e •\n  \u003ca href=\"#test-discovery\"\u003eTest Discovery\u003c/a\u003e •\n  \u003ca href=\"#interactive-mode\"\u003eInteractive Mode\u003c/a\u003e •\n  \u003ca href=\"#programmatic-api\"\u003eProgrammatic API\u003c/a\u003e •\n  \u003ca href=\"#roadmap\"\u003eRoadmap\u003c/a\u003e •\n  \u003ca href=\"#contributing\"\u003eContributing\u003c/a\u003e •\n  \u003ca href=\"#license\"\u003eLicense\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n**`test-mcp`** is a headless MCP client for automated testing of MCP servers and agents.\n\nIf you’re building an MCP server or agent, `test-mcp` lets you run natural-language test scripts and assertions end-to-end, so you can validate behavior in a fast and repeatable way.\n\nhttps://github.com/user-attachments/assets/c4e5295b-7217-47f2-96d9-76befcea8b21\n\n---\n\n\u003ch2 id=\"whats-test-mcp\"\u003e💡 What's `test-mcp`?\u003c/h2\u003e\n\n`test-mcp` gives you three core components:\n\n* **Configuration** – define your MCP servers and LLM provider in a single JSON file.\n* **Test Files** – write flows of natural-language prompts and assertions in YAML.\n* **Runner** – run tests from the CLI, get clear pass/fail results.\n\nTogether, these let you automate and validate MCP server behavior with simple, repeatable tests.\n\n**Supported Transports \u0026 Providers:**\n- **MCP Servers**: STDIO (local) and HTTP (remote)\n- **LLM Providers**: Anthropic Claude and OpenAI GPT models\n- **MCP Features**: Tools (done), Resources/Prompts/Sampling (planned)\n\n---\n\n\u003ch2 id=\"installation\"\u003e🏗️ Installation\u003c/h2\u003e\n\n```bash\n# using npm\nnpm install -g @loadmill/test-mcp\n\n# or with pnpm\npnpm add -g @loadmill/test-mcp\n````\n\n\u003e [!NOTE]  \n\u003e While `test-mcp` itself can run on Node.js 18 or higher, many popular MCP servers require Node.js 20.  \n\u003e For the smoothest experience, we recommend using **Node.js 20**.\n\nWhen running from source:\n\n```bash\ngit clone https://github.com/loadmill/test-mcp\ncd test-mcp\nnpm install\n# For OpenAI (default example)\necho \"OPENAI_API_KEY=your_api_key_here\" \u003e .env\n# Or for Anthropic\necho \"ANTHROPIC_API_KEY=your_api_key_here\" \u003e .env\nnpm run build\n```\n\n---\n\n\u003ch2 id=\"getting-started\"\u003e🚀 Getting Started\u003c/h2\u003e\n\nTo try `test-mcp` quickly with the included examples:\n\n```bash\n# from source\nnode build/index.js\n```\n\nThis will run a demonstration that shows both local STDIO and remote HTTP MCP servers working together. The test rolls a local dice server and queries a remote MCP server registry.\n\n---\n\n\u003ch2 id=\"configuration--test-format\"\u003e📑 Configuration \u0026 Test Format\u003c/h2\u003e\n\n**1) Example config (`mcp.config.json`)**\n\n```json\n{\n  \"mcpClient\": {\n    \"provider\": \"openai\",\n    \"model\": \"gpt-4o-mini\",\n    \"api_key\": \"${env:OPENAI_API_KEY}\"\n  },\n  \"mcpServers\": {\n    \"loadmill\": {\n      \"type\": \"stdio\",\n      \"command\": \"npx\",\n      \"args\": [\"@loadmill/mcp\"],\n      \"env\": {\n        \"LOADMILL_API_TOKEN\": \"${env:LOADMILL_API_TOKEN}\"\n      }\n    }\n  }\n}\n```\n\nOpenAI is also supported - see configuration variations in the `examples/` folder.\n\n**2) Example test (`tests/bank-transaction.test.yaml`)**\n\n```yaml\ndescription: \"Maker Checker Bank - Transaction Creation and Rejection Flow\"\n\nsteps:\n  - prompt: \"Login with username alice and password alice123 and transfer $100 to Bob\"\n  - prompt: \"Login with username bob and password bob456, reject transaction from Alice\"\n  - assert: \"Validate the transaction was created and rejected successfully\"\n```\n\n---\n\n\u003ch2 id=\"how-to-run\"\u003e▶️ How to run\u003c/h2\u003e\n\nBy default, `test-mcp` looks for `mcp.config.json` in the project root and runs tests in the `tests/` folder.\n\n**Globally installed:**\n\n```bash\ntest-mcp\n```\n\n**From source:**\n\n```bash\nnode build/index.js\n```\n\nPoint to a specific config or tests directory:\n\n```bash\ntest-mcp --config mcp.config.json --tests-dir ./tests\n```\n\n---\n\n\u003ch2 id=\"cli-flags\"\u003e💻 CLI Flags\u003c/h2\u003e\n\n```\nOptions:\n  -c, --config \u003cfile\u003e   Path to config file (default: mcp.config.json)\n  -t, --tests-dir \u003cdir\u003e Directory containing test files (default: tests)\n  -i, --interactive     Run in interactive chat mode\n      --trace           Enable detailed tracing output\n  -h, --help            Show help\n```\n\n---\n\n\u003ch2 id=\"test-discovery\"\u003e🔎 Test Discovery\u003c/h2\u003e\n\nAll files ending in `.test.yaml` under the `tests/` directory are executed.\nRecursive discovery and full glob patterns are planned for later.\n\n---\n\n\u003ch2 id=\"interactive-mode\"\u003e💬 Interactive Mode\u003c/h2\u003e\n\nRun the client without tests and chat with your MCP servers:\n\n```bash\ntest-mcp -i\n```\n\n---\n\n\u003ch2 id=\"programmatic-api\"\u003e🔧 Programmatic API\u003c/h2\u003e\n\nYou can use `test-mcp` programmatically in your Node.js code:\n\n```javascript\nimport { TestMCPClient } from '@loadmill/test-mcp';\n\nconst client = new TestMCPClient({\n  llm: {\n    provider: 'openai',\n    model: 'gpt-4o-mini',\n    apiKey: process.env.OPENAI_API_KEY\n  },\n  servers: {\n    myServer: {\n      type: 'stdio',\n      command: 'node',\n      args: ['./server.js']\n    }\n  }\n});\n\nawait client.connect();\nconst response = await client.prompt('Your question');\nconst assertion = await client.assert('Expected behavior');\nawait client.disconnect();\n```\n\nSee `examples/api-example.js` for a complete example.\n\n---\n\n\u003ch2 id=\"roadmap\"\u003e🛣️ Roadmap\u003c/h2\u003e\n\n* [x] Headless MCP client with Anthropic support\n* [x] Support for `stdio` transport\n* [x] Support for MCP tools\n* [x] Evaluator for natural-language assertions\n* [x] OpenAI support\n* [x] Support for `http` transport\n* [ ] Test parameterization with `${}`\n* [ ] CI-friendly reports\n* [ ] Support for MCP resources\n* [ ] Support for MCP prompts\n* [ ] Support for MCP sampling\n\n---\n\n\u003ch2 id=\"contributing\"\u003e🤝 Contributing\u003c/h2\u003e\n\nContributions, ideas, and bug reports are welcome! See [CONTRIBUTING.md](https://github.com/loadmill/test-mcp/blob/main/.github/CONTRIBUTING.md).\n\n---\n\n\u003ch2 id=\"license\"\u003e📄 License\u003c/h2\u003e\n\nApache License 2.0 © [Loadmill](https://github.com/loadmill/test-mcp/blob/main/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Floadmill%2Ftest-mcp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Floadmill%2Ftest-mcp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Floadmill%2Ftest-mcp/lists"}