https://github.com/loadmill/test-mcp

Automated testing tool for MCP servers and agents
https://github.com/loadmill/test-mcp

Last synced: 5 months ago
JSON representation

Automated testing tool for MCP servers and agents

Host: GitHub
URL: https://github.com/loadmill/test-mcp
Owner: loadmill
License: apache-2.0
Created: 2025-08-19T15:57:18.000Z (10 months ago)
Default Branch: master
Last Pushed: 2025-09-02T13:13:29.000Z (9 months ago)
Last Synced: 2025-09-02T14:32:18.253Z (9 months ago)
Language: TypeScript
Homepage:
Size: 120 KB
Stars: 0
Watchers: 0
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md
- Contributing: .github/CONTRIBUTING.md
- License: LICENSE

Awesome Lists containing this project

README

# 📦 test-mcp

What's test-mcp? •
Installation •
Getting Started •
Configuration & Test Format •
How to Run •
CLI Flags •
Test Discovery •
Interactive Mode •
Programmatic API •
Roadmap •
Contributing •
License

---

**`test-mcp`** is a headless MCP client for automated testing of MCP servers and agents.

If you’re building an MCP server or agent, `test-mcp` lets you run natural-language test scripts and assertions end-to-end, so you can validate behavior in a fast and repeatable way.

https://github.com/user-attachments/assets/c4e5295b-7217-47f2-96d9-76befcea8b21

---

💡 What's `test-mcp`?

`test-mcp` gives you three core components:

* **Configuration** – define your MCP servers and LLM provider in a single JSON file.
* **Test Files** – write flows of natural-language prompts and assertions in YAML.
* **Runner** – run tests from the CLI, get clear pass/fail results.

Together, these let you automate and validate MCP server behavior with simple, repeatable tests.

**Supported Transports & Providers:**
- **MCP Servers**: STDIO (local) and HTTP (remote)
- **LLM Providers**: Anthropic Claude and OpenAI GPT models
- **MCP Features**: Tools (done), Resources/Prompts/Sampling (planned)

---

🏗️ Installation

```bash
# using npm
npm install -g @loadmill/test-mcp

# or with pnpm
pnpm add -g @loadmill/test-mcp
````

> [!NOTE]
> While `test-mcp` itself can run on Node.js 18 or higher, many popular MCP servers require Node.js 20.
> For the smoothest experience, we recommend using **Node.js 20**.

When running from source:

```bash
git clone https://github.com/loadmill/test-mcp
cd test-mcp
npm install
# For OpenAI (default example)
echo "OPENAI_API_KEY=your_api_key_here" > .env
# Or for Anthropic
echo "ANTHROPIC_API_KEY=your_api_key_here" > .env
npm run build
```

---

🚀 Getting Started

To try `test-mcp` quickly with the included examples:

```bash
# from source
node build/index.js
```

This will run a demonstration that shows both local STDIO and remote HTTP MCP servers working together. The test rolls a local dice server and queries a remote MCP server registry.

---

📑 Configuration & Test Format

**1) Example config (`mcp.config.json`)**

```json
{
"mcpClient": {
"provider": "openai",
"model": "gpt-4o-mini",
"api_key": "${env:OPENAI_API_KEY}"
},
"mcpServers": {
"loadmill": {
"type": "stdio",
"command": "npx",
"args": ["@loadmill/mcp"],
"env": {
"LOADMILL_API_TOKEN": "${env:LOADMILL_API_TOKEN}"
}
}
}
}
```

OpenAI is also supported - see configuration variations in the `examples/` folder.

**2) Example test (`tests/bank-transaction.test.yaml`)**

```yaml
description: "Maker Checker Bank - Transaction Creation and Rejection Flow"

steps:
- prompt: "Login with username alice and password alice123 and transfer $100 to Bob"
- prompt: "Login with username bob and password bob456, reject transaction from Alice"
- assert: "Validate the transaction was created and rejected successfully"
```

---

▶️ How to run

By default, `test-mcp` looks for `mcp.config.json` in the project root and runs tests in the `tests/` folder.

**Globally installed:**

```bash
test-mcp
```

**From source:**

```bash
node build/index.js
```

Point to a specific config or tests directory:

```bash
test-mcp --config mcp.config.json --tests-dir ./tests
```

---

💻 CLI Flags

```
Options:
-c, --config Path to config file (default: mcp.config.json)
-t, --tests-dir Directory containing test files (default: tests)
-i, --interactive Run in interactive chat mode
--trace Enable detailed tracing output
-h, --help Show help
```

---

🔎 Test Discovery

All files ending in `.test.yaml` under the `tests/` directory are executed.
Recursive discovery and full glob patterns are planned for later.

---

💬 Interactive Mode

Run the client without tests and chat with your MCP servers:

```bash
test-mcp -i
```

---

🔧 Programmatic API

You can use `test-mcp` programmatically in your Node.js code:

```javascript
import { TestMCPClient } from '@loadmill/test-mcp';

const client = new TestMCPClient({
llm: {
provider: 'openai',
model: 'gpt-4o-mini',
apiKey: process.env.OPENAI_API_KEY
},
servers: {
myServer: {
type: 'stdio',
command: 'node',
args: ['./server.js']
}
}
});

await client.connect();
const response = await client.prompt('Your question');
const assertion = await client.assert('Expected behavior');
await client.disconnect();
```

See `examples/api-example.js` for a complete example.

---

🛣️ Roadmap

* [x] Headless MCP client with Anthropic support
* [x] Support for `stdio` transport
* [x] Support for MCP tools
* [x] Evaluator for natural-language assertions
* [x] OpenAI support
* [x] Support for `http` transport
* [ ] Test parameterization with `${}`
* [ ] CI-friendly reports
* [ ] Support for MCP resources
* [ ] Support for MCP prompts
* [ ] Support for MCP sampling

---

🤝 Contributing

Contributions, ideas, and bug reports are welcome! See [CONTRIBUTING.md](https://github.com/loadmill/test-mcp/blob/main/.github/CONTRIBUTING.md).

---

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome