{"id":46665971,"url":"https://github.com/inspectr-hq/mcplab","last_synced_at":"2026-04-17T10:11:57.319Z","repository":{"id":341396836,"uuid":"1151743556","full_name":"inspectr-hq/mcplab","owner":"inspectr-hq","description":"MCPLab - Test and evaluate MCP servers with LLMs.","archived":false,"fork":false,"pushed_at":"2026-03-01T14:16:13.000Z","size":2812,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-01T17:43:08.186Z","etag":null,"topics":["agents","ai","cli","evaluation","llm","mcp","model-context-protocol","testing"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/inspectr-hq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-06T21:02:04.000Z","updated_at":"2026-03-01T14:16:16.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/inspectr-hq/mcplab","commit_stats":null,"previous_names":["inspectr-hq/mcplab"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/inspectr-hq/mcplab","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inspectr-hq%2Fmcplab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inspectr-hq%2Fmcplab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inspectr-hq%2Fmcplab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inspectr-hq%2Fmcplab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/inspectr-hq","download_url":"https://codeload.github.com/inspectr-hq/mcplab/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/inspectr-hq%2Fmcplab/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30175865,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T11:48:51.886Z","status":"ssl_error","status_checked_at":"2026-03-06T11:48:51.460Z","response_time":250,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agents","ai","cli","evaluation","llm","mcp","model-context-protocol","testing"],"created_at":"2026-03-08T18:03:44.282Z","updated_at":"2026-04-17T10:11:57.310Z","avatar_url":"https://github.com/inspectr-hq.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MCPLab 🧪\n\n\u003e **Test, debug and evaluate Model Context Protocol servers with LLMs**\n\nTest how well LLM agents use your MCP tools, compare different models, and track quality over time with automated testing and detailed reports.\n\n\u003cp align=\"left\"\u003e\n  \u003ca href=\"https://www.npmjs.com/package/@inspectr/mcplab\"\u003e\u003cimg src=\"https://img.shields.io/npm/v/@inspectr/mcplab\" alt=\"Total Downloads\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.npmjs.com/package/@inspectr/mcplab\"\u003e\u003cimg src=\"https://img.shields.io/npm/dw/@inspectr/mcplab\" alt=\"Latest Stable Version\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n## What is MCPLab?\n\nMCPLab is a testing and evaluation framework for [MCP servers](https://modelcontextprotocol.io). It helps you:\n\n- **Validate** that LLM agents correctly use your MCP tools\n- **Compare** different LLMs (Claude, GPT-4, etc.) on the same tasks\n- **Track** tool usage patterns, success rates, and performance metrics\n- **Automate** quality assurance in CI/CD pipelines\n- **Debug** agent behavior with detailed execution traces\n\nPerfect for MCP server developers who want to ensure their tools work reliably across different AI models.\n\n[![MCPLab](https://mcplab.inspectr.dev/screenshots/evaluation-results-assistance.png)](https://mcplab.inspectr.dev/)\n\nVisit [mcplab.inspectr.dev](https://mcplab.inspectr.dev/) to learn more.\n\n---\n\n## ✨ Features\n\n### Core Capabilities\n- **HTTP SSE Transport** - Test MCP servers over Streamable HTTP\n- **Multi-LLM Support** - OpenAI, Anthropic Claude, Azure OpenAI\n- **Rich Assertions** - Validate tool usage, sequences, and response content\n- **Variance Testing** - Run multiple iterations to measure stability\n- **Detailed Traces** - JSONL logs of every tool call and LLM response\n\n### Analysis \u0026 Reporting\n- **Trend Analysis** - Track pass rates and performance over time\n- **LLM Comparison** - Built-in tools to compare agent behavior\n- **Multiple Outputs** - HTML report, JSON results, Markdown summary, JSONL trace\n- **Custom Metrics** - Extract values and track domain-specific KPIs\n- **Markdown Reports** - Store and browse custom analysis notes alongside runs\n\n### AI-Powered Tools (App Mode)\n- **Scenario Assistant** - AI chat to help design and refine eval scenarios\n- **Result Assistant** - AI chat to analyze and explain completed run results\n- **MCP Tool Analysis** - Automated review of MCP tool quality and safety\n\n### Developer Experience\n- **Watch Mode** - Auto-rerun tests when configs change\n- **YAML Configuration** - Declarative, version-controllable eval specs\n- **Interactive Reports** - Self-contained HTML with filtering and drill-down\n- **Multi-Agent Testing** - Compare LLMs with a single CLI flag\n- **Scenario Isolation** - Run specific tests or full suites\n\n---\n\n## 🚀 Quick Start\n\n### 1. Install\n\n```bash\nnpx @inspectr/mcplab --help\n```\n\nOr install globally:\n\n```bash\nnpm install -g @inspectr/mcplab\n```\n\n### 2. Set up environment\n\n```bash\ncp .env.example .env\n# Edit .env and add your API keys:\n# ANTHROPIC_API_KEY=sk-ant-...\n# OPENAI_API_KEY=sk-...\n```\n\nAdd your API keys to `.env`. See [Environment Variables](#-environment-variables) for full examples.\n\n### 3. Run your first evaluation\n\n```bash\n# Run the app (frontend + local API bridge)\nnpx @inspectr/mcplab app --open\n\n# Run an evaluation from a config file\nnpx @inspectr/mcplab run -c mcplab/evals/eval.yaml\n\n# View the results\nopen mcplab/results/evaluation-runs/$(ls -t mcplab/results/evaluation-runs | head -1)/report.html\n```\n\n### 4. Create your own test\n\nCreate `my-eval.yaml`:\n\n```yaml\nservers:\n  - id: my-server\n    transport: \"http\"\n    url: \"http://localhost:3000/mcp\"\n\nagents:\n  - id: claude\n    provider: \"anthropic\"\n    model: \"claude-haiku-4-5-20251001\"\n    temperature: 0\n    max_tokens: 2048\n\nscenarios:\n  - id: \"basic-test\"\n    servers: [\"my-server\"]\n    prompt: \"Use the available tools to complete this task...\"\n    eval:\n      tool_constraints:\n        required_tools: [\"my_tool\"]\n      response_assertions:\n        - type: \"regex\"\n          pattern: \"success|completed\"\n```\n\nRun it:\n\n```bash\nmcplab run -c my-eval.yaml\n```\n\n---\n\n## 📖 Configuration Guide\n\n### Structure Overview\n\nAdd this at the top of your eval file for editor validation/autocomplete:\n\n```yaml\n# yaml-language-server: $schema=./config-schema.json\n```\n\n```yaml\nservers:     # MCP servers to test against\n  - id: local-server\n    transport: \"http\"\n    url: \"http://localhost:3000/mcp\"\n  - ref: \"shared-server\"\n\nagents:      # LLM agents to use for testing\n  - id: local-agent\n    provider: \"anthropic\"\n    model: \"claude-sonnet-4-6\"\n  - ref: \"claude-sonnet-46\"\n\nscenarios:   # Test scenarios to run\n  - id: \"basic-test\"\n    servers: [\"local-server\"]\n    prompt: \"...\"\n  - ref: \"scn-shared-basic\"\n```\n\n### Servers\n\nDefine MCP servers with connection details and authentication:\n\n```yaml\nservers:\n  - id: my-server\n    transport: \"http\"\n    url: \"https://api.example.com/mcp\"\n    auth:\n      type: \"bearer\"\n      token: ${MCP_TOKEN}        # env var reference\n      # token: my-secret-token   # or direct value\n```\n\n**Authentication types:**\n\n**Bearer Token:**\n```yaml\nauth:\n  type: \"bearer\"\n  token: ${MCP_TOKEN}        # env var reference\n  # token: my-secret-token   # or direct value\n```\n\n**API Key:**\n```yaml\nauth:\n  type: \"api_key\"\n  header_name: \"X-API-Key\"     # optional, defaults to X-API-Key\n  value: ${MY_API_KEY}\n```\n\n**OAuth Client Credentials:**\n```yaml\nauth:\n  type: \"oauth_client_credentials\"\n  token_url: \"https://auth.example.com/token\"\n  client_id_env: \"CLIENT_ID\"\n  client_secret_env: \"CLIENT_SECRET\"\n  scope: \"read:data\"              # Optional\n  audience: \"https://api.example.com\"  # Optional\n```\n\n### Agents\n\nConfigure LLM agents with provider-specific settings:\n\n**Anthropic (Claude):**\n```yaml\nagents:\n  - id: claude-sonnet\n    provider: \"anthropic\"\n    model: \"claude-sonnet-4-6\"\n    temperature: 0\n    max_tokens: 2048\n    system: \"You are a helpful assistant.\"\n```\n\n**OpenAI (ChatGPT):**\n```yaml\nagents:\n  - id: gpt-4\n    provider: \"openai\"\n    model: \"gpt-4o-mini\"\n    temperature: 0\n    max_tokens: 2048\n    system: \"You are a helpful assistant.\"\n```\n\n**Azure OpenAI:**\n```yaml\nagents:\n  - id: azure-gpt\n    provider: \"azure_openai\"\n    model: \"gpt-4o\"  # Deployment name\n    temperature: 0\n    max_tokens: 2048\n    system: \"You are a helpful assistant.\"\n```\n\n**Required environment variables:**\n- Anthropic: `ANTHROPIC_API_KEY`\n- OpenAI: `OPENAI_API_KEY`\n- Azure: `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_DEPLOYMENT`\n\n### Scenarios\n\nDefine test scenarios with prompts and evaluation criteria:\n\n```yaml\nscenarios:\n  - id: \"search-and-analyze\"\n    servers: [\"my-server\"]\n    prompt: |\n      Search for items matching criteria X,\n      then analyze the results and provide insights.\n\n    eval:\n      # Validate tool usage\n      tool_constraints:\n        required_tools: [\"search\", \"analyze\"]\n        forbidden_tools: [\"delete\"]\n\n      # Validate tool sequence\n      tool_sequence:\n        allow:\n          - [\"search\", \"analyze\"]\n          - [\"search\", \"search\", \"analyze\"]\n\n      # Validate response content\n      response_assertions:\n        - type: \"regex\"\n          pattern: \"found \\\\d+ items\"\n        - type: \"jsonpath\"\n          path: \"$.summary.count\"\n          equals: 10\n\n    # Extract metrics\n    extract:\n      - name: \"item_count\"\n        from: \"final_text\"\n        regex: \"found (?\u003cvalue\u003e\\\\d+) items\"\n\nrun_defaults:\n  selected_agents:\n    - claude-sonnet\n```\n\n**Evaluation options:**\n\n- **`tool_constraints`** - Which tools must/must not be used\n  - `required_tools`: Tools that must be called\n  - `forbidden_tools`: Tools that must not be called\n\n- **`tool_sequence`** - Valid sequences of tool calls\n  - `allow`: List of allowed sequences (e.g., `[[\"search\", \"analyze\"]]`)\n\n- **`response_assertions`** - Validate the final response\n  - `regex`: Pattern matching on response text\n  - `jsonpath`: Query and validate JSON responses\n\n- **`extract`** - Extract metrics from responses\n  - Capture values using regex named groups: `(?\u003cvalue\u003e...)`\n\n---\n\n## 🔑 Environment Variables\n\nAdd your LLM Agent API keys to `.env` for each provider you want to use:\n\n**Anthropic (Claude models):**\n```env\n# -----------------------------------------------------------------------------\n# Anthropic Configuration\n# -----------------------------------------------------------------------------\n# Required for testing Claude models (claude-haiku-4, claude-sonnet-4)\n\nANTHROPIC_API_KEY=sk-ant-api03-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n```\n\n**Azure OpenAI (GPT models):**\n```env\n# -----------------------------------------------------------------------------\n# Azure OpenAI Configuration\n# -----------------------------------------------------------------------------\n# Required for testing GPT models (gpt-4o-mini, gpt-4o, etc.)\n\nAZURE_OPENAI_API_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\nAZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com\nAZURE_OPENAI_DEPLOYMENT=\"gpt-5.2-chat\"\nAZURE_OPENAI_API_VERSION=\"2025-04-01-preview\"\n```\n\n**OpenAI:**\n```env\n# -----------------------------------------------------------------------------\n# OpenAI Configuration\n# -----------------------------------------------------------------------------\nOPENAI_API_KEY=sk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n```\n\n---\n\n## 💡 Usage Examples\n\n### Basic Usage\n\n```bash\n# Run all scenarios\nmcplab run -c mcplab/evals/eval.yaml\n\n# Interactive mode (choose config + agents)\nmcplab run --interactive\n\n# Run specific scenario\nmcplab run -c mcplab/evals/eval.yaml -s basic-test\n\n# Run with variance testing (5 iterations)\nmcplab run -c mcplab/evals/eval.yaml -n 5\n```\n\n### App Mode\n\nServe the web app and local API in one process:\n\n```bash\nmcplab app --open\n\n# Interactive startup (host/port/paths prompt + summary)\nmcplab app --interactive\n```\n\nOptional custom paths:\n\n```bash\nmcplab app --evals-dir mcplab/evals --runs-dir mcplab/results/evaluation-runs --port 8787 --open\n```\n\nOptional development mode (proxy frontend to Vite, keep API local):\n\n```bash\nmcplab app --dev\n```\n\n### Multi-LLM Testing\n\nCompare how different LLMs perform on the same tasks:\n\n```bash\n# Test with multiple agents\nmcplab run -c examples/eval.yaml \\\n  --agents claude-haiku,gpt-4o-mini,gpt-4o\n\n# This runs each scenario with each agent automatically\n# 3 scenarios × 3 agents = 9 tests\n\n# Compare results\nnode scripts/compare-llm-results.mjs mcplab/results/evaluation-runs/LATEST/results.json\n```\n\nOutput:\n```\n📊 LLM Performance Comparison\n\nLLM              | Pass Rate | Avg Tools/Run | Avg Duration (ms)\n-----------------|-----------|---------------|------------------\nclaude-haiku     |    100.0% |           2.5 |               850\ngpt-4o-mini      |     88.9% |           2.8 |               950\ngpt-4o           |     88.9% |           3.2 |              1200\n\n💡 Key Insights\n• Highest Pass Rate: claude-haiku (100.0%)\n• Fastest: claude-haiku (850ms avg)\n• Most Efficient: claude-haiku (2.5 tools/run)\n```\n\n### Watch Mode\n\nAuto-rerun tests when config changes:\n\n```bash\nmcplab watch -c examples/eval.yaml\n\n# With multi-agent testing\nmcplab watch -c examples/eval.yaml \\\n  --agents claude-haiku,gpt-4o-mini\n```\n\n### Snapshot Baselines\n\nCreate a smart baseline from a fully passing run, then compare later runs against it:\n\n```bash\n# Create a snapshot (source run must be fully passing)\nmcplab snapshot create --run 20260208-140213 --name \"weather-api-baseline-v1\"\n\n# List snapshots\nmcplab snapshot list\n\n# Compare run against snapshot\nmcplab snapshot compare --id \u003csnapshotId\u003e --run 20260208-150045\n```\n\nOptional: compare immediately after a run:\n\n```bash\nmcplab run -c mcplab/evals/eval.yaml --compare-snapshot \u003csnapshotId\u003e\n```\n\nConfig-first snapshot eval workflow:\n\n```bash\n# Initialize snapshot eval policy in a config from a fully passing run\nmcplab snapshot eval-init --config mcplab/evals/eval.yaml --run 20260208-140213 --name \"baseline-v1\"\n\n# Update snapshot eval policy mode\nmcplab snapshot eval-policy --config mcplab/evals/eval.yaml --enabled true --mode fail_on_drift\n\n# Apply config snapshot policy during run (warn or fail_on_drift)\nmcplab run -c mcplab/evals/eval.yaml --snapshot-eval\n```\n\n### Generate Reports\n\n```bash\n# Regenerate HTML report from previous run\nmcplab report --input mcplab/results/evaluation-runs/20260206-212239\n\n# Interactive run selection from recent runs\nmcplab report --interactive\n```\n\n---\n\n## 🤖 AI-Powered Features\n\nThese features are available through the web app (`mcplab app`).\n\n### Scenario Assistant\n\nAn interactive AI chat that helps you design and refine evaluation scenarios. Given a scenario, it can suggest improvements to the prompt, evaluation rules, and extraction patterns — and can call your MCP server's tools directly to demonstrate expected behavior.\n\nOpen the app, navigate to an eval, and open the **Scenario Assistant** panel on any scenario.\n\n### Result Assistant\n\nAn AI chat that analyzes completed evaluation runs. Ask it to explain failures, identify patterns across scenarios, or summarize what went wrong in a specific run. It has read-only access to run artifacts, traces, and results.\n\nOpen a run in the app and click **Result Assistant**.\n\n### MCP Tool Analysis\n\nAutomated quality review of your MCP server's tools. Connects to your server, discovers all tools, and produces a report covering:\n\n- Name and description quality\n- Schema completeness\n- Safety classification (read-like vs. potentially destructive)\n- Sample call behavior (optional — runs real calls against your server)\n\nReports are saved to `mcplab/results/tool-analysis/` and viewable in the app.\n\nNavigate to **Tool Analysis** in the app sidebar to start an analysis job.\n\n### Markdown Reports\n\nStore and browse custom analysis notes, comparison docs, or generated reports alongside your eval runs. Place `.md` files in `mcplab/reports/` and they become accessible in the app under **Reports**.\n\n---\n\n## 📚 Reusable configurations\n\nDefine servers, agents, and scenarios once and reuse them across multiple eval files.\n\n```\nmcplab/\n├── servers.yaml       # Shared MCP server definitions\n├── agents.yaml        # Shared LLM agent definitions\n└── scenarios/\n    ├── scenario-a.yaml\n    └── scenario-b.yaml\n```\n\nReference library items in eval configs:\n\n```yaml\nservers:\n  - ref: \"my-server\"          # from servers.yaml\nagents:\n  - ref: \"claude-sonnet\"      # from agents.yaml\nscenarios:\n  - ref: \"scenario-a\"         # from scenarios/scenario-a.yaml\n```\n\nLibraries can be managed through the app's **Libraries** page.\n\n---\n\n## 📂 Output Structure\n\nEach evaluation run creates a timestamped directory:\n\n```\nmcplab/results/evaluation-runs/20260206-212239/\n├── trace.jsonl        # Detailed execution log (every tool call, LLM response)\n├── results.json       # Structured results (pass/fail, metrics, aggregates)\n├── summary.md         # Human-readable summary table\n└── report.html        # Interactive HTML report (self-contained)\n```\n\nOther output directories:\n\n```\nmcplab/\n├── evals/                          # Eval definition YAML files\n├── results/\n│   ├── evaluation-runs/            # Run artifacts\n│   └── tool-analysis/              # Saved tool analysis reports\n├── snapshots/                      # Snapshot baselines\n├── reports/                        # Custom markdown reports\n├── servers.yaml                    # Library: shared server definitions\n├── agents.yaml                     # Library: shared agent definitions\n└── scenarios/                      # Library: shared scenario files\n```\n\n### Trace Format (JSONL)\n\n```jsonl\n{\"type\":\"run_started\",\"run_id\":\"...\",\"ts\":\"2026-02-06T20:03:54.585Z\"}\n{\"type\":\"scenario_started\",\"scenario_id\":\"search-tags\",\"agent\":\"claude-haiku\",\"ts\":\"...\"}\n{\"type\":\"llm_request\",\"messages_summary\":\"user:Search for tags...\",\"ts\":\"...\"}\n{\"type\":\"llm_response\",\"raw_or_summary\":\"tool_calls:search_tags\",\"ts\":\"...\"}\n{\"type\":\"tool_call\",\"server\":\"demo\",\"tool\":\"search_tags\",\"args\":{...},\"ts_start\":\"...\"}\n{\"type\":\"tool_result\",\"server\":\"demo\",\"tool\":\"search_tags\",\"ok\":true,\"result_summary\":\"...\",\"ts_end\":\"...\",\"duration_ms\":1114}\n{\"type\":\"final_answer\",\"text\":\"Found 42 tags matching...\",\"ts\":\"...\"}\n{\"type\":\"scenario_finished\",\"scenario_id\":\"search-tags\",\"pass\":true,\"metrics\":{...},\"ts\":\"...\"}\n```\n\n### Results Format (JSON)\n\n```json\n{\n  \"metadata\": {\n    \"run_id\": \"20260206-212239\",\n    \"timestamp\": \"2026-02-06T20:22:39.000Z\",\n    \"config_hash\": \"abc123...\",\n    \"git_commit\": \"def456...\"\n  },\n  \"summary\": {\n    \"total_scenarios\": 8,\n    \"total_runs\": 8,\n    \"pass_rate\": 1.0,\n    \"avg_tool_calls_per_run\": 2.5,\n    \"avg_tool_latency_ms\": 950\n  },\n  \"scenarios\": [...]\n}\n```\n\n---\n\n## 🎓 Real-World Examples\n\n### Example 1: Weather MCP Server\n\nTest a weather data MCP server:\n\n```bash\n# Run comprehensive test suite (9 scenarios)\nmcplab run -c examples/eval-weather-comprehensive.yaml\n\n# Test a specific scenario\nmcplab run -c examples/eval-weather-comprehensive.yaml \\\n  -s forecast-accuracy\n\n# Compare Claude vs GPT-4 on all scenarios\nmcplab run -c examples/eval-weather-simple.yaml \\\n  --agents claude-haiku,gpt-4o-mini\n```\n\n**Included scenarios:**\n- Current conditions lookup\n- Multi-day forecast retrieval\n- Location search and resolution\n- Severe weather alerts\n- Historical data queries\n- Unit conversion (metric/imperial)\n\n### Example 2: Multi-Agent Comparison\n\nCreate `multi-agent-eval.yaml` with one agent defined:\n\n```yaml\nagents:\n  - id: claude-haiku\n    provider: anthropic\n    model: claude-haiku-4-5-20251001\n  - id: gpt-4o-mini\n    provider: openai\n    model: gpt-4o-mini\n  - id: gpt-4o\n    provider: openai\n    model: gpt-4o\n\nscenarios:\n  - id: \"complex-task\"\n    prompt: \"...\"\n\nrun_defaults:\n  selected_agents:\n    - claude-haiku\n```\n\nRun with all agents:\n\n```bash\nmcplab run -c multi-agent-eval.yaml \\\n  --agents claude-haiku,gpt-4o-mini,gpt-4o \\\n  -n 5\n\n# 1 scenario × 3 agents × 5 runs = 15 tests\n```\n\n### Example 3: CI/CD Integration\n\nAdd to `.github/workflows/mcp-eval.yml`:\n\n```yaml\nname: MCP Evaluation\n\non: [push, pull_request]\n\njobs:\n  evaluate:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: actions/setup-node@v4\n        with:\n          node-version: '22'\n\n      - run: npm install\n      - run: npm run build\n\n      - name: Run evaluations\n        run: mcplab run -c examples/eval.yaml -n 3\n        env:\n          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}\n          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\n\n      - name: Upload results\n        uses: actions/upload-artifact@v4\n        with:\n          name: evaluation-results\n          path: mcplab/results/evaluation-runs/\n```\n\n---\n\n## 🛠️ Advanced Features\n\n### Custom Analysis Scripts\n\nAnalyze results with custom logic:\n\n```javascript\n// my-analysis.mjs\nimport { readFileSync } from 'fs';\n\nconst results = JSON.parse(readFileSync('mcplab/results/evaluation-runs/LATEST/results.json'));\n\n// Calculate custom metrics\nfor (const scenario of results.scenarios) {\n  const efficiency = scenario.pass_rate / scenario.runs[0].tool_call_count;\n  console.log(`${scenario.scenario_id}: ${efficiency.toFixed(2)} success/tool`);\n}\n```\n\n### Generate Multi-LLM Configs\n\nAuto-generate multi-agent configs:\n\n```bash\n# Creates eval-weather-multi-llm.yaml\nnode scripts/generate-multi-llm-config.mjs examples/eval-weather.yaml\n```\n\n### Compare LLM Performance\n\nBuilt-in comparison script:\n\n```bash\nnode scripts/compare-llm-results.mjs mcplab/results/evaluation-runs/20260206-212239/results.json\n```\n\nShows:\n- Pass rates by LLM\n- Tool usage efficiency\n- Response times\n- Scenario-by-scenario breakdown\n\n---\n\n## 🔧 Development\n\n### Project Structure\n\n```\nmcp-evaluation/\n├── packages/\n│   ├── cli/           # CLI tool (run, watch, report, app commands)\n│   ├── app/           # Web frontend (React)\n│   ├── core/          # Evaluation engine, agent adapters, MCP client\n│   └── reporting/     # HTML report generation\n├── examples/          # Example evaluation configs\n├── scripts/           # Utility scripts (multi-LLM, comparison)\n├── mcplab/results/    # Evaluation results + analysis (gitignored)\n└── .claude/           # Claude Code skills (optional)\n```\n\n### Run in Development Mode\n\n```bash\n# Build all packages\nnpm run build\n\n# Run CLI directly with tsx (no build needed)\nnpm run dev -- app --dev\n\n# Or run just the frontend dev server\nnpm run app:dev:ui\n```\n\n### Run Tests\n\n```bash\nnpm test\n```\n\n---\n\n## 🤝 Contributing\n\nContributions welcome! Please:\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n---\n\n## License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\n---\n\n## Acknowledgments\n\nBuilt with:\n- [Model Context Protocol SDK](https://github.com/modelcontextprotocol/sdk)\n- [Anthropic SDK](https://github.com/anthropics/anthropic-sdk-typescript)\n- [OpenAI SDK](https://github.com/openai/openai-node)\n\n---\n\n## Support\n\n- [Issue Tracker](https://github.com/inspectr-hq/mcplab/issues)\n- [Discussions](https://github.com/inspectr-hq/mcplab/discussions)\n- [MCP Protocol](https://modelcontextprotocol.io)\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n**⭐ Star this repo if you find it useful!**\n\nMade with ❤️ by [Inspectr](https://inspectr.dev) for the MCP community\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finspectr-hq%2Fmcplab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finspectr-hq%2Fmcplab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finspectr-hq%2Fmcplab/lists"}