{"id":31425977,"url":"https://github.com/alpic-ai/mcp-eval","last_synced_at":"2026-02-09T21:17:58.107Z","repository":{"id":313754606,"uuid":"1048960278","full_name":"alpic-ai/mcp-eval","owner":"alpic-ai","description":"Evaluation tool for MCP servers","archived":false,"fork":false,"pushed_at":"2025-09-29T11:31:40.000Z","size":198,"stargazers_count":9,"open_issues_count":3,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-29T13:25:08.614Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alpic-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-02T09:28:08.000Z","updated_at":"2025-09-29T11:31:44.000Z","dependencies_parsed_at":null,"dependency_job_id":"3ea32ee8-0fe6-4005-b920-fc8efe17db90","html_url":"https://github.com/alpic-ai/mcp-eval","commit_stats":null,"previous_names":["alpic-ai/mcp-eval"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/alpic-ai/mcp-eval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alpic-ai%2Fmcp-eval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alpic-ai%2Fmcp-eval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alpic-ai%2Fmcp-eval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alpic-ai%2Fmcp-eval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alpic-ai","download_url":"https://codeload.github.com/alpic-ai/mcp-eval/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alpic-ai%2Fmcp-eval/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":277526730,"owners_count":25833447,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-29T02:00:09.175Z","response_time":84,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-30T05:02:18.558Z","updated_at":"2026-02-09T21:17:58.096Z","avatar_url":"https://github.com/alpic-ai.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"\u003ch1\u003emcp-eval\u003c/h1\u003e\n\nA CLI to evaluate MCP servers performance\n\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\n[![Version](https://img.shields.io/npm/v/mcp-eval.svg)](https://npmjs.org/package/mcp-eval)\n[![Downloads/week](https://img.shields.io/npm/dw/mcp-eval.svg)](https://npmjs.org/package/mcp-eval)\n\n\u003c!-- toc --\u003e\n* [Quick start](#quick-start)\n* [Requirements](#requirements)\n* [Usage](#usage)\n* [Commands](#commands)\n* [Assistant configuration](#assistant-configuration)\n* [Test Suite Syntax](#test-suite-syntax)\n\u003c!-- tocstop --\u003e\n\n# Quick start\n\n- Export your Openrouter API key as `OPENROUTER_API_KEY` environment variable``\n\n```\n$ export OPENROUTER_API_KEY=\u003cyour-key\u003e\n```\n\n- Write your `myserver.yml` test case\n\n```yml\ntest_cases:\n  - name: \"Open a contribution PR on Github\"\n    input_prompt: \"I'd like to contribute to mcp-eval. I want to enable ... feature. I'll let you go ahead and implement the feature as you see fit. Open a pull request with the proposed modification once you're done.\"\n    expected_tool_call:\n      tool_name: \"open-pr\"\n      parameters:\n        branch: \"new-feature\"\n```\n\n- Run your test suite\n\n```\n$ npx -y @alpic-ai/mcp-eval@latest run --url=https://mcp.github.com ./myserver.yml\n```\n\n- Et voilà 🎉!\n\n# Requirements\n\n- Nodejs \u003e= 22\n- StreamableHTTP or SSE compatible public MCP server\n\n# Usage\n\n\u003c!-- usage --\u003e\n```sh-session\n$ npm install -g @alpic-ai/mcp-eval\n$ mcp-eval COMMAND\nrunning command...\n$ mcp-eval (--version)\n@alpic-ai/mcp-eval/0.11.0 darwin-arm64 node-v22.20.0\n$ mcp-eval --help [COMMAND]\nUSAGE\n  $ mcp-eval COMMAND\n...\n```\n\u003c!-- usagestop --\u003e\n\n# Commands\n\n\u003c!-- commands --\u003e\n* [`mcp-eval run TESTFILE`](#mcp-eval-run-testfile)\n\n## `mcp-eval run TESTFILE`\n\nRun the test suite described in the provided YAML file.\n\n```\nUSAGE\n  $ mcp-eval run TESTFILE -u \u003cvalue\u003e [-a anthropic/claude|openai/chatgpt|mistral/le-chat]\n\nARGUMENTS\n  TESTFILE  YAML file path containing the test suite\n\nFLAGS\n  -a, --assistant=\u003coption\u003e  [default: anthropic/claude] Assistant configuration to use (impact model and system prompt)\n                            \u003coptions: anthropic/claude|openai/chatgpt|mistral/le-chat\u003e\n  -u, --url=\u003cvalue\u003e         (required) URL of the MCP server\n\nDESCRIPTION\n  Run the test suite described in the provided YAML file.\n\nEXAMPLES\n  $ mcp-eval run\n```\n\n_See code: [src/commands/run.ts](https://github.com/alpic-ai/mcp-eval/blob/v0.11.0/src/commands/run.ts)_\n\u003c!-- commandsstop --\u003e\n\n# Assistant configuration\n\nYou can use the `-a` flag to specify the assistant configuration to use.\n\nCurrently, both `anthropic/claude` and `openai/chatgpt` are supported.\n\nThis will impact the model used, the system prompt and the default tools available to the assistant.\n\n## anthropic/claude\n\n- Model: `anthropic/claude-3.7-sonnet`\n- Default tools: `drive_search`, `web_search`\n\n## openai/chatgpt\n\n- Model: `openai/gpt-5`\n- Default tools: `bio`, `automations`, `canmore_create_textdoc`, `canmore_update_textdoc`, `canmore_comment_textdoc`, `file_search`, `image_gen`, `python`, `guardian_tool`, `web`\n\n# Test Suite Syntax\n\nTest suite should be written in YAML.\nA test suite file should have a root `test_cases` property with at least one test.\n\nEach test requires:\n\n- `name`: a convenient name for your test\n- `input_conversation`: the conversation to send to the assistant. This can be either a single user message or a multi-turn conversation containing both assistant and user messages. It can contain tool calls that already happened during the model thinking process.\n- `expected_tool_call`: an object detailing the expected tool to be called with:\n  - `tool_name`: the name as advertized by the MCP server of the tool to be called\n  - `parameters`: the expected set of parameters the tool is expected to be called with. Only these specified properties will be checked during comparison with the actual tool call. Extra properties set by the model will not cause the test to fail.\n\n## Simple user message example\n\n```yml\ntest_cases:\n  - name: \"Find flights from Paris to Tokyo\"\n    input_prompt: \"I'd like to plan a trip to Tokyo, Japan. Find me a flight from Paris to Tokyo on October 3rd and returning on October 5th.\"\n    expected_tool_call:\n      tool_name: \"search-flight\"\n      parameters:\n        flyFrom: Paris\n        flyTo: Tokyo\n        departureDate: 03/10/2025\n        returnDate: 05/10/2025\n```\n\n## Multi-turn conversation example\n\n```yml\ntest_cases:\n  - name: \"Create issue in frontend team for login bug\"\n    input_conversation:\n      - role: user\n        content: \"I'm seeing a bug where the login button doesn't work. Can you create an issue for this?\"\n      - role: assistant\n        content: \"Sure, first let me check which team to assign the issue to. Listing your teams now.\"\n      - role: tool\n        tool_name: list_teams\n        parameters: {}\n        response: |\n          [\n            {\"id\": \"team_123\", \"name\": \"Frontend\"},\n            {\"id\": \"team_456\", \"name\": \"Backend\"}\n          ]\n      - role: assistant\n        content: \"Now that I see the available teams, I'll assign the issue to the Frontend team.\"\n    expected_tool_call:\n      tool_name: \"create_issue\"\n      parameters:\n        title: \"Login button doesn't work\"\n        description: \"User reports that the login button is not functioning.\"\n        team_id: \"team_123\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpic-ai%2Fmcp-eval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falpic-ai%2Fmcp-eval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpic-ai%2Fmcp-eval/lists"}