{"id":27168483,"url":"https://github.com/strowk/mcp-autotest","last_synced_at":"2025-04-09T05:34:10.363Z","repository":{"id":285437599,"uuid":"958177890","full_name":"strowk/mcp-autotest","owner":"strowk","description":"Utility for autotesting MCP servers","archived":false,"fork":false,"pushed_at":"2025-03-31T19:47:12.000Z","size":0,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-31T19:51:08.549Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/strowk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-31T19:12:47.000Z","updated_at":"2025-03-31T19:47:14.000Z","dependencies_parsed_at":"2025-03-31T19:51:08.938Z","dependency_job_id":null,"html_url":"https://github.com/strowk/mcp-autotest","commit_stats":null,"previous_names":["strowk/mcptest","strowk/mcp-autotest"],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/strowk%2Fmcp-autotest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/strowk%2Fmcp-autotest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/strowk%2Fmcp-autotest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/strowk%2Fmcp-autotest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/strowk","download_url":"https://codeload.github.com/strowk/mcp-autotest/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247987060,"owners_count":21028891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-09T05:34:09.816Z","updated_at":"2025-04-09T05:34:10.311Z","avatar_url":"https://github.com/strowk.png","language":"Go","funding_links":[],"categories":["Utilities","⚙️ DevOps"],"sub_categories":["Development Tools"],"readme":"\u003ch4 align=\"center\"\u003eAutotest MCP servers in a language-agnostic way\u003c/h4\u003e\r\n\r\n\u003ch1 align=\"center\"\u003e\r\n   \u003cimg src=\"docs/images/logo.png\" width=\"180\"/\u003e\r\n   \u003cbr/\u003e\r\n   mcp-autotest\r\n\u003c/h1\u003e\r\n\r\n\u003cp align=\"center\"\u003e\r\n    \u003ca href=\"https://github.com/strowk/mcp-autotest/actions/workflows/test.yaml\"\u003e\u003cimg src=\"https://github.com/strowk/mcp-autotest/actions/workflows/test.yaml/badge.svg\"\u003e\u003c/a\u003e\r\n\t\u003ca href=\"https://github.com/strowk/mcp-autotest/actions/workflows/golangci-lint.yaml\"\u003e\u003cimg src=\"https://github.com/strowk/mcp-autotest/actions/workflows/golangci-lint.yaml/badge.svg\"/\u003e\u003c/a\u003e\r\n    \u003ca href=\"https://goreportcard.com/report/github.com/strowk/mcp-autotest\"\u003e\u003cimg src=\"https://goreportcard.com/badge/github.com/strowk/mcp-autotest\" alt=\"Go Report Card\"\u003e\u003c/a\u003e\r\n\u003c/p\u003e\r\n\r\n\u003cp align=\"center\"\u003e\r\n  \u003ca href=\"#installation\"\u003eInstallation\u003c/a\u003e ⚙\r\n  \u003ca href=\"#usage\"\u003eUsage\u003c/a\u003e ⚙\r\n  \u003ca href=\"#quick-demo\"\u003eQuick Demo\u003c/a\u003e ⚙\r\n  \u003ca href=\"#dynamic-matching\"\u003eDynamic Matching\u003c/a\u003e\r\n\u003c/p\u003e\r\n\r\nA simple tool that allows to test your MCP servers using MCP protocol by defining YAML files with requests and responses.\r\n\r\nMCP cases file is a multi-document YAML, which defines every case as a separated document, like this:\r\n\r\n```yaml\r\ncase: Listing tools\r\nin: { \"jsonrpc\": \"2.0\", \"method\": \"tools/list\", \"id\": 1 }\r\nout:\r\n  {\r\n    \"jsonrpc\": \"2.0\",\r\n    \"id\": 1,\r\n    \"result\":\r\n      {\r\n        \"tools\":\r\n          [\r\n            {\r\n              \"description\": \"Run a read-only SQL query\",\r\n              \"inputSchema\":\r\n                {\r\n                  \"type\": \"object\",\r\n                  \"properties\": { \"sql\": { \"type\": \"string\" } },\r\n                },\r\n              \"name\": \"query\",\r\n            },\r\n          ],\r\n      }\r\n  }\r\n\r\n---\r\n# next case...\r\n```\r\n\r\nFiles must be named with `_test.yaml` suffix to be recognized as test cases files.\r\n\r\n## Installation\r\n\r\n## npm\r\n\r\n```bash\r\nnpm install -g mcp-autotest\r\n```\r\n\r\n## Github Releases\r\n\r\nDownload prebulit binaries from the [releases](https://github.com/strowk/mcp-autotest/releases) page and put in your PATH\r\n\r\n## Build from source\r\n\r\n```bash\r\ngo install github.com/strowk/mcp-autotest@latest\r\n```\r\n\r\n## Usage\r\n\r\n```bash\r\nmcp-autotest [flags] run path/to/tests/folder [--] command-to-run-server [server-args]\r\n```\r\n\r\nExample:\r\n```bash\r\n# start go MCP server and test via stdio transport\r\nmcp-autotest run testdata go run main.go\r\n# start Postgres MCP server and test via stdio transport\r\nmcp-autotest run -v testdata -- npx -y @modelcontextprotocol/server-postgres localhost:5432\r\n# start go MCP server and test via Streamable HTTP transport\r\nmcp-autotest run --url http://localhost:8080/mcp testdata go run main.go\r\n```\r\n\r\n### Transports\r\n\r\nmcp-autotest by default would use stdio transport, but if you want to use HTTP transport instead, you can use `--url` flag to specify the URL of the MCP server. \r\nThe URL must be in the format `http://host:port/path` or `https://host:port/path`.\r\n\r\nCurrently from Streamable HTTP only `POST` method is supported, but testing via `GET` is planned for the future.\r\n\r\n## Quick Demo\r\n\r\nIn bash shell run following\r\n\r\n```bash\r\n\r\n# create folder for test data\r\nmkdir -p testdata\r\n\r\n# create test cases file\r\ncat \u003c\u003c EOF \u003e testdata/list_tools_test.yaml\r\n# This is a test cases file. It contains a list of test cases in YAML format.\r\n# Each test case has variable number of inputs (keyst starting with 'in') and outputs (keys starting with 'out').\r\n# The test cases are separated by '---' (three dashes) on a new line, making it multi-document YAML file.\r\n# File name must end with '_test.yaml' to be recognized as a test cases file.\r\n\r\ncase: List tools\r\n\r\n# requesting list of tools\r\nin: { \"jsonrpc\": \"2.0\", \"method\": \"tools/list\", \"id\": 1 }\r\n\r\n# expect one tool in the list\r\nout:\r\n  {\r\n    \"jsonrpc\": \"2.0\",\r\n    \"id\": 1,\r\n    \"result\":\r\n      {\r\n        \"tools\":\r\n          [\r\n            {\r\n              \"description\": \"Run a read-only SQL query\",\r\n              \"inputSchema\":\r\n                {\r\n                  \"type\": \"object\",\r\n                  \"properties\": { \"sql\": { \"type\": \"string\" } },\r\n                },\r\n              \"name\": \"query\",\r\n            },\r\n          ],\r\n      }\r\n  }\r\nEOF\r\n\r\n# Now running autotest\r\nnpx mcp-autotest run testdata -- npx -y @modelcontextprotocol/server-postgres localhost:5432\r\n```\r\n\r\nThe output should simply print one word `PASS`.\r\n\r\nNow if you change something in line `\"description\": \"Run a read-only SQL query\",`, f.e to `\"description\": \"Run a read-only SQL query 2\"`, and run the last command again, you should see the output like this:\r\n\r\n```bash\r\n2025/03/31 22:23:39 actual json did not match expectation,\r\n got: '{\"id\":1,\"jsonrpc\":\"2.0\",\"result\":{\"tools\":[{\"description\":\"Run a read-only SQL query\",\"inputSchema\":{\"properties\":{\"sql\":{\"type\":\"string\"}},\"type\":\"object\"},\"name\":\"query\"}]}}'\r\n diff with expected:\r\n  \"result\": {\r\n    \"tools\": {\r\n      \"0\": {\r\n        \"description\": {\r\n        ^ value mismatch,\r\nexpected string: 'Run a read-only SQL query 2',\r\n     got string: 'Run a read-only SQL query'\r\nFAIL\r\n```\r\n\r\n## Dynamic Matching\r\n\r\nSometimes when you have dynamic values in your responses, you might want to use regular expressions to match them. You can do this by using `!!re` tag in your expected output. For example:\r\n\r\n```yaml\r\ncase: List tools\r\nin_list_tools: {\"jsonrpc\":\"2.0\",\"method\":\"tools/list\",\"id\":1}\r\nout_list_tools: {\"jsonrpc\":\"2.0\",\"result\":{\"tools\":[ { \"name\": !!re \"list-[a-z]+\" } ]},\"id\":1}\r\n```\r\n\r\nThis example is somewhat oversimplified, in reality you would probably use `!!re` with timestamps, UUIDs or other dynamic values, rather than tool names.\r\nIt does, however, demonstrate that strings with `!!re` tag would be treated as regular expressions and would be matched against actual values using regular expression matching instead of string equality.\r\n\r\nSee working examples for testing [postgres server](https://github.com/strowk/mcp-autotest/tree/main/examples/postgres).\r\n\r\n### Embedded regex\r\n\r\nEmbedded regular expression is just a regular expression that is embedded in a string. It is used to match a part of the string. For example:\r\n\r\n```yaml\r\ncase: List tools\r\nin_list_tools: {\"jsonrpc\":\"2.0\",\"method\":\"tools/list\",\"id\":1}\r\nout_list_tools: {\"jsonrpc\":\"2.0\",\"result\":{\"tools\":[ { \"name\": !!ere \"list-/[a-z]+/-tool\" } ]},\"id\":1}\r\n```\r\n\r\nEssentially `!!ere` allows you to treat most of the string as just regular string, but have a part (or parts) of it treated as regular expression.\r\nEverything inside slashes `/` would be treated as regular expression, here \"list-\" and \"-tool\" are regular strings, but `[a-z]+` is a regular expression that would match any lowercase letters non-empty string, so it would match \"list-abc-tool\", \"list-xyz-tool\" and so on.\r\n\r\nThis approach allows you not to think how to escape regular expression syntax in the rest of your string and only match bits that you need to be dynamic.\r\n\r\n### Escaping forward slashes\r\n\r\nFor embedded regex you might need to escape slashes `/` in places of your string where you want to use them, but not designate regular expression, for example: `\"url\": !! \"https:\\\\/\\\\/github.com\\\\//[a-z]+/\"` would match `\"url\": \"https://github.com/strowk\"`.\r\nIn here `\\\\/` is used to become `/` and `[a-z]+` is used to match any lowercase letters non-empty string. The reason why there are two backslashes `\\\\` is because in YAML strings backslash is an escape character, so to have a single backslash in the string you need to escape it with another backslash.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstrowk%2Fmcp-autotest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstrowk%2Fmcp-autotest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstrowk%2Fmcp-autotest/lists"}