{"id":41949661,"url":"https://github.com/dotsetlabs/bellwether","last_synced_at":"2026-02-16T23:25:43.552Z","repository":{"id":332266653,"uuid":"1133233397","full_name":"dotsetlabs/bellwether","owner":"dotsetlabs","description":"Open-source testing tool for Model Context Protocol (MCP) servers. Detect breaking schema changes before they reach production. Free deterministic validation with optional LLM-powered behavioral exploration. CI/CD ready with GitHub Actions support.","archived":false,"fork":false,"pushed_at":"2026-02-07T23:36:30.000Z","size":3081,"stargazers_count":1,"open_issues_count":4,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-08T01:09:05.327Z","etag":null,"topics":["ai-agents","anthropic","api-testing","automation","ci-cd","claude","cli","contract-testing","developer-tools","devops","documentation-generator","drift-detection","github-actions","llm","mcp","model-context-protocol","openai","schema-validation","testing","typescript"],"latest_commit_sha":null,"homepage":"https://docs.bellwether.sh","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dotsetlabs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-13T04:14:46.000Z","updated_at":"2026-02-07T23:35:09.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dotsetlabs/bellwether","commit_stats":null,"previous_names":["dotsetlabs/bellwether"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/dotsetlabs/bellwether","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dotsetlabs%2Fbellwether","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dotsetlabs%2Fbellwether/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dotsetlabs%2Fbellwether/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dotsetlabs%2Fbellwether/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dotsetlabs","download_url":"https://codeload.github.com/dotsetlabs/bellwether/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dotsetlabs%2Fbellwether/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29412675,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-13T06:24:03.484Z","status":"ssl_error","status_checked_at":"2026-02-13T06:23:12.830Z","response_time":78,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","anthropic","api-testing","automation","ci-cd","claude","cli","contract-testing","developer-tools","devops","documentation-generator","drift-detection","github-actions","llm","mcp","model-context-protocol","openai","schema-validation","testing","typescript"],"created_at":"2026-01-25T20:17:17.404Z","updated_at":"2026-02-16T23:25:43.546Z","avatar_url":"https://github.com/dotsetlabs.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bellwether\n\n[![Build Status](https://github.com/dotsetlabs/bellwether/actions/workflows/ci.yml/badge.svg)](https://github.com/dotsetlabs/bellwether/actions)\n[![npm version](https://img.shields.io/npm/v/@dotsetlabs/bellwether)](https://www.npmjs.com/package/@dotsetlabs/bellwether)\n[![Documentation](https://img.shields.io/badge/docs-docs.bellwether.sh-blue)](https://docs.bellwether.sh)\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](./LICENSE)\n[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](./CONTRIBUTING.md)\n\n\u003e **The open-source MCP testing tool. Catch drift before your users do.**\n\n**What is MCP?** [MCP (Model Context Protocol)](https://modelcontextprotocol.io/) is how AI assistants like Claude connect to external tools—read files, query databases, call APIs. When those tool schemas change, AI workflows break silently.\n\n## Why Bellwether?\n\nMCP servers expose tools with JSON schemas. When those schemas change—a parameter renamed, a type modified, a tool removed—AI agents break silently. Bellwether catches these changes before they reach production.\n\n| Problem | Solution |\n|:--------|:---------|\n| Breaking changes slip into production | **Drift detection** catches schema changes in CI |\n| No standard for MCP testing | **Native MCP support** understands tools, prompts, resources |\n| Manual testing misses edge cases | **Automated exploration** covers what humans miss |\n| Documentation gets stale | **CONTRACT.md** generated from actual behavior |\n\n## Quick Start\n\n```bash\nnpm install -g @dotsetlabs/bellwether\nbellwether init npx @mcp/your-server\nbellwether check\n```\n\nThat's it. No API keys. No LLM costs. Runs in seconds.\n\n## Product Focus\n\nBellwether is intentionally opinionated:\n\n- **Core workflow (default)**: `init` -\u003e `check` -\u003e `baseline`\n- **Advanced workflow (opt-in)**: `explore`, `watch`, `discover`, `golden`, `contract`, `registry`\n\nIf you only need CI-safe drift detection, you can stay entirely in the core workflow.\n\n## Two Modes\n\n| Mode | Purpose | Cost | When to Use |\n|:-----|:--------|:-----|:------------|\n| `check` | Schema drift detection | **Free** | CI/CD, every PR |\n| `explore` | LLM-powered behavioral testing | LLM API costs | Local dev, deep analysis |\n\n**Most users only need `check`.** It's deterministic, fast, and catches the changes that break AI agents.\n\n## CI/CD Workflow\n\nStore your baseline in git. Run checks in CI. No account needed.\n\n```bash\n# 1. Initialize and save baseline (one-time setup)\nbellwether init npx @mcp/your-server\nbellwether check\nbellwether baseline save\ngit add bellwether.yaml bellwether-baseline.json\ngit commit -m \"Add Bellwether baseline\"\n```\n\n```yaml\n# 2. Add to CI (.github/workflows/bellwether.yml)\nname: MCP Drift Detection\non: [pull_request]\n\njobs:\n  check:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - run: npx @dotsetlabs/bellwether check --fail-on-drift\n```\n\n## What It Detects\n\n| Change | Example | Severity |\n|:-------|:--------|:---------|\n| Tool added/removed | `delete_file` appears or disappears | Breaking |\n| Schema changed | Parameter `path` becomes required | Breaking |\n| Parameter renamed | `path` to `file_path` | Breaking |\n| Description changed | Tool help text updated | Warning |\n| Performance regression | Latency increased \u003e10% | Warning |\n| Tool annotations changed | `readOnlyHint` flipped to `false` | Warning |\n| Output schema changed | Return type structure modified | Warning |\n| Entity title changed | Tool/prompt/resource title updated | Info |\n| Task support changed | Execution mode switched to `async` | Warning |\n| Server instructions changed | Server-level instructions updated | Info |\n| Prompt added/removed | Prompt template appears or disappears | Breaking |\n| Resource changed | Resource URI or MIME type modified | Warning |\n\nComparisons are **protocol-version-aware** — version-specific fields (annotations, titles, output schemas, etc.) are only compared when both baselines support the relevant MCP protocol version.\n\n## Command Tiers\n\n### Core Commands (Recommended)\n\n| Command | Purpose |\n|:--------|:--------|\n| `init` | Create `bellwether.yaml` |\n| `check` | Deterministic schema drift detection |\n| `baseline save` | Save snapshot for future comparisons |\n| `baseline compare` | Compare latest check output to saved baseline |\n\n### Advanced Commands (Optional)\n\n| Command | Purpose |\n|:--------|:--------|\n| `explore` | LLM behavioral testing and `AGENTS.md` generation |\n| `watch` | Continuous checking on file changes |\n| `discover` | Capability inspection without tests |\n| `registry` | Search MCP Registry |\n| `golden` | Golden output regression testing |\n| `contract` | Contract validation and generation |\n| `auth` | Manage LLM provider API keys |\n| `validate-config` | Validate `bellwether.yaml` without running tests |\n\n## CI/CD Exit Codes\n\n| Code | Meaning | Suggested Action |\n|:-----|:--------|:-----------------|\n| `0` | No changes | Pass |\n| `1` | Info-level changes | Pass or warn |\n| `2` | Warning-level changes | Warn |\n| `3` | Breaking changes | Fail |\n| `4` | Runtime error | Fail |\n| `5` | Low confidence metrics | Warn or fail |\n\n## GitHub Action\n\n```yaml\n- uses: dotsetlabs/bellwether@v2.1.2\n  with:\n    version: '2.1.2'\n    server-command: 'npx @mcp/your-server'\n    baseline-path: './bellwether-baseline.json'\n    fail-on-severity: 'warning'\n```\n\n## Configuration\n\nAll settings live in `bellwether.yaml`. Create one with presets:\n\n```bash\nbellwether init npx @mcp/your-server           # Default (free, fast)\nbellwether init --preset ci npx @mcp/server    # Optimized for CI/CD\nbellwether init --preset local npx @mcp/server # Local Ollama (free)\n```\n\nFor remote MCP servers that require auth headers, configure:\n\n```yaml\nserver:\n  transport: sse\n  url: \"https://api.example.com/mcp\"\n  headers:\n    Authorization: \"Bearer ${MCP_SERVER_TOKEN}\"\n```\n\nOr use one-off CLI overrides:\n\n```bash\nbellwether check -H \"Authorization: Bearer $MCP_SERVER_TOKEN\"\n```\n\n## Environment Variables\n\n| Variable | Description |\n|:---------|:------------|\n| `OPENAI_API_KEY` | OpenAI API key (explore only) |\n| `ANTHROPIC_API_KEY` | Anthropic API key (explore only) |\n| `OLLAMA_BASE_URL` | Ollama URL (default: `http://localhost:11434`) |\n\n## Documentation\n\n**[docs.bellwether.sh](https://docs.bellwether.sh)** — Full reference for configuration and commands.\n\n- [Quick Start](https://docs.bellwether.sh/quickstart)\n- [Core vs Advanced](https://docs.bellwether.sh/concepts/core-vs-advanced)\n- [CLI Reference](https://docs.bellwether.sh/cli/init)\n- [CI/CD Integration](https://docs.bellwether.sh/guides/ci-cd)\n- [Golden Paths](https://docs.bellwether.sh/guides/golden-paths)\n- [Compatibility Policy](https://docs.bellwether.sh/concepts/compatibility-policy)\n\n## Project Governance\n\n- [Changelog](./CHANGELOG.md)\n- [Releases](https://github.com/dotsetlabs/bellwether/releases)\n\n## Community\n\n- [GitHub Discussions](https://github.com/dotsetlabs/bellwether/discussions) - Questions and ideas\n- [GitHub Issues](https://github.com/dotsetlabs/bellwether/issues) - Bug reports\n- [Contributing](./CONTRIBUTING.md) - How to contribute\n\n## Development\n\n```bash\ngit clone https://github.com/dotsetlabs/bellwether\ncd bellwether\nnpm install\nnpm run build\nnpm test\n```\n\n## License\n\nMIT License - see [LICENSE](./LICENSE) for details.\n\n---\n\n\u003cp align=\"center\"\u003e\n  Built by \u003ca href=\"https://dotsetlabs.com\"\u003eDotset Labs\u003c/a\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdotsetlabs%2Fbellwether","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdotsetlabs%2Fbellwether","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdotsetlabs%2Fbellwether/lists"}