{"id":47635750,"url":"https://github.com/temikus/butter","last_synced_at":"2026-04-25T06:03:55.231Z","repository":{"id":345392080,"uuid":"1185684935","full_name":"Temikus/butter","owner":"Temikus","description":"A blazingly fast AI proxy gateway.","archived":false,"fork":false,"pushed_at":"2026-04-23T10:03:44.000Z","size":1133,"stargazers_count":9,"open_issues_count":7,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-23T10:33:15.976Z","etag":null,"topics":["ai","gateway","gateway-services","generative-ai","guardrails","llm","llm-gateway","load-balancing","model-router","token-management"],"latest_commit_sha":null,"homepage":"https://butterproxy.org/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Temikus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-18T20:57:20.000Z","updated_at":"2026-04-23T10:03:48.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Temikus/butter","commit_stats":null,"previous_names":["temikus/butter"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/Temikus/butter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Temikus%2Fbutter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Temikus%2Fbutter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Temikus%2Fbutter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Temikus%2Fbutter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Temikus","download_url":"https://codeload.github.com/Temikus/butter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Temikus%2Fbutter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32251812,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-25T04:23:17.126Z","status":"ssl_error","status_checked_at":"2026-04-25T04:21:53.360Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","gateway","gateway-services","generative-ai","guardrails","llm","llm-gateway","load-balancing","model-router","token-management"],"created_at":"2026-04-02T00:04:12.381Z","updated_at":"2026-04-25T06:03:55.219Z","avatar_url":"https://github.com/Temikus.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/logo.jpg\" alt=\"Butter logo\" width=\"200\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/comic.jpg\" alt=\"Butter comic\" width=\"1024\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/Temikus/butter/releases\"\u003e\u003cimg src=\"https://img.shields.io/github/v/release/Temikus/butter\" alt=\"Release\"\u003e\u003c/a\u003e \u003ca href=\"https://github.com/Temikus/butter/actions/workflows/ci.yml\"\u003e\u003cimg src=\"https://github.com/Temikus/butter/actions/workflows/ci.yml/badge.svg?branch=main\" alt=\"CI\"\u003e\u003c/a\u003e \u003ca href=\"https://goreportcard.com/report/github.com/temikus/butter\"\u003e\u003cimg src=\"https://goreportcard.com/badge/github.com/temikus/butter\" alt=\"Go Report Card\"\u003e\u003c/a\u003e \u003ca href=\"https://github.com/Temikus/butter/blob/main/go.mod\"\u003e\u003cimg src=\"https://img.shields.io/github/go-mod/go-version/Temikus/butter\" alt=\"Go Version\"\u003e\u003c/a\u003e \u003ca href=\"https://github.com/Temikus/butter/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/github/license/Temikus/butter\" alt=\"License\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nA blazingly fast AI proxy gateway written in Go. Butter sits between your application and AI providers, offering a unified OpenAI-compatible API with minimal latency overhead.\n\nInspired by [Bifrost](https://github.com/maximhq/bifrost), but with a focus on simplicity, extensibility via WASM plugins, and raw performance.\n\n```\nYour App ──▶ Butter ──▶ OpenAI / Anthropic / Bedrock / Gemini / Groq / Mistral / ...\n                │\n                ├── Unified OpenAI-compatible API\n                ├── Automatic failover \u0026 retries\n                ├── Weighted key rotation\n                └── Plugin hooks (Go + WASM)\n```\n\n## Features\n\n- OpenAI-compatible API — `/v1/chat/completions` (streaming \u0026 non-streaming), `/v1/embeddings`, `/v1/models`\n- Anthropic Messages API — `POST /v1/messages` (streaming \u0026 non-streaming), with failover across Anthropic-native providers\n- **10 providers**: OpenAI, Anthropic, AWS Bedrock, Gemini, OpenRouter, Groq, Mistral, Together.ai, Fireworks, Perplexity — shared `openaicompat` base for OpenAI-compatible APIs; `AnthropicNativeHandler` interface for Anthropic-format providers (Anthropic, Bedrock)\n- Anthropic \u0026 Gemini format translation (OpenAI requests automatically converted to/from native formats)\n- Multi-provider routing with model-specific provider lists and priority/round-robin strategies\n- Weighted random key selection with per-key model allowlists\n- Multi-provider failover with configurable retry-on status codes and exponential backoff\n- **WASM plugin sandbox** via [Extism](https://extism.org/)/wazero — load external `.wasm` plugins with zero CGo, full sandbox isolation\n- **Prompt injection guard** WASM plugin — scans requests for ~60 injection patterns across 7 categories (instruction override, jailbreak, prompt extraction, etc.) with Unicode bypass detection; block, log, or tag modes\n- Plugin system with ordered hook chains (`pre_http`, `post_http`, `pre_llm`, `post_llm`, stream chunks, observability traces)\n- Plugin short-circuit support (plugins can reject or rewrite requests before they reach the provider)\n- Built-in rate limiter plugin (token bucket, global or per-IP, configurable RPM)\n- Built-in request logging plugin (structured slog, provider/model/status/duration)\n- Built-in Prometheus metrics plugin (OTel SDK instruments, `/metrics` endpoint)\n- Built-in distributed tracing plugin (OTel SDK, OTLP HTTP export)\n- Response caching (in-memory LRU or Redis backend; SHA256 cache key; temperature=0 non-streaming only)\n- Config hot-reload (mtime polling, atomic engine swap — no restart required)\n- **Application keys** for usage tracking and attribution — vend `btr_` tokens, track per-key request/token counts, optional enforcement via `require_key`\n- Per-provider `credential_mode` — `stored` (default, inject managed keys) or `passthrough` (forward client's own auth headers unchanged)\n- Raw HTTP passthrough for provider-native endpoints (`/native/{provider}/*`)\n- Health check endpoint (`/healthz`)\n- Graceful shutdown (SIGINT/SIGTERM)\n- Multi-stage Docker image (distroless base)\n\n**Coming soon:**\n- More providers (Azure OpenAI, Vertex AI)\n\n## Quick Start\n\n### Prerequisites\n\n- Go 1.25+ (uses enhanced `ServeMux` pattern routing)\n- An API key for a supported provider ([OpenAI](https://platform.openai.com/), [Anthropic](https://console.anthropic.com/), [AWS Bedrock](https://aws.amazon.com/bedrock/), [Google Gemini](https://ai.google.dev/), [OpenRouter](https://openrouter.ai/), [Groq](https://console.groq.com/), [Mistral](https://console.mistral.ai/), [Together.ai](https://api.together.xyz/), [Fireworks](https://fireworks.ai/), [Perplexity](https://www.perplexity.ai/), or any OpenAI-compatible API)\n\n### 1. Install\n\nDownload the latest binary from [GitHub Releases](https://github.com/temikus/butter/releases), or build from source:\n\n```bash\ngit clone https://github.com/temikus/butter.git\ncd butter\ngo build -o pkg/bin/butter ./cmd/butter/\n```\n\n### 2. Configure\n\n```bash\ncp config.example.yaml config.yaml\n```\n\nEdit `config.yaml` or set environment variables:\n\n```bash\nexport OPENAI_API_KEY=\"sk-...\"\nexport ANTHROPIC_API_KEY=\"sk-ant-...\"\nexport OPENROUTER_API_KEY=\"sk-or-v1-...\"\n```\n\nThe config file supports `${ENV_VAR}` substitution, so the default `config.example.yaml` works out of the box once the environment variables are set.\n\n\u003cdetails\u003e\n\u003csummary\u003eExample config.yaml\u003c/summary\u003e\n\n```yaml\nserver:\n  address: \":8080\"\n  read_timeout: 30s\n  write_timeout: 120s\n\nproviders:\n  openai:\n    base_url: https://api.openai.com/v1\n    keys:\n      - key: \"${OPENAI_API_KEY}\"\n        weight: 1\n\n  anthropic:\n    base_url: https://api.anthropic.com/v1\n    keys:\n      - key: \"${ANTHROPIC_API_KEY}\"\n        weight: 1\n\n  openrouter:\n    base_url: https://openrouter.ai/api/v1\n    keys:\n      - key: \"${OPENROUTER_API_KEY}\"\n        weight: 1\n\nrouting:\n  default_provider: openrouter\n  models:\n    \"gpt-4o\":\n      providers: [openai, openrouter]\n      strategy: priority\n    \"claude-sonnet-4-20250514\":\n      providers: [anthropic, openrouter]\n      strategy: priority\n  failover:\n    enabled: true\n    max_retries: 3\n    retry_on: [429, 500, 502, 503, 504]\n    backoff:\n      initial: 100ms\n      multiplier: 2.0\n      max: 5s\n\nplugins:\n  ratelimit:\n    requests_per_minute: 60\n    per_ip: false\n  requestlog:\n    level: info\n  metrics: {}\n```\n\n\u003c/details\u003e\n\n### 3. Run\n\n```bash\n./pkg/bin/butter -config config.yaml\n```\n\nYou should see:\n\n```json\n{\"level\":\"INFO\",\"msg\":\"butter listening\",\"address\":\":8080\"}\n```\n\n### 4. Send a request\n\n**Non-streaming:**\n\n```bash\ncurl http://localhost:8080/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"openai/gpt-4o-mini\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Say hello in three languages\"}]\n  }'\n```\n\n**Streaming:**\n\n```bash\ncurl http://localhost:8080/v1/chat/completions \\\n  --no-buffer \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"openai/gpt-4o-mini\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Count to 5\"}],\n    \"stream\": true\n  }'\n```\n\n**Health check:**\n\n```bash\ncurl http://localhost:8080/healthz\n# ok\n```\n\n### Drop-in replacement\n\nButter is compatible with any OpenAI SDK client. Just point the base URL at your Butter instance:\n\n**Python (openai SDK):**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http://localhost:8080/v1\",\n    api_key=\"unused\",  # Butter uses its own configured keys\n)\n\nresponse = client.chat.completions.create(\n    model=\"openai/gpt-4o-mini\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n)\nprint(response.choices[0].message.content)\n```\n\n**Node.js (openai SDK):**\n\n```javascript\nimport OpenAI from \"openai\";\n\nconst client = new OpenAI({\n  baseURL: \"http://localhost:8080/v1\",\n  apiKey: \"unused\",\n});\n\nconst completion = await client.chat.completions.create({\n  model: \"openai/gpt-4o-mini\",\n  messages: [{ role: \"user\", content: \"Hello!\" }],\n});\nconsole.log(completion.choices[0].message.content);\n```\n\n## Development\n\nA [`justfile`](https://github.com/casey/just) is provided for common tasks:\n\n```bash\njust build              # Build binary (with commit hash)\njust build-release      # Build with full version info from git\njust serve              # Run with config (auto-loads API keys from ~/.openai/api-key etc.)\njust test               # Run all tests with race detector (unit + integration)\njust test integration   # Run integration tests only (mock servers, no real API calls)\njust test one ./internal/proxy/ TestDispatch  # Run a single test\njust lint               # Run golangci-lint\njust check              # Run vet + lint + all tests\njust bench              # Run benchmarks with allocation reporting\njust release-snapshot   # Test GoReleaser locally (no publish)\njust docker-build       # Build multi-stage Docker image\njust docker-run         # Run Docker container with config\njust build-example-wasm # Compile example WASM plugin (requires TinyGo)\njust build-injection-guard # Compile prompt injection guard plugin (requires TinyGo)\njust build-wasm         # Build all WASM plugins\njust clean              # Remove built binaries and compiled WASM plugins\n```\n\nOr use Go directly:\n\n```bash\ngo run ./cmd/butter/ -config config.yaml\ngo test ./... -v -race -count=1\ngo test ./... -bench=. -benchmem\n```\n\n### Project structure\n\n```\nbutter/\n├── cmd/butter/                  Main binary\n├── internal/\n│   ├── config/                  YAML config with env var substitution + hot-reload watcher\n│   ├── transport/               HTTP server and handlers\n│   ├── proxy/                   Core dispatch engine (routing, failover, key selection)\n│   ├── appkey/                  Application key store (usage tracking, token counting)\n│   ├── cache/                   Cache interface + in-memory LRU and Redis backends\n│   ├── plugin/                  Plugin system (interfaces, chain, manager)\n│   │   ├── wasm/                WASM plugin host (Extism/wazero)\n│   │   └── builtin/\n│   │       ├── ratelimit/       Token bucket rate limiter plugin\n│   │       ├── requestlog/      Request logging plugin\n│   │       ├── metrics/         Prometheus metrics plugin (OTel SDK)\n│   │       └── tracing/         Distributed tracing plugin (OTel, OTLP HTTP)\n│   └── provider/\n│       ├── provider.go          Provider interface \u0026 types\n│       ├── registry.go          Thread-safe provider registry\n│       ├── openaicompat/        Reusable base for OpenAI-compatible APIs\n│       ├── openai/              OpenAI provider\n│       ├── anthropic/           Anthropic provider (format translation)\n│       ├── bedrock/             AWS Bedrock provider (SigV4, AnthropicNativeHandler)\n│       ├── gemini/              Google Gemini provider (format translation)\n│       ├── openrouter/          OpenRouter provider\n│       ├── groq/                Groq provider\n│       ├── mistral/             Mistral provider\n│       ├── together/            Together.ai provider\n│       ├── fireworks/           Fireworks provider\n│       └── perplexity/          Perplexity provider\n├── plugin/sdk/                  Public JSON ABI types for WASM plugin authors\n├── plugins/\n│   ├── example-wasm/            Example WASM plugin (TinyGo, pre_http hook)\n│   └── prompt-injection-guard/  Prompt injection detection WASM plugin\n├── tests/integration/           Integration tests with mock provider servers\n├── config.example.yaml\n├── Dockerfile                   Multi-stage distroless image\n├── justfile\n└── go.mod\n```\n\n## Performance Targets\n\n| Metric | Target |\n|--------|--------|\n| Per-request overhead (no plugins) | \u003c50us |\n| Per-request overhead (built-in plugins) | \u003c100us |\n| Per-request overhead (1 WASM plugin) | \u003c150us |\n| Streaming TTFB overhead | \u003c1ms |\n| Memory at idle | \u003c30MB |\n\n## License\n\nApache 2.0 License. See [LICENSE](LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftemikus%2Fbutter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftemikus%2Fbutter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftemikus%2Fbutter/lists"}