{"id":50411533,"url":"https://github.com/bluefunda/llmrouter","last_synced_at":"2026-05-31T04:00:48.722Z","repository":{"id":356397531,"uuid":"1139408183","full_name":"bluefunda/llmrouter","owner":"bluefunda","description":"LLM Router","archived":false,"fork":false,"pushed_at":"2026-05-30T06:15:55.000Z","size":200,"stargazers_count":2,"open_issues_count":9,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-30T08:09:38.367Z","etag":null,"topics":["active"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bluefunda.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-01-21T23:20:41.000Z","updated_at":"2026-05-30T06:15:56.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/bluefunda/llmrouter","commit_stats":null,"previous_names":["bluefunda/llm-router"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/bluefunda/llmrouter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluefunda%2Fllmrouter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluefunda%2Fllmrouter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluefunda%2Fllmrouter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluefunda%2Fllmrouter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bluefunda","download_url":"https://codeload.github.com/bluefunda/llmrouter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluefunda%2Fllmrouter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33718446,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-31T02:00:06.040Z","response_time":95,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["active"],"created_at":"2026-05-31T04:00:47.864Z","updated_at":"2026-05-31T04:00:48.716Z","avatar_url":"https://github.com/bluefunda.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# llmrouter\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/bluefunda/llmrouter.svg)](https://pkg.go.dev/github.com/bluefunda/llmrouter)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)\n[![Go Report Card](https://goreportcard.com/badge/github.com/bluefunda/llmrouter)](https://goreportcard.com/report/github.com/bluefunda/llmrouter)\n\nA Go library that provides a unified interface for routing requests across multiple LLM providers. Write against one API, deploy across OpenAI, Anthropic, Google Gemini, and any OpenAI-compatible service.\n\n## Prerequisites\n\n- Go 1.25+\n- API key for at least one supported provider\n\n## Installation\n\n```bash\ngo get github.com/bluefunda/llmrouter\n```\n\n## Quick Start\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"fmt\"\n    \"time\"\n\n    llmrouter \"github.com/bluefunda/llmrouter\"\n    \"github.com/bluefunda/llmrouter/middleware\"\n    \"github.com/bluefunda/llmrouter/providers/anthropic\"\n    \"github.com/bluefunda/llmrouter/providers/openai\"\n)\n\nfunc main() {\n    router := llmrouter.New(\n        llmrouter.WithProvider(\"openai\", openai.NewFromEnv(\"openai\", \"OPENAI_API_KEY\")),\n        llmrouter.WithProvider(\"anthropic\", anthropic.NewFromEnv()),\n        llmrouter.WithMiddleware(\n            middleware.Retry(3, time.Second),\n            middleware.Timeout(60*time.Second),\n        ),\n    )\n\n    resp, err := router.Complete(context.Background(), \u0026llmrouter.Request{\n        Model: \"gpt-4o-mini\",\n        Messages: []llmrouter.Message{\n            {Role: llmrouter.RoleUser, Content: \"Hello!\"},\n        },\n    })\n    if err != nil {\n        panic(err)\n    }\n\n    fmt.Println(resp.Choices[0].Message.Content)\n}\n```\n\n## Providers\n\nEach provider is configured via environment variables or explicit options.\n\n| Provider   | Package                                  | Env Variable         | Models                                          |\n|------------|------------------------------------------|----------------------|-------------------------------------------------|\n| OpenAI     | `providers/openai`                       | `OPENAI_API_KEY`     | gpt-4o, gpt-4o-mini, gpt-4.1, o4-mini          |\n| Anthropic  | `providers/anthropic`                    | `ANTHROPIC_API_KEY`  | claude-opus-4, claude-sonnet-4, claude-haiku-3.5 |\n| Gemini     | `providers/gemini`                       | `GEMINI_API_KEY`     | gemini-1.5-pro, gemini-1.5-flash, gemini-2.0-flash-exp |\n| DeepSeek   | `providers/openai` (preset: `deepseek`)  | `DEEPSEEK_API_KEY`   | deepseek-chat, deepseek-coder                   |\n| Groq       | `providers/openai` (preset: `groq`)      | `GROQ_API_KEY`       | llama-3.3-70b-versatile, mixtral-8x7b           |\n| Together   | `providers/openai` (preset: `together`)  | `TOGETHER_API_KEY`   | llama-3.3-70b, mixtral-8x7b                     |\n| Ollama     | `providers/openai` (preset: `ollama`)    | —                    | Any locally hosted model                        |\n| Sarvam     | `providers/openai` (preset: `sarvam`)    | `SARVAM_API_KEY`     | sarvam-m, sarvam-30b, sarvam-105b               |\n\n### OpenAI-compatible providers\n\nDeepSeek, Groq, Together AI, Ollama, and Sarvam all use the OpenAI provider with a preset name:\n\n```go\nopenai.NewFromEnv(\"deepseek\", \"DEEPSEEK_API_KEY\")\nopenai.NewFromEnv(\"groq\", \"GROQ_API_KEY\")\nopenai.NewFromEnv(\"ollama\", \"\")  // no key needed for local\nopenai.NewFromEnv(\"sarvam\", \"SARVAM_API_KEY\")\n```\n\n### Gemini\n\nGemini requires explicit error handling at construction time and holds a gRPC connection — call `router.Close()` (or `provider.Close()` directly) on shutdown:\n\n```go\ngeminiProvider, err := gemini.NewFromEnv()\nif err != nil {\n    log.Fatal(err)\n}\ndefer router.Close()\n```\n\n## Configuration\n\n### Router options\n\n```go\nrouter := llmrouter.New(\n    llmrouter.WithProvider(\"openai\", openaiProvider),\n    llmrouter.WithProvider(\"anthropic\", anthropicProvider),\n    llmrouter.WithModelMapping(\"gpt-4o\", \"openai\"),\n    llmrouter.WithModelMapping(\"claude-sonnet-4-20250514\", \"anthropic\"),\n    llmrouter.WithFallback(\"openai\", \"anthropic\"),\n    llmrouter.WithMiddleware(retryMw, cb.Wrap, timeoutMw),\n)\n```\n\n| Option             | Description                                           |\n|--------------------|-------------------------------------------------------|\n| `WithProvider`     | Register a named provider                             |\n| `WithModelMapping` | Route a model name to a specific provider             |\n| `WithFallback`     | Set fallback provider order on primary failure        |\n| `WithMiddleware`   | Attach middleware to the processing chain             |\n\n### Model resolution\n\nThe router resolves a model to a provider in this order:\n\n1. **Explicit mapping** — `WithModelMapping(\"gpt-4o\", \"openai\")`\n2. **Provider name match** — model name equals a registered provider name\n3. **Provider model list** — iterates providers in registration order and checks `Models()`\n\n## Middleware\n\nMiddleware is a `MiddlewareFunc` — a plain `func(Provider) Provider`. It is applied in declaration order (first declared = outermost wrapper).\n\n### Retry\n\nExponential backoff with configurable max attempts. Non-retryable errors (auth failures, invalid requests, context cancellation) short-circuit immediately.\n\n```go\nmiddleware.Retry(3, time.Second)\nmiddleware.Retry(3, time.Second, middleware.WithMaxDelay(10*time.Second))\nmiddleware.Retry(3, time.Second, middleware.WithRetryFunc(myRetryPolicy))\n```\n\n### Circuit Breaker\n\nStdlib-only three-state circuit breaker (Closed → Open → HalfOpen). Opens after consecutive failures exceed the threshold; recovers after the timeout period. No external dependencies.\n\nBecause the circuit breaker has observable state, it is constructed separately and passed via `cb.Wrap`:\n\n```go\ncb := middleware.NewCircuitBreaker(5, 30*time.Second)\nrouter := llmrouter.New(\n    llmrouter.WithMiddleware(cb.Wrap),\n)\nfmt.Println(cb.State()) // CBStateClosed / CBStateOpen / CBStateHalfOpen\n```\n\n### Timeout\n\nEnforces a deadline on both `Complete` and `Stream` calls. On timeout, `Stream` surfaces the error through `StreamResult.Err()`.\n\n```go\nmiddleware.Timeout(60 * time.Second)\n```\n\n### Custom middleware\n\nAny `func(llmrouter.Provider) llmrouter.Provider` satisfies `MiddlewareFunc` directly:\n\n```go\nfunc Logging(next llmrouter.Provider) llmrouter.Provider {\n    return \u0026loggingProvider{Provider: next}\n}\n\nrouter := llmrouter.New(llmrouter.WithMiddleware(Logging))\n```\n\n## Streaming\n\n`Stream` returns a `*StreamResult` iterator. Advance it with `Next()`, read the current event with `Event()`, and check errors after the loop with `Err()`. Always `defer stream.Close()` to release resources.\n\n```go\nstream, err := router.Stream(ctx, \u0026llmrouter.Request{\n    Model:    \"claude-sonnet-4-20250514\",\n    Messages: []llmrouter.Message{\n        {Role: llmrouter.RoleUser, Content: \"Write a haiku about Go.\"},\n    },\n})\nif err != nil {\n    log.Fatal(err)\n}\ndefer stream.Close()\n\nfor stream.Next() {\n    event := stream.Event()\n    switch event.Type {\n    case llmrouter.EventContentDelta:\n        fmt.Print(event.Content)\n    case llmrouter.EventToolCallDelta:\n        // handle tool call delta\n    case llmrouter.EventDone:\n        // event.Response holds the final response with usage stats\n    }\n}\nif err := stream.Err(); err != nil {\n    log.Fatal(err)\n}\n```\n\n## Tool Calling\n\nDefine tools once and use them across any provider that supports function calling:\n\n```go\ntool := llmrouter.Tool{\n    Type: \"function\",\n    Function: llmrouter.Function{\n        Name:        \"get_weather\",\n        Description: \"Get current weather for a location\",\n        Parameters:  json.RawMessage(`{\n            \"type\": \"object\",\n            \"properties\": {\n                \"location\": {\"type\": \"string\"}\n            },\n            \"required\": [\"location\"]\n        }`),\n    },\n}\n\nresp, _ := router.Complete(ctx, \u0026llmrouter.Request{\n    Model:    \"gpt-4o-mini\",\n    Messages: messages,\n    Tools:    []llmrouter.Tool{tool},\n})\n```\n\n## Multimodal\n\nMessages support text, images, and documents via `ContentParts`:\n\n```go\nmsg := llmrouter.Message{\n    Role: llmrouter.RoleUser,\n    ContentParts: []llmrouter.ContentPart{\n        {Type: \"text\", Text: \"What's in this image?\"},\n        {Type: \"image_url\", ImageURL: \u0026llmrouter.ImageURL{URL: \"https://...\"}},\n    },\n}\n```\n\n## Prompt Caching\n\nMark static content for provider-level caching. Anthropic uses explicit `CacheControl` annotations; OpenAI and Gemini cache automatically. Observe savings via `Usage.CachedPromptTokens`:\n\n```go\nreq := \u0026llmrouter.Request{\n    Model: \"claude-sonnet-4-20250514\",\n    Messages: []llmrouter.Message{\n        {\n            Role:         llmrouter.RoleSystem,\n            Content:      longSystemPrompt,\n            CacheControl: \u0026llmrouter.CacheControl{Type: \"ephemeral\"},\n        },\n        {Role: llmrouter.RoleUser, Content: userQuery},\n    },\n}\n```\n\n## Error Handling\n\nThe library classifies errors for intelligent retry and routing decisions:\n\n| Error                   | Retryable | Description                     |\n|-------------------------|-----------|---------------------------------|\n| `ErrRateLimited`        | Yes       | Provider rate limit (429)       |\n| `ErrAuthFailed`         | No        | Invalid API key (401/403)       |\n| `ErrInvalidRequest`     | No        | Malformed request (400)         |\n| `ErrCircuitOpen`        | No        | Circuit breaker is open         |\n| `ErrMaxRetriesExceeded` | No        | All retry attempts exhausted    |\n| `ErrUnknownModel`       | No        | Model not found in any provider |\n| `ErrNoProviders`        | No        | No providers registered         |\n\nUse `llmrouter.IsRetryable(err)` and `llmrouter.IsRateLimited(err)` for programmatic checks.\n\n## Project Structure\n\n```\nrouter.go                      # Core router — provider registry, model resolution, middleware chain\nprovider.go                    # Provider interface and MiddlewareFunc type\ntypes.go                       # Unified request/response types, streaming events, tool definitions\noptions.go                     # Functional options for router configuration\nerrors.go                      # Error types and retryability classification\nmiddleware/\n  retry.go                     # Retry with exponential backoff\n  timeout.go                   # Request timeout enforcement\n  breaker.go                   # Circuit breaker state machine (stdlib only)\n  circuitbreaker.go            # Circuit breaker middleware wrapper\nproviders/\n  openai/                      # OpenAI + compatible providers (DeepSeek, Groq, Together, Ollama, Sarvam)\n  anthropic/                   # Anthropic Claude\n  gemini/                      # Google Gemini\nexamples/\n  simple/                      # Basic completion\n  streaming/                   # Streaming responses\n  tools/                       # Function calling\n  fallback/                    # Multi-provider with middleware\n```\n\n## License\n\nApache 2.0 — see [LICENSE](LICENSE).\n\nBuilt by BlueFunda — open-sourced under Apache 2.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbluefunda%2Fllmrouter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbluefunda%2Fllmrouter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbluefunda%2Fllmrouter/lists"}