{"id":35881721,"url":"https://github.com/ukrocks007/ai-gateway-kit","last_synced_at":"2026-01-08T18:04:07.277Z","repository":{"id":331363466,"uuid":"1126341482","full_name":"ukrocks007/ai-gateway-kit","owner":"ukrocks007","description":"Provider-agnostic AI gateway with capability-based routing, in-memory rate limiting, and observability hooks.","archived":false,"fork":false,"pushed_at":"2026-01-02T10:37:05.000Z","size":46,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-07T00:36:22.027Z","etag":null,"topics":["ai","capability-routing","fallback","gateway","gemini","github-models","llm","observability","openai","rate-limiting"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ukrocks007.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-01T17:44:56.000Z","updated_at":"2026-01-03T06:07:19.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ukrocks007/ai-gateway-kit","commit_stats":null,"previous_names":["ukrocks007/ai-gateway-kit"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ukrocks007/ai-gateway-kit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ukrocks007%2Fai-gateway-kit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ukrocks007%2Fai-gateway-kit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ukrocks007%2Fai-gateway-kit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ukrocks007%2Fai-gateway-kit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ukrocks007","download_url":"https://codeload.github.com/ukrocks007/ai-gateway-kit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ukrocks007%2Fai-gateway-kit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28247286,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2026-01-08T02:00:06.591Z","response_time":241,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","capability-routing","fallback","gateway","gemini","github-models","llm","observability","openai","rate-limiting"],"created_at":"2026-01-08T18:01:33.463Z","updated_at":"2026-01-08T18:04:07.268Z","avatar_url":"https://github.com/ukrocks007.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ai-gateway-kit\n\nA boring, provider-agnostic AI Gateway for Node.js.\n\nThis library exists to solve the “production gateway” problems around LLM usage:\n\n- **Capability-based routing** (agents request *capabilities*, not models)\n- **Ordered fallback** (graceful degradation, never silent failure)\n- **In-memory rate limiting** (instance-scoped by design)\n- **Observability hooks** (you choose logging/metrics/tracing)\n\n## Why capability-based routing?\n\nModel names change, providers change, and quotas fluctuate.\nA gateway that routes by *capability* lets your agents stay stable while the model fleet evolves.\n\nExample capabilities:\n- `fast_text`\n- `deep_reasoning`\n- `search`\n- `speech_to_text`\n\n## Why in-memory state?\n\nThis kit intentionally uses **in-memory** rate limit state.\n\n- Works in serverless environments (Vercel-compatible)\n- No shared storage dependency\n- Predictable failure modes\n\nTrade-off: **multi-instance deployments do not share quotas**. Each instance enforces limits based on its own in-memory view.\n\nIf you need cross-instance coordination, you can replace the in-memory `RateLimitManager` with your own implementation.\n\n## This is not a chat wrapper\n\nThis library is infrastructure:\n- routing\n- backoff\n- fallbacks\n- hooks\n\nIt does **not** provide prompt templates, product policies, UI, or agent logic.\n\n## Install\n\n```bash\nnpm install ai-gateway-kit\n```\n\n## Quick start\n\n```ts\nimport { createAIGateway, createGitHubModelsProvider } from \"ai-gateway-kit\";\n\nconst gateway = createAIGateway({\n  models: [\n    {\n      id: \"gpt-4o-mini\",\n      provider: \"github\",\n      capabilities: [\"fast_text\"],\n      limits: { rpm: 15, rpd: 150, tpmInput: 150000, tpmOutput: 20000, concurrency: 3 }\n    }\n  ],\n  providers: {\n    github: createGitHubModelsProvider({\n      token: process.env.GITHUB_TOKEN!\n    })\n  }\n});\n\nconst result = await gateway.execute({\n  capability: \"fast_text\",\n  input: {\n    kind: \"chat\",\n    messages: [{ role: \"user\", content: \"Say hi.\" }]\n  }\n});\n\nconsole.log(result.output);\n```\n\n**📚 [See more examples →](./examples/)**\n\n## Core Features\n\n### Capability-based routing\nRoute requests by capability, not model names. See [examples/02-capability-routing.ts](./examples/02-capability-routing.ts).\n\n### Automatic fallback\nGraceful degradation across models. See [examples/03-fallback-handling.ts](./examples/03-fallback-handling.ts).\n\n### Rate limiting\nIn-memory rate limits (rpm, rpd, tpm, concurrency). See [examples/03-fallback-handling.ts](./examples/03-fallback-handling.ts).\n\n### Multiple providers\nGitHub Models, Gemini, or custom providers. See [examples/04-multi-provider.ts](./examples/04-multi-provider.ts).\n\n### Advanced features\n- JSON mode: [examples/06-json-mode.ts](./examples/06-json-mode.ts)\n- Web search: [examples/07-search-capability.ts](./examples/07-search-capability.ts)\n- Temperature control: [examples/08-temperature-control.ts](./examples/08-temperature-control.ts)\n- Request cancellation: [examples/11-abort-requests.ts](./examples/11-abort-requests.ts)\n- Dynamic registration: [examples/12-dynamic-registration.ts](./examples/12-dynamic-registration.ts)\n\n## Providers\n\n- **GitHub Models**: OpenAI models via GitHub ([docs](./examples/04-multi-provider.ts))\n- **Gemini**: Google Gemini models with search ([docs](./examples/07-search-capability.ts))\n- **Custom provider**: Implement `ProviderAdapter` interface\n\n## Observability hooks\n\nYou can subscribe to lifecycle events without taking a dependency on any logging stack:\n\n- `onRequestStart` - When a request begins\n- `onRequestEnd` - When a request completes (success or failure)\n- `onRateLimit` - When rate limits are encountered\n- `onFallback` - When falling back to another model\n- `onError` - When errors occur\n\n**Example:** [examples/09-observability-hooks.ts](./examples/09-observability-hooks.ts)\n\n```ts\nimport { createAIGateway, createGitHubModelsProvider, type GatewayHooks } from \"ai-gateway-kit\";\n\nconst hooks: GatewayHooks = {\n  onRequestStart: (event) =\u003e {\n    console.log(`Starting: ${event.modelId}`);\n  },\n  onRequestEnd: (event) =\u003e {\n    const duration = event.endedAt - event.startedAt;\n    console.log(`${event.ok ? 'Success' : 'Failed'}: ${event.modelId} (${duration}ms)`);\n  },\n  onRateLimit: (event) =\u003e {\n    console.log(`Rate limit: ${event.modelId} - ${event.decision.reason}`);\n  },\n  onFallback: (event) =\u003e {\n    console.log(`Fallback: ${event.fromModelId} → ${event.toModelId}`);\n  },\n  onError: (event) =\u003e {\n    console.error(`Error: ${event.modelId} - ${event.error.message}`);\n  }\n};\n\nconst gateway = createAIGateway({\n  models: [...],\n  providers: {\n    github: createGitHubModelsProvider({ token: process.env.GITHUB_TOKEN! })\n  },\n  hooks\n});\n```\n\n## Examples\n\nThe [examples](./examples/) directory contains comprehensive examples for all features:\n\n| Example | Description |\n|---------|-------------|\n| [01-basic-setup.ts](./examples/01-basic-setup.ts) | Minimal setup to get started |\n| [02-capability-routing.ts](./examples/02-capability-routing.ts) | Route by capability, not model name |\n| [03-fallback-handling.ts](./examples/03-fallback-handling.ts) | Automatic fallback when rate limited |\n| [04-multi-provider.ts](./examples/04-multi-provider.ts) | Use GitHub + Gemini together |\n| [05-custom-routing.ts](./examples/05-custom-routing.ts) | Implement custom routing logic |\n| [06-json-mode.ts](./examples/06-json-mode.ts) | Request structured JSON output |\n| [07-search-capability.ts](./examples/07-search-capability.ts) | Web search with Gemini |\n| [08-temperature-control.ts](./examples/08-temperature-control.ts) | Control creativity with temperature |\n| [09-observability-hooks.ts](./examples/09-observability-hooks.ts) | Monitor with lifecycle hooks |\n| [10-agent-tracking.ts](./examples/10-agent-tracking.ts) | Track multi-agent systems |\n| [11-abort-requests.ts](./examples/11-abort-requests.ts) | Cancel in-flight requests |\n| [12-dynamic-registration.ts](./examples/12-dynamic-registration.ts) | Add models at runtime |\n\n**[View all examples →](./examples/)**\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fukrocks007%2Fai-gateway-kit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fukrocks007%2Fai-gateway-kit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fukrocks007%2Fai-gateway-kit/lists"}