{"id":43175180,"url":"https://github.com/8monkey-ai/hebo-gateway","last_synced_at":"2026-04-07T05:02:08.267Z","repository":{"id":333272701,"uuid":"1136757614","full_name":"8monkey-ai/hebo-gateway","owner":"8monkey-ai","description":"AI Gateway as a Framework. Full control over models, routing \u0026 lifecycle. OpenAI-compatible /chat/completions, /embeddings \u0026 /models endpoints.","archived":false,"fork":false,"pushed_at":"2026-03-04T11:59:52.000Z","size":856,"stargazers_count":16,"open_issues_count":8,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-04T12:47:35.547Z","etag":null,"topics":["ai","ai-gateway","anthrophic","bedrock","elysiajs","gateway","groq","hono","javascript","llm","llm-gateway","llmops","nextjs","openai","tanstack-start","typescript","vercel-ai-sdk","vertex"],"latest_commit_sha":null,"homepage":"https://hebo.ai/gateway","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/8monkey-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-01-18T09:50:56.000Z","updated_at":"2026-03-04T11:59:55.000Z","dependencies_parsed_at":"2026-02-18T04:03:23.965Z","dependency_job_id":null,"html_url":"https://github.com/8monkey-ai/hebo-gateway","commit_stats":null,"previous_names":["8monkey-ai/hebo-gateway"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/8monkey-ai/hebo-gateway","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/8monkey-ai%2Fhebo-gateway","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/8monkey-ai%2Fhebo-gateway/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/8monkey-ai%2Fhebo-gateway/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/8monkey-ai%2Fhebo-gateway/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/8monkey-ai","download_url":"https://codeload.github.com/8monkey-ai/hebo-gateway/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/8monkey-ai%2Fhebo-gateway/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30113124,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T03:40:26.266Z","status":"ssl_error","status_checked_at":"2026-03-05T03:39:15.902Z","response_time":93,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-gateway","anthrophic","bedrock","elysiajs","gateway","groq","hono","javascript","llm","llm-gateway","llmops","nextjs","openai","tanstack-start","typescript","vercel-ai-sdk","vertex"],"created_at":"2026-02-01T03:03:21.815Z","updated_at":"2026-04-07T05:02:08.253Z","avatar_url":"https://github.com/8monkey-ai.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hebo Gateway\n\nRoll your own AI gateway for full control over models, providers, routing logic, guardrails, observability and more ...\n\n## 🐒 Overview\n\nExisting AI gateways like OpenRouter, Vercel AI Gateway, LiteLLM, and Portkey work out of the box, but they’re hard to extend once your needs go beyond configuration.\n\nHebo Gateway is an open-source, embeddable AI gateway framework built to live inside your app. It gives you full control over providers, models, routing, and the request lifecycle.\n\nLearn more in our blog post: [Yet Another AI Gateway?](https://hebo.ai/blog/260127-hebo-gateway/) (`https://hebo.ai/blog/260127-hebo-gateway/`)\n\n## 🍌 Features\n\n- 🌐 OpenAI-compatible /chat/completions, /embeddings \u0026 /models endpoints.\n- 🔄 /responses endpoint implementing the Open Responses API (stateless).\n- 💬 /conversations endpoint built on top of the Responses API.\n- 🔌 Integrate into your existing Hono, Elysia, Next.js \u0026 TanStack apps.\n- 🧩 Provider registry compatible with Vercel AI SDK providers.\n- 🧭 Canonical model IDs and parameter naming across providers.\n- 🗂️ Model catalog with extensible metadata capabilities.\n- 🪝 Hook system to customize routing, auth, rate limits, and shape responses.\n- 🧰 Low-level OpenAI-compatible schema, converters, and middleware helpers.\n- 👁️ Observability via OTel GenAI semantic conventions (Langfuse-compatible).\n\n## 📦 Installation\n\n```bash\nbun install @hebo-ai/gateway\n```\n\n## ☰ Table of Contents\n\n- Quickstart\n  - [Setup A Gateway Instance](#setup-a-gateway-instance) | [Mount Route Handlers](#mount-route-handlers) | [Call the Gateway](#call-the-gateway)\n- Configuration Reference\n  - [Providers](#providers) | [Models](#models) | [Hooks](#hooks) | [Storage](#storage) | [Logger](#logger-settings) | [Observability](#observability) | [Timeouts](#timeout-settings)\n- Framework Support\n  - [ElysiaJS](#elysiajs) | [Hono](#hono) | [Next.js](#nextjs) | [TanStack Start](#tanstack-start)\n- Runtime Support\n  - [Vercel Edge](#vercel-edge) | [Cloudflare Workers](#cloudflare-workers) | [Deno Deploy](#deno-deploy) | [AWS Lambda](#aws-lambda)\n- Endpoints\n  - [/chat/completions](#chatcompletions) | [/embeddings](#embeddings) | [/models](#models) | [/responses](#responses) | [/conversations](#conversations)\n- OpenAI Extensions\n  - [Reasoning](#reasoning) | [Service Tier](#service-tier) | [Prompt Caching](#prompt-caching)\n- Advanced Usage\n  - [Passing Framework State to Hooks](#passing-framework-state-to-hooks) | [Selective Route Mounting](#selective-route-mounting) | [Low-level Schemas \u0026 Converters](#low-level-schemas--converters)\n\n## 🚀 Quickstart\n\n### Setup A Gateway Instance\n\nStart by creating a gateway instance with at least one provider and a few models.\n\n```ts\nimport { createGroq } from \"@ai-sdk/groq\";\nimport { gateway, defineModelCatalog } from \"@hebo-ai/gateway\";\nimport { withCanonicalIdsForGroq } from \"@hebo-ai/gateway/providers/groq\";\nimport { gptOss20b, gptOss } from \"@hebo-ai/gateway/models/openai\";\n\nexport const gw = gateway({\n  // PROVIDER REGISTRY\n  providers: {\n    // Any Vercel AI SDK provider + withCanonicalIdsForX helper\n    groq: withCanonicalIdsForGroq(\n      createGroq({\n        apiKey: process.env.GROQ_API_KEY,\n      }),\n    ),\n  },\n\n  // MODEL CATALOG\n  models: defineModelCatalog(\n    // Choose a pre-configured preset for common SOTA models\n    gptOss20b,\n    // Or add a whole model family with your own provider list\n    gptOss[\"all\"].map((preset) =\u003e\n      preset({\n        providers: [\"groq\"],\n      }),\n    ),\n  ),\n});\n```\n\n\u003e [!NOTE]\n\u003e Don't forget to install the Groq provider package too: `@ai-sdk/groq`.\n\n\u003e [!TIP]\n\u003e Why `withCanonicalIdsForX`? In most cases you want your gateway to route using model IDs that are consistent across providers (e.g. `openai/gpt-oss-20b` rather than `openai.gpt-oss-20b-v1:0`). We call that `Canonical IDs` - they are what enable routing, fallbacks, and policy rules. Without this wrapper, providers only understands their native IDs, which would make cross-provider routing impossible.\n\n### Mount Route Handlers\n\nHebo Gateway plugs into your favorite web framework. Simply mount the gateway’s `handler` under a prefix, and keep using your existing lifecycle hooks for authentication, logging, observability, and more.\n\nHere is an example using **ElysiaJS** (our favorite):\n\n`src/index.ts`\n\n```ts\nimport { Elysia } from \"elysia\";\n\n// Previously created gateway instance\nconst gw = gateway({\n  /// ...\n});\n\nconst app = new Elysia().mount(\"/v1/gateway/\", gw.handler).listen(3000);\n\nconsole.log(`🐒 Hebo Gateway is running with Elysia at ${app.server?.url}`);\n```\n\n### Call the Gateway\n\nSince Hebo Gateway exposes OpenAI-compatible endpoints, it can be used with a broad set of common AI SDKs like **Vercel AI SDK**, **TanStack AI**, **LangChain**, the official **OpenAI SDK** and others.\n\nHere is a quick example using the Vercel AI SDK:\n\n```ts\nimport { createOpenAICompatible } from \"@ai-sdk/openai-compatible\";\nimport { generateText } from \"ai\";\n\nconst hebo = createOpenAICompatible({\n  name: \"hebo\",\n  baseURL: \"http://localhost:3000/v1/gateway\",\n});\n\nconst { text } = await generateText({\n  // Notice how this is using 'openai/gpt-oss-20b' instead of 'gpt-oss-20b'\n  // The gateway automatically maps modelIDs to the upstream provider ones\n  model: hebo(\"openai/gpt-oss-20b\"),\n  prompt: \"Tell me a joke about monkeys\",\n});\n\nconsole.log(text);\n```\n\n## ⚙️ Configuration Reference\n\n### Providers\n\nHebo Gateway’s provider registry accepts any **Vercel AI SDK Provider**. For Hebo to be able to route a model across different providers, the names need to be canonicalized to a common form, for example 'openai/gpt-4.1-mini' instead of 'gpt-4.1-mini'.\n\nWe currently provide out-of-the-box canonical providers for: `Bedrock`, `Anthropic`, `Cohere`, `Vertex`, `Groq`, `OpenAI`, and `Voyage`. Import the helper from the matching package path:\n\n```ts\n// pattern: @hebo-ai/gateway/providers/\u003cprovider\u003e\nimport { withCanonicalIdsForGroq } from \"@hebo-ai/gateway/providers/groq\";\n```\n\nIf an adapter is not yet provided, you can create your own by wrapping the provider instance with the `withCanonicalIds` helper and define your custom canonicalization mapping \u0026 rules.\n\nFor Azure, use `createAzure` from `@ai-sdk/azure` directly. Name each [Azure AI Foundry](https://learn.microsoft.com/en-us/azure/ai-foundry/foundry-models/concepts/endpoints) deployment after its Hebo canonical ID (e.g. `anthropic/claude-sonnet-4.5`).\n\nFor other providers, use `withCanonicalIds` with an explicit `mapping`:\n\n```ts\nimport { createOpenAICompatible } from \"@ai-sdk/openai-compatible\";\nimport { gateway, withCanonicalIds } from \"@hebo-ai/gateway\";\n\nconst myProvider = withCanonicalIds(\n  createOpenAICompatible({\n    name: \"my-provider\",\n    baseURL: \"https://api.my-provider.com/v1\",\n    apiKey: process.env[\"MY_PROVIDER_API_KEY\"],\n  }),\n  {\n    mapping: {\n      \"openai/gpt-4.1-mini\": \"gpt-4.1-mini-custom\",\n      \"anthropic/claude-sonnet-4.5\": \"claude-sonnet-4-5\",\n    },\n  },\n);\n\nconst gw = gateway({\n  providers: {\n    myProvider,\n  },\n  models: {\n    // ...your models pointing at canonical IDs above\n  },\n});\n```\n\n### Models\n\nRegister models to tell the gateway what's available, under which canonical ID and what capabilities each one has.\n\n#### Model Presets\n\nTo simplify the registration, Hebo Gateway ships a set of model presets under `@hebo-ai/gateway/models`. Use these when you want ready-to-use catalog entries with sane defaults for common SOTA models.\n\nPresets come in two forms:\n\n- Individual presets (e.g. `gptOss20b`, `claudeSonnet45`) for a single model.\n- Family presets (e.g. `claude`, `gemini`, `llama`) which group multiple models and expose helpers like `latest`, `all`, `vX` (e.g. `claude[\"v4.5\"]`).\n\n```ts\nimport { defineModelCatalog } from \"@hebo-ai/gateway\";\nimport { gptOss20b } from \"@hebo-ai/gateway/models/openai\";\nimport { claudeSonnet45, claude } from \"@hebo-ai/gateway/models/anthropic\";\n\n// Individual preset\nconst models = defineModelCatalog(\n  gptOss20b({ providers: [\"groq\"] }),\n  claudeSonnet45({ providers: [\"bedrock\"] }),\n);\n\n// Family preset (pick a group and apply the same override to each)\nconst modelsFromFamily = defineModelCatalog(\n  claude[\"latest\"].map((preset) =\u003e preset({ providers: [\"anthropic\"] })),\n);\n```\n\nOut-of-the-box model presets:\n\n- **Amazon** — `@hebo-ai/gateway/models/amazon`  \n  Nova: `nova` (`v1`, `v2`, `v1.x`, `v2.x`, `latest`, `embeddings`, `all`)\n\n- **Anthropic** — `@hebo-ai/gateway/models/anthropic`  \n  Claude: `claude` (`v4.6`, `v4.5`, `v4.1`, `v4`, `v3.7`, `v3.5`, `v3`, `v4.x`, `v3.x`, `haiku`, `sonnet`, `opus`, `latest`, `all`)\n\n- **Cohere** — `@hebo-ai/gateway/models/cohere`  \n  Command: `command` (`A`, `R`, `latest`, `all`)\n  Embed: `embed` (`v4`, `v3`, `latest`, `all`)\n\n- **Google** — `@hebo-ai/gateway/models/google`  \n  Gemini: `gemini` (`v2.5`, `v3-preview`, `v2.x`, `v3.x`, `embeddings`, `latest`, `preview`, `all`)\n\n- **Meta** — `@hebo-ai/gateway/models/meta`  \n  Llama: `llama` (`v3.1`, `v3.2`, `v3.3`, `v4`, `v3.x`, `v4.x`, `latest`, `all`)\n\n- **OpenAI** — `@hebo-ai/gateway/models/openai`  \n  GPT: `gpt` (`v5`, `v5.1`, `v5.2`, `v5.3`, `v5.x`, `chat`, `codex`, `pro`, `latest`, `all`)  \n  GPT-OSS: `gptOss` (`v1`, `v1.x`, `latest`, `all`)\n  Embeddings: `textEmbeddings` (`v3`, `v3.x`, `latest`, `all`)\n\n- **Voyage** — `@hebo-ai/gateway/models/voyage`  \n  Voyage: `voyage` (`v2`, `v3`, `v3.5`, `v4`, `v2.x`, `v3.x`, `v4.x`, `latest`, `all`)\n\n#### User-defined Models\n\nAs the ecosystem is moving faster than anyone can keep-up with, you can always register your own model entries by following the `CatalogModel` type.\n\n```ts\nconst gw = gateway({\n  providers: {\n    // ...\n  },\n  models: {\n    \"openai/gpt-5.2\": {\n      name: \"GPT 5.2\",\n      created: \"2025-12-11\",\n      knowledge: \"2025-08\",\n      modalities: {\n        input: [\"text\", \"image\", \"pdf\", \"file\"],\n        output: [\"text\"],\n      },\n      context: 400000,\n      capabilities: [\"attachments\", \"reasoning\", \"tool_call\", \"structured_output\", \"temperature\"],\n      providers: [\"openai\"],\n      // Additional properties are merged into the model object\n      additionalProperties: {\n        customProperty: \"customValue\",\n      },\n    },\n    // ...\n  },\n});\n```\n\n\u003e [!NOTE]\n\u003e The only mandatory property is the `providers` array, everything else is optional metadata.\n\n### Hooks\n\nHooks allow you to plug into the lifecycle of the gateway and enrich it with additional functionality, like your actual routing logic. All hooks are available as async and non-async.\n\n```ts\nconst gw = gateway({\n  providers: {\n    // ...\n  },\n  models: {\n    // ...\n  },\n  hooks: {\n    /**\n     * Runs before any endpoint handler logic.\n     * @param ctx.request Incoming request.\n     * @returns Optional Response to short-circuit the request.\n     */\n    onRequest: async (ctx: { request: Request }): Promise\u003cResponse | void\u003e =\u003e {\n      // Example Use Cases:\n      // - Verify authentication\n      // - Enforce rate limits\n      return undefined;\n    },\n    /**\n     * Runs after body is parsed \u0026 validated.\n     * @param ctx.body Parsed request body.\n     * @returns Replacement parsed body, or undefined to keep original body unchanged.\n     */\n    before: async (ctx: {\n      body: ChatCompletionsBody | EmbeddingsBody;\n      operation: \"chat\" | \"embeddings\";\n    }): Promise\u003cChatCompletionsBody | EmbeddingsBody | void\u003e =\u003e {\n      // Example Use Cases:\n      // - Transform request body\n      // - Observability integration\n      return undefined;\n    },\n    /**\n     * Maps a user-provided model ID or alias to a canonical ID.\n     * @param ctx.body The parsed body object with all call parameters.\n     * @param ctx.modelId Incoming model ID.\n     * @returns Canonical model ID or undefined to keep original.\n     */\n    resolveModelId: async (ctx: {\n      body: ChatCompletionsBody | EmbeddingsBody;\n      modelId: ModelId;\n    }): Promise\u003cModelId | void\u003e =\u003e {\n      // Example Use Cases:\n      // - Resolve modelAlias to modelId\n      return undefined;\n    },\n    /**\n     * Picks a provider instance for the request.\n     * @param ctx.providers ProviderRegistry from config.\n     * @param ctx.models ModelCatalog from config.\n     * @param ctx.body The parsed body object with all call parameters.\n     * @param ctx.resolvedModelId Resolved model ID.\n     * @param ctx.operation Operation type (\"chat\" | \"embeddings\").\n     * @returns ProviderV3 to override, or undefined to use default.\n     */\n    resolveProvider: async (ctx: {\n      providers: ProviderRegistry;\n      models: ModelCatalog;\n      body: ChatCompletionsBody | EmbeddingsBody;\n      resolvedModelId: ModelId;\n      operation: \"chat\" | \"embeddings\";\n    }): Promise\u003cProviderV3 | void\u003e =\u003e {\n      // Example Use Cases:\n      // - Routing logic between providers\n      // - Bring-your-own-key authentication\n      return undefined;\n    },\n    /**\n     * Runs after the endpoint handler.\n     * @param ctx.result Result object returned by the handler.\n     * @returns Modified result, or undefined to keep original.\n     */\n    after: async (ctx: {\n      result: ChatCompletions | ChatCompletionsStream | Embeddings;\n    }): Promise\u003cChatCompletions | ChatCompletionsStream | Embeddings | void\u003e =\u003e {\n      // Example Use Cases:\n      // - Transform result\n      // - Result logging\n      return undefined;\n    },\n    /**\n     * Runs after the gateway has produced the final Response.\n     * @param ctx.response Response object returned by the lifecycle.\n     * @returns Replacement response, or undefined to keep original.\n     */\n    onResponse: async (ctx: { response: Response }): Promise\u003cResponse | void\u003e =\u003e {\n      // Example Use Cases:\n      // - Add response headers\n      // - Replace or redact response payload\n      return undefined;\n    },\n    /**\n     * Runs when the lifecycle catches an error.\n     * @param ctx.error The thrown error.\n     * @returns Replacement error response, or undefined to use the default OpenAI-compatible error response.\n     */\n    onError: async (ctx: { error: unknown }): Promise\u003cResponse | void\u003e =\u003e {\n      // Example Use Cases:\n      // - Map internal errors to custom API responses\n      // - Add app-specific logging or alerting\n      return undefined;\n    },\n  },\n});\n```\n\nThe `ctx` object is **readonly for core fields**. Use return values to override request / parsed body / result / response and to provide modelId / provider instances.\n\n\u003e [!TIP]\n\u003e To pass data between hooks, use `ctx.state`. It’s a per-request mutable bag in which you can stash things like auth info, routing decisions, timers, or trace IDs and read them later again in any of the other hooks.\n\n### Storage\n\nThe `/conversations` endpoint stores conversation history and associated items. By default, the gateway uses an in-memory storage, which is suitable for development but not for production as data is lost when the server restarts.\n\n#### In-Memory Storage\n\nYou can configure the size of the in-memory storage (default is 256MB).\n\n```ts\nimport { gateway } from \"@hebo-ai/gateway\";\nimport { InMemoryStorage } from \"@hebo-ai/gateway/storage/memory\";\n\nconst gw = gateway({\n  // ...\n  storage: new InMemoryStorage({\n    maxSize: 512 * 1024 * 1024, // 512MB\n  }),\n});\n```\n\n#### SQL Storage\n\nHebo Gateway provides high-performance SQL adapters for **PostgreSQL**, **SQLite**, **MySQL**, and **GrepTimeDB**. It supports common drivers like `pg`, `postgres.js`, `mysql2`, `better-sqlite3`, `@libsql/client`, and `Bun.SQL`.\n\n```ts\nimport { gateway } from \"@hebo-ai/gateway\";\nimport { SqlStorage, PostgresDialect } from \"@hebo-ai/gateway/storage/sql\";\nimport { Pool } from \"pg\";\n\n// 1. Setup dialect-specific client (e.g. pg, mysql2, sqlite, bun)\nconst client = new Pool({ connectionString: process.env.DATABASE_URL });\n\n// 2. Setup storage with matching dialect (PostgresDialect, SqliteDialect, MysqlDialect, ...)\nconst storage = new SqlStorage({\n  dialect: new PostgresDialect({ client }),\n});\n\n// 3. Run migrations\nawait storage.migrate();\n\nconst gw = gateway({ storage });\n```\n\n\u003e [!TIP]\n\u003e The `PostgresDialect` includes optimized `JSONB` storage and high-performance `BRIN` indexing for time-ordered data by default.\n\n## 🧩 Framework Support\n\nHebo Gateway exposes **WinterCG-compatible** handlers that integrate with almost any existing framework.\n\n### ElysiaJS\n\n`src/index.ts`\n\n```ts\nimport { Elysia } from \"elysia\";\n\nconst app = new Elysia().mount(\"/v1/gateway/\", gw.handler).listen(3000);\n\nconsole.log(`🐒 Hebo Gateway is running with Elysia at ${app.server?.url}`);\n```\n\n### Hono\n\n`src/index.ts`\n\n```ts\nimport { Hono } from \"hono\";\n\nexport default new Hono().mount(\"/v1/gateway/\", gw.handler);\n\nconsole.log(`🐒 Hebo Gateway is running with Hono framework`);\n```\n\n### Next.js\n\n#### App Router\n\n`app/api/gateway/[...all]/route.ts`\n\n```ts\nconst gw = gateway({\n  // Required: add `basePath` to your gateway config\n  basePath: \"/api/gateway\",\n  // ...\n});\n\nexport const POST = gw.handler,\n  GET = gw.handler;\n```\n\n#### Pages Router\n\n`pages/api/gateway/[...all].ts`\n\n```ts\n// Requires `@mjackson/node-fetch-server` npm package\nimport { createRequest, sendResponse } from \"@mjackson/node-fetch-server\";\n\nconst gw = gateway({\n  // Required: add `basePath` to your gateway config\n  basePath: \"/api/gateway\",\n  // ...\n});\n\nexport default async function handler(req, res) {\n  await sendResponse(res, await gw.handler(createRequest(req, res)));\n}\n```\n\n### TanStack Start\n\n`routes/api/$.ts`\n\n```ts\nconst gw = gateway({\n  // Required: add `basePath` to your gateway config\n  basePath: \"/api/gateway\",\n  // ...\n});\n\nexport const Route = createFileRoute(\"/api/$\")({\n  server: {\n    handlers: {\n      GET: ({ request }) =\u003e gw.handler(request),\n      POST: ({ request }) =\u003e gw.handler(request),\n    },\n  },\n});\n```\n\n## 🌍 Runtime Support\n\nHebo Gateway also works directly with runtime-level `Request -\u003e Response` handlers.\n\n### Vercel Edge\n\n`api/gateway.ts`\n\n```ts\nexport const runtime = \"edge\";\n\nconst gw = gateway({\n  // ...\n});\n\nexport default gw.handler;\n```\n\n### Cloudflare Workers\n\n`src/index.ts`\n\n```ts\nconst gw = gateway({\n  // ...\n});\n\nexport default {\n  fetch: gw.handler,\n};\n```\n\n### Deno Deploy\n\n`main.ts`\n\n```ts\nimport { serve } from \"https://deno.land/std/http/server.ts\";\n\nconst gw = gateway({\n  // ...\n});\n\nserve((request: Request) =\u003e gw.handler(request));\n```\n\n### AWS Lambda\n\n`src/lambda.ts`\n\n```ts\nimport { awsLambdaEventHandler } from \"@hattip/adapter-aws-lambda\";\n\nconst gw = gateway({\n  // ...\n});\n\nexport const handler = awsLambdaEventHandler({\n  handler: gw.handler,\n});\n```\n\n## 🚀 Endpoints\n\nHebo Gateway provides several OpenAI-compatible and standard-based endpoints.\n\n### `/chat/completions`\n\nThe primary endpoint for generating chat completions.\n\nOfficial documentation: [OpenAI API Reference](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create)\n\nIt supports:\n\n- Streaming responses (Server-Sent Events).\n- Tool calling / Function calling.\n- Advanced extensions like [Reasoning](#reasoning), [Service Tier](#service-tier), and [Prompt Caching](#prompt-caching).\n- Usage tracking and metadata.\n\n\u003e [!IMPORTANT]\n\u003e **Compatibility \u0026 Roadmap:**\n\u003e We are actively working to expand support for the full OpenAI spec:\n\n- **`logprobs` / `top_logprobs`**: Token-level logprobs.\n- **`logit_bias`**: Logit bias in the request body.\n- **`n` \u003e 1**: Multi-choice completions.\n\n### `/embeddings`\n\nGenerates vector representations for text inputs, compatible with OpenAI's embeddings API.\n\nOfficial documentation: [OpenAI API Reference](https://developers.openai.com/api/reference/resources/embeddings/methods/create)\n\nIt supports:\n\n- Text and token array inputs.\n- Custom dimensions (for `v3` models).\n- Standard `float` and `base64` encoding formats.\n\n\u003e [!IMPORTANT]\n\u003e **Compatibility \u0026 Roadmap:**\n\n- **`encoding_format`**: `base64` results.\n\n### `/models`\n\nLists all available models in your [Model Catalog](#models), including their capabilities and metadata.\n\nOfficial documentation: [OpenAI API Reference](https://developers.openai.com/api/reference/resources/models/methods/list)\n\nIt supports:\n\n- Comprehensive model metadata (capabilities, context limits, knowledge cutoffs).\n- Canonical model ID resolution.\n- Provider-specific availability filtering.\n\n### `/responses`\n\nHebo Gateway provides a `/responses` endpoint implementing the [Open Responses API](https://www.openresponses.org/reference).\n\nOfficial documentation: [Open Responses API Reference](https://www.openresponses.org/reference)\n\nIt supports:\n\n- The same models, providers, hooks, and extensions as `/chat/completions`.\n- Responses API request/response format.\n- Tool calling and multimodal inputs.\n- Normalized reasoning and thought signatures.\n\n\u003e [!IMPORTANT]\n\u003e **Compatibility \u0026 Roadmap:**\n\u003e We are working towards full Open Responses parity:\n\n- **Persistence**: Server-side response storage (`store`), background orchestration (`background`), and chaining via `previous_response_id`.\n- **`conversation`**: Directly passing conversation IDs for automatic context management.\n- **`context_management`**: Support for automatic compaction strategies.\n- **`prompt`**: Reusable prompt templates with variables.\n- **`phase`**: Support for `commentary` vs `final_answer` reasoning phases.\n- **`safety_identifier`**: Custom safety and moderation policies.\n- **`truncation`**: Context window management strategies.\n- **`text.verbosity`**: Control over response detail (low/medium/high).\n- **`logprobs` / `top_logprobs`**: Token-level logprobs.\n- **`include`**: Selective response fields (e.g., `logprobs`, `reasoning.encrypted_content`, and tool-specific outputs).\n- **`stream_options.include_obfuscation`**: Normalizing payload sizes to mitigate side-channel attacks.\n\n### `/conversations`\n\nHebo Gateway provides a dedicated `/conversations` endpoint for managing persistent conversation state. It is designed as an extension of the [OpenAI Conversations API](https://developers.openai.com/api/reference/resources/conversations/methods/create) and supports standard CRUD operations alongside advanced listing with metadata filtering.\n\nOfficial documentation: [OpenAI Conversations API](https://developers.openai.com/api/reference/resources/conversations/methods/create)\n\n#### List \u0026 Filter Conversations (Hebo Extension)\n\nSince standard OpenAI APIs (like Threads) do not support global listing of conversations, Hebo Gateway provides this capability as an extension. You can list all conversations using cursor-based pagination and filter by any metadata key using the `metadata.KEY=VALUE` pattern.\n\n```bash\n# List conversations for a specific user using metadata filtering\ncurl \"https://api.gateway.com/conversations?limit=10\u0026metadata.user_id=123\"\n```\n\nThe response follows the standard OpenAI list object:\n\n```json\n{\n  \"object\": \"list\",\n  \"data\": [\n    {\n      \"id\": \"conv_abc123\",\n      \"object\": \"conversation\",\n      \"created_at\": 1678531200,\n      \"metadata\": { \"user_id\": \"123\" }\n    }\n  ],\n  \"first_id\": \"conv_abc123\",\n  \"last_id\": \"conv_abc123\",\n  \"has_more\": false\n}\n```\n\n## 🧠 OpenAI Extensions\n\n### Reasoning\n\nIn addition to the official `reasoning_effort` parameter, the chat completions endpoint accepts a `reasoning` object for more fine-grained control of the budget. It's treated as provider-agnostic input and normalized before hitting the upstream model.\n\n```json\n{\n  \"model\": \"anthropic/claude-4-sonnet\",\n  \"messages\": [{ \"role\": \"user\", \"content\": \"Explain the tradeoffs.\" }],\n  \"reasoning\": { \"effort\": \"medium\" }\n}\n```\n\nNormalization rules:\n\n- `enabled` -\u003e fall-back to model default if none provided\n- `max_tokens`: fall-back to model default if model supports\n- `effort` supports: `none`, `minimal`, `low`, `medium`, `high`, `xhigh`\n- Generic `effort` -\u003e budget = percentage of `max_tokens`\n  - `none`: 0%\n  - `minimal`: 10%\n  - `low`: 20%\n  - `medium`: 50% (default)\n  - `high`: 80%\n  - `xhigh`: 95%\n\nReasoning output is surfaced as extension to the `completion` object.\n\n- When present, it is returned on the assistant message as `reasoning_content`. Reasoning token counts (when available) are returned on `usage.completion_tokens_details.reasoning_tokens`.\n- For stream responses, reasoning text is sent incrementally as `reasoning_content` part (separate from normal text `content` deltas). Token counts land in the final `usage` object on the terminating chunk.\n\nMost SDKs handle these fields out-of-the-box.\n\n#### Thinking Blocks \u0026 Context Preservation\n\nAdvanced models (like Anthropic Claude 3.7 or Gemini 3) surface structured reasoning steps and signatures that act as a \"save state\" for the model's internal reasoning process. To maintain this context across multi-turn conversations and tool-calling workflows, you should pass back the following extensions in subsequent messages:\n\n- **reasoning_details**: Standardized array of reasoning steps and generic signatures.\n- **extra_content**: Provider-specific extensions, such as **Google's thought signatures** on Vertex AI.\n\nFor **Gemini 3** models, returning the thought signature via `extra_content` is mandatory to resume the chain-of-thought; failing to do so may result in errors or degraded performance.\n\n### Service Tier\n\nThe chat completions endpoint accepts a provider-agnostic `service_tier` extension:\n\n- `auto`, `default`, `flex`, `priority`, `scale`\n\nProvider-specific mapping:\n\n- **OpenAI**: forwards as OpenAI `serviceTier` (no middleware remap).\n- **Groq**: maps to Groq `serviceTier` (`default` -\u003e `on_demand`, `scale`/`priority` -\u003e `performance`).\n- **Google Vertex**: maps to request headers via middleware:\n  - `default` -\u003e `x-vertex-ai-llm-request-type: shared`\n  - `flex` -\u003e `x-vertex-ai-llm-request-type: shared` + `x-vertex-ai-llm-shared-request-type: flex`\n  - `priority` -\u003e `x-vertex-ai-llm-request-type: shared` + `x-vertex-ai-llm-shared-request-type: priority`\n  - `scale` -\u003e `x-vertex-ai-llm-request-type: dedicated`\n- **Amazon Bedrock**: maps to Bedrock `serviceTier.type` (`default`, `flex`, `priority`, `reserved`; `scale` -\u003e `reserved`, `auto` -\u003e omitted/default).\n\nWhen available, the resolved value is echoed back on response as `service_tier`.\n\n### Prompt Caching\n\nThe chat completions endpoint supports both implicit (provider-managed) and explicit prompt caching across OpenAI-compatible providers.\n\nAccepted request fields:\n\n- `prompt_cache_key` + `prompt_cache_retention` (OpenAI style)\n- `cache_control` (OpenRouter / Vercel / Claude style)\n- `extra_body { google: { cached_content } }` (Gemini style)\n\n```json\n{\n  \"model\": \"anthropic/claude-sonnet-4.6\",\n  \"messages\": [\n    {\n      \"role\": \"system\",\n      \"content\": \"Reusable policy and instructions\",\n      \"cache_control\": { \"type\": \"ephemeral\", \"ttl\": \"1h\" }\n    },\n    { \"role\": \"user\", \"content\": \"Apply policy to this request.\" }\n  ]\n}\n```\n\nProvider behavior:\n\n- **OpenAI-compatible**: forwards `prompt_cache_key` and `prompt_cache_retention` as native provider options.\n- **Anthropic Claude**: maps top-level caching to Anthropic cache control, while message/part `cache_control` breakpoints are preserved.\n- **Google Gemini**: maps `cached_content` to Gemini `cachedContent`.\n- **Amazon Nova (Bedrock)**: maps `cache_control` to Bedrock `cachePoints` and inserts an automatic cache point on a stable prefix when none is provided.\n\n## 🧪 Advanced Usage\n\n### Logger Settings\n\nYou can configure logging via the `logger` field in the gateway config. By default, the logger uses `console` and sets the level to `debug` in non-production and `info` in production (based on the `NODE_ENV` environment variable).\n\n```ts\nimport { gateway } from \"@hebo-ai/gateway\";\n\nconst gw = gateway({\n  // ...\n  logger: {\n    level: \"debug\", // \"trace\" | \"debug\" | \"info\" | \"warn\" | \"error\" | \"silent\"\n  },\n});\n```\n\nIf you provide a custom logger, it must implement `trace`, `debug`, `info`, `warn`, and `error` methods.\n\nExample with **pino**:\n\n```ts\nimport pino from \"pino\";\nimport { gateway } from \"@hebo-ai/gateway\";\n\nconst gw = gateway({\n  // ...\n  logger: pino({\n    level: \"info\",\n  }),\n});\n```\n\n\u003e [!TIP]\n\u003e For production workloads, we recommend `pino` for better logging performance and lower overhead.\n\n### Observability\n\nHebo Gateway can forward traces \u0026 metrics via the `telemetry` config field.\n\n```ts\nimport { gateway } from \"@hebo-ai/gateway\";\nimport { trace } from \"@opentelemetry/api\";\n\nconst gw = gateway({\n  // ...\n  telemetry: {\n    // default: false\n    enabled: true,\n    // default: TraceProvider from @opentelemetry/api singleton\n    tracer: trace.getTracer(\"my-gateway\"),\n    // Telemetry levels by namespace:\n    // \"off\" | \"required\" | \"recommended\" | \"full\"\n    signals: {\n      // gen_ai.* semantic attributes\n      gen_ai: \"full\",\n      // http.*, url.*, server.* semantic attributes\n      http: \"recommended\",\n      // hebo-specific telemetry:\n      // - recommended: hebo.* span events\n      // - full: hebo.* span events + fetch instrumentation\n      hebo: \"recommended\",\n    },\n  },\n});\n```\n\nAttribute names and span \u0026 metrics semantics follow OpenTelemetry GenAI semantic conventions:\nhttps://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/\nhttps://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/\n\n\u003e [!TIP]\n\u003e To populate custom span attributes, the inbound W3C `baggage` header is supported. Keys in the `hebo.` namespace are mapped to span attributes, with the namespace stripped. For example: `baggage: hebo.user_id=u-123` becomes span attribute `user_id=u-123`.  \n\u003e For `/chat/completions` and `/embeddings`, request `metadata` (`Record\u003cstring, string\u003e`, key 1-64 chars, value up to 512 chars) is also forwarded to spans as `gen_ai.request.metadata.\u003ckey\u003e`.\n\nFor observability integration that is not otel compliant, you can disable built-in telemetry and manually instrument requests during `before` / `after` hooks.\n\n#### Metrics\n\nThe Gateway also emits `gen_ai` metrics:\n\n- `gen_ai.server.request.duration` (histogram, seconds)\n- `gen_ai.server.time_per_output_token` (histogram, seconds)\n- `gen_ai.client.token.usage` (histogram, tokens; tagged with `gen_ai.token.type=input|output`)\n\nTo capture them, configure a global `MeterProvider` before creating the gateway:\n\n```ts\nimport { metrics } from \"@opentelemetry/api\";\nimport { OTLPMetricExporter } from \"@opentelemetry/exporter-metrics-otlp-http\";\nimport { MeterProvider, PeriodicExportingMetricReader } from \"@opentelemetry/sdk-metrics\";\nimport { gateway } from \"@hebo-ai/gateway\";\n\nmetrics.setGlobalMeterProvider(\n  new MeterProvider({\n    readers: [\n      new PeriodicExportingMetricReader({\n        exporter: new OTLPMetricExporter({\n          url: process.env.OTEL_EXPORTER_OTLP_METRICS_ENDPOINT,\n        }),\n      }),\n    ],\n  }),\n);\n\nconst gw = gateway({\n  // ...\n  telemetry: {\n    enabled: true,\n    signals: {\n      gen_ai: \"recommended\",\n    },\n  },\n});\n```\n\n\u003e [!NOTE]\n\u003e `telemetry.tracer` controls traces; metrics export is controlled by the global `MeterProvider`.\n\n#### Langfuse\n\nHebo telemetry spans are OpenTelemetry-compatible, so you can send them to Langfuse via `@langfuse/otel`.\n\n```ts\nimport { gateway } from \"@hebo-ai/gateway\";\nimport { LangfuseSpanProcessor } from \"@langfuse/otel\";\nimport { context } from \"@opentelemetry/api\";\nimport { AsyncLocalStorageContextManager } from \"@opentelemetry/context-async-hooks\";\nimport { BasicTracerProvider } from \"@opentelemetry/sdk-trace-base\";\n\ncontext.setGlobalContextManager(new AsyncLocalStorageContextManager().enable());\n\nconst gw = gateway({\n  // ...\n  telemetry: {\n    enabled: true,\n    tracer: new BasicTracerProvider({\n      spanProcessors: [new LangfuseSpanProcessor()],\n    }).getTracer(\"hebo\"),\n  },\n});\n```\n\nLangfuse credentials are read from environment variables by the Langfuse OTel SDK (`LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_BASE_URL`).\n\n### Timeout Settings\n\nYou can configure request timeouts via the `timeouts` field:\n\n```ts\nimport { gateway } from \"@hebo-ai/gateway\";\n\nconst gw = gateway({\n  // ...\n  // default timeout is 300_000 (5 minutes).\n  // You can set one timeout for all tiers...\n  timeouts: 60_000,\n  // ...disable timeouts completely:\n  // timeouts: null,\n  // ...or split by service tier:\n  // - normal: all non-flex tiers (set null to disable)\n  // - flex: defaults to 3x normal when omitted (set null to disable)\n  // timeouts: { normal: 30_000, flex: null },\n});\n```\n\n\u003e [!NOTE]\n\u003e **Runtime/engine timeout limits**\n\u003e Runtime-level `fetch()` clients may enforce their own timeouts. Configure those runtime/platform limits in addition to gateway `timeouts`.\n\u003e\n\u003e - Node.js runtimes use Undici: https://github.com/nodejs/undici/issues/1373 (Node.js, Vercel Serverless Functions, AWS Lambda)\n\u003e - Bun context: https://github.com/oven-sh/bun/issues/16682\n\u003e\n\u003e **Provider/service timeout limits**\n\u003e Serverless platforms (e.g. Cloudflare Workers, Vercel Edge/Serverless, AWS Lambda) also enforce platform time limits (roughly ~25-100s on edge paths, ~300s for streaming, and up to ~900s configurable for some).\n\n### Passing Framework State to Hooks\n\nYou can pass per-request info from your framework into the gateway via the second `state` argument on the handler, then read it in hooks through `ctx.state`.\n\n```ts\nimport { Elysia } from \"elysia\";\nimport { gateway } from \"@hebo-ai/gateway\";\n\nconst basePath = \"/v1/gateway\";\n\nconst gw = gateway({\n  basePath,\n  providers: {\n    // ...\n  },\n  models: {\n    // ...\n  },\n  hooks: {\n    resolveProvider: async (ctx) =\u003e {\n      // Select provider based on userId\n      const user = ctx.state.auth.userId;\n      if (user.startsWith(\"vip:\")) {\n        return ctx.providers[\"openai\"];\n      } else {\n        return ctx.providers[\"groq\"];\n      }\n    },\n  },\n});\n\nconst app = new Elysia()\n  .derive(({ headers }) =\u003e ({\n    auth: {\n      userId: headers[\"x-user-id\"],\n    },\n  }))\n  .all(`${basePath}/*`, ({ request, auth }) =\u003e gw.handler(request, { auth }), { parse: \"none\" })\n  .listen(3000);\n```\n\n\u003e [!NOTE]\n\u003e The `parse: 'none'` hook is required to prevent Elysia from consuming the body.\n\n### Selective Route Mounting\n\nIf you want to have more flexibility, for example for custom rate limit checks per route, you can also choose to only mount individual routes from the gateway's `routes` property.\n\n```ts\nconst gw = gateway({\n  /// ...\n});\n\nconst app = new Elysia()\n  .mount(\"/v1/gateway/chat\", gw.routes[\"/chat/completions\"].handler)\n  .listen(3000);\n\nconsole.log(`🐒 /chat/completions mounted to ${app.server?.url}/chat`);\n```\n\n### Low-level Schemas \u0026 Converters\n\nWe also provide full schemas, helper functions and types to convert between **OpenAI \u003c\u003e Vercel AI SDK** for advanced use cases like creating your own endpoint. They are available via deep-imports and completely tree-shakeable.\n\n```ts\nimport { streamText, wrapLanguageModel } from \"ai\";\nimport { createGroq } from \"@ai-sdk/groq\";\nimport * as z from \"zod\";\nimport {\n  ChatCompletionsBodySchema,\n  convertToTextCallOptions,\n  toChatCompletionsStreamResponse,\n} from \"@hebo-ai/gateway/endpoints/chat-completions\";\nimport { forwardParamsMiddleware } from \"@hebo-ai/gateway/middleware/common\";\n\nconst groq = createGroq({ apiKey: process.env.GROQ_API_KEY });\n\nexport async function handler(req: Request): Promise\u003cResponse\u003e {\n  const body = await req.json();\n\n  const parsed = ChatCompletionsBodySchema.safeParse(body);\n  if (!parsed.success) {\n    return new Response(z.prettifyError(parsed.error), { status: 422 });\n  }\n\n  const { model, ...inputs } = parsed.data;\n\n  const textOptions = convertToTextCallOptions(inputs);\n\n  const result = await streamText({\n    model: wrapLanguageModel({\n      model: groq(model),\n      middleware: forwardParamsMiddleware(\"groq\"),\n    }),\n    ...textOptions,\n  });\n\n  return toChatCompletionsStreamResponse(result, model);\n}\n```\n\nNon-streaming versions are available via `toChatCompletionsResponse`. Equivalent schemas and helpers are available in the `conversations`, `embeddings` and `models` endpoints.\n\n\u003e [!TIP]\n\u003e Since Zod v4.3 you can generate a JSON Schema from any zod object by calling `z.toJSONSchema(...)`. This is useful for producing OpenAPI documentation from the same source of truth.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F8monkey-ai%2Fhebo-gateway","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F8monkey-ai%2Fhebo-gateway","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F8monkey-ai%2Fhebo-gateway/lists"}