{"id":15036657,"url":"https://github.com/portkey-ai/gateway","last_synced_at":"2025-05-13T21:02:59.751Z","repository":{"id":193448607,"uuid":"682080300","full_name":"Portkey-AI/gateway","owner":"Portkey-AI","description":"A blazing fast AI Gateway with integrated guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast \u0026 friendly API.","archived":false,"fork":false,"pushed_at":"2025-05-06T06:30:12.000Z","size":63314,"stargazers_count":7776,"open_issues_count":102,"forks_count":578,"subscribers_count":41,"default_branch":"main","last_synced_at":"2025-05-06T20:22:55.216Z","etag":null,"topics":["ai-gateway","gateway","generative-ai","hacktoberfest","langchain","llama-index","llmops","llms","openai","prompt-engineering","router"],"latest_commit_sha":null,"homepage":"https://portkey.ai/features/ai-gateway","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Portkey-AI.png","metadata":{"files":{"readme":".github/README.cn.md","changelog":null,"contributing":".github/CONTRIBUTING.cn.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":".github/SECURITY.md","support":".github/SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-08-23T11:52:47.000Z","updated_at":"2025-05-06T19:42:58.000Z","dependencies_parsed_at":"2023-09-08T08:28:14.031Z","dependency_job_id":"c21c74ee-84dd-4b17-ba26-1f13f973cd57","html_url":"https://github.com/Portkey-AI/gateway","commit_stats":{"total_commits":992,"total_committers":57,"mean_commits":"17.403508771929825","dds":0.5796370967741935,"last_synced_commit":"5d247d970cdad10ad21eaf7a00c445c485e764b3"},"previous_names":["portkey-ai/rubeus"],"tags_count":60,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Portkey-AI%2Fgateway","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Portkey-AI%2Fgateway/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Portkey-AI%2Fgateway/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Portkey-AI%2Fgateway/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Portkey-AI","download_url":"https://codeload.github.com/Portkey-AI/gateway/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253673324,"owners_count":21945586,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-gateway","gateway","generative-ai","hacktoberfest","langchain","llama-index","llmops","llms","openai","prompt-engineering","router"],"created_at":"2024-09-24T20:31:49.455Z","updated_at":"2025-05-13T21:02:54.744Z","avatar_url":"https://github.com/Portkey-AI.png","language":"TypeScript","funding_links":[],"categories":["Prompts","Privacy and Safety"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"/docs/images/gateway-border.png\" width=350\u003e\n\n\u003cp align=\"right\"\u003e\n\u003ca href=\"../README.md\"\u003eEnglish\u003c/a\u003e | \u003cstrong\u003e中文\u003c/strong\u003e | \u003ca href=\"./README.jp.md\"\u003e日本語\u003c/a\u003e\n\u003c/p\u003e\n\n# AI Gateway\n\n### 通过一个快速友好的API链接超过100个大型语言模型。\n\n[![License](https://img.shields.io/github/license/Ileriayo/markdown-badges)](./LICENSE)\n[![Discord](https://img.shields.io/discord/1143393887742861333)](https://portkey.ai/community)\n[![Twitter](https://img.shields.io/twitter/url/https/twitter/follow/portkeyai?style=social\u0026label=Follow%20%40PortkeyAI)](https://twitter.com/portkeyai)\n[![npm version](https://badge.fury.io/js/%40portkey-ai%2Fgateway.svg)](https://www.npmjs.com/package/@portkey-ai/gateway)\n\u003c!-- ![example workflow](https://github.com/github/docs/actions/workflows/main.yml/badge.svg) --\u003e\n\n\u003c/div\u003e\n\u003cbr\u003e\u003cbr\u003e\n\n[Portkey的AI网关](https://portkey.ai/features/ai-gateway) 是您的应用程序与托管的大型语言模型(LLMs)之间的接口。它通过统一的API简化了对OpenAI、Anthropic、Mistral、LLama2、Anyscale、Google Gemini等的API请求。\n\n✅ 极速响应（快9.9倍），占用空间极小（安装后约45kb）\u003cbr\u003e✅ 跨多个模型、提供商和密钥进行**负载均衡**\u003cbr\u003e✅ 通过**备用方案**确保应用程序的稳定性\u003cbr\u003e✅ 默认提供具有指数级备用方案的**自动重试**\u003cbr\u003e✅ 根据需要插入中间件\u003cbr\u003e✅ 经过超过**1000亿令牌**的实战测试\u003cbr\u003e \u003cbr\u003e\n\n## 入门指南\n\n### 安装\n\n如果您熟悉Node.js和`npx`，您可以在本地运行您的私有AI网关。\n\n```\nnpx @portkey-ai/gateway\n```\n\n\u003csup\u003e\n[ 其它部署选项 ]\n\u0026nbsp; \u003ca href=\"https://portkey.wiki/gh-18\"\u003e\u003cimg height=\"12\" width=\"12\" src=\"https://cfassets.portkey.ai/logo/dew-color.svg\" /\u003e Portkey Cloud 官方部署(推荐)\u003c/a\u003e\n\u0026nbsp; \u003ca href=\"../docs/installation-deployments.md#docker\"\u003e\u003cimg height=\"12\" width=\"12\" src=\"https://cdn.simpleicons.org/docker/3776AB\" /\u003e Docker\u003c/a\u003e\n\u0026nbsp; \u003ca href=\"../docs/installation-deployments.md#nodejs-server\"\u003e\u003cimg height=\"12\" width=\"12\" src=\"https://cdn.simpleicons.org/node.js/3776AB\" /\u003e Node.js\u003c/a\u003e\n\u0026nbsp; \u003ca href=\"../docs/installation-deployments.md#cloudflare-workers\"\u003e\u003cimg height=\"12\" width=\"12\" src=\"https://cdn.simpleicons.org/cloudflare/3776AB\" /\u003e Cloudflare\u003c/a\u003e\n\u0026nbsp; \u003ca href=\"../docs/installation-deployments.md#replit\"\u003e\u003cimg height=\"12\" width=\"12\" src=\"https://cdn.simpleicons.org/replit/3776AB\" /\u003e Replit\u003c/a\u003e\n\u0026nbsp; \u003ca href=\"../docs/installation-deployments.md\"\u003e 其它...\u003c/a\u003e\n\n\u003c/sup\u003e\n\u003e 您的AI网关现在运行在 [http://localhost:8787](http://localhost:8787/) 🚀 \u003cbr\u003e\n\n### 使用方法\n\n让我们尝试通过AI网关向OpenAI发起一个**聊天**请求：\n\n```\nbashCopy codecurl '127.0.0.1:8787/v1/chat/completions' \\\n  -H 'x-portkey-provider: openai' \\\n  -H \"Authorization: Bearer $OPENAI_KEY\" \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"messages\": [{\"role\": \"user\",\"content\": \"Say this is test.\"}], \"max_tokens\": 20, \"model\": \"gpt-4\"}'\n```\n\n[支持的SDK完整列表](#supported-sdks)\n\n\u003cbr\u003e\n\n\n## 支持的AI厂商\n\n|| AI厂商 | 支持 | 流式 | 支持的端点 |\n|---|---|---|---|--|\n| \u003cimg src=\"docs/images/openai.png\" width=25 /\u003e| OpenAI | ✅  |✅  | `/completions`, `/chat/completions`,`/embeddings`, `/assistants`, `/threads`, `/runs` |\n| \u003cimg src=\"docs/images/azure.png\" width=25\u003e| Azure OpenAI | ✅  |✅  | `/completions`, `/chat/completions`,`/embeddings` |\n| \u003cimg src=\"docs/images/anyscale.png\" width=25\u003e| Anyscale | ✅   | ✅  | `/chat/completions` |\n| \u003cimg src=\"https://upload.wikimedia.org/wikipedia/commons/2/2d/Google-favicon-2015.png\" width=25\u003e| Google Gemini \u0026 Palm | ✅  |✅  | `/generateMessage`, `/generateText`, `/embedText` |\n| \u003cimg src=\"docs/images/anthropic.png\" width=25\u003e| Anthropic  | ✅  |✅  | `/messages`, `/complete` |\n| \u003cimg src=\"docs/images/cohere.png\" width=25\u003e| Cohere  | ✅  |✅  | `/generate`, `/embed`, `/rerank` |\n| \u003cimg src=\"https://assets-global.website-files.com/64f6f2c0e3f4c5a91c1e823a/654693d569494912cfc0c0d4_favicon.svg\" width=25\u003e| Together AI  | ✅  |✅  | `/chat/completions`, `/completions`, `/inference` |\n| \u003cimg src=\"https://www.perplexity.ai/favicon.svg\" width=25\u003e| Perplexity  | ✅  |✅  | `/chat/completions` |\n| \u003cimg src=\"https://docs.mistral.ai/img/favicon.ico\" width=25\u003e| Mistral  | ✅  |✅  | `/chat/completions`, `/embeddings` |\n\n\u003e [在这里查看支持的100多个模型的完整列表](https://portkey.ai/docs/welcome/what-is-portkey#ai-providers-supported)\n\u003cbr /\u003e\n\n## 特点\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\n      \u003ch4\u003e\u003ca href=\"https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/universal-api\"\u003e统一API签名\u003c/a\u003e\u003c/h4\u003e\n      使用OpenAI的API签名连接100多个LLM。AI网关处理请求、响应和错误转换，因此您无需对代码进行任何更改。您可以使用OpenAI SDK本身连接到任何支持的LLM。\n      \u003cbr\u003e\u003cbr\u003e\n      \u003cimg src=\"docs/images/openai.png\" height=40 /\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u003cimg src=\"docs/images/azure.png\" height=40 /\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\n      \u003cimg src=\"docs/images/anyscale.png\" height=40 /\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\n      \u003cimg src=\"https://upload.wikimedia.org/wikipedia/commons/2/2d/Google-favicon-2015.png\" height=40 /\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u003cbr\u003e\u003cbr\u003e\n      \u003cimg src=\"docs/images/anthropic.png\" height=40 /\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\n      \u003cimg src=\"docs/images/cohere.png\" height=40 /\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\n      \u003cimg src=\"https://assets-global.website-files.com/64f6f2c0e3f4c5a91c1e823a/654693d569494912cfc0c0d4_favicon.svg\" height=40 /\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\u003cbr\u003e\u003cbr\u003e\n      \u003cimg src=\"https://www.perplexity.ai/favicon.svg\" height=40 /\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\n      \u003cimg src=\"https://docs.mistral.ai/img/favicon.ico\" height=40 /\u003e\u0026nbsp;\u0026nbsp;\u0026nbsp;\n      \u003cimg src=\"https://1000logos.net/wp-content/uploads/2021/10/logo-Meta.png\" height=40 /\u003e\n     \u003cbr\u003e\u003cbr\u003e\n    \u003c/td\u003e\n    \u003ctd\u003e\n      \u003ch4\u003e\u003ca href=\"https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/fallbacks\"\u003e备用方案\u003c/a\u003e\u003c/h4\u003e\n      不要让失败阻止您。备用功能允许您按优先顺序指定语言模型API（LLMs）列表。如果主LLM无法响应或遇到错误，Portkey将自动备用到列表中的下一个LLM，确保您的应用程序的稳定性和可靠性。\n      \u003cbr\u003e\u003cbr\u003e\n      \u003cimg src=\"https://framerusercontent.com/images/gmlOW8yeKP2pGuIsObM6gKLzeMI.png\" height=200 /\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\n      \u003ch4\u003e\u003ca href=\"https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/automatic-retries\"\u003e自动重试\u003c/a\u003e\u003c/h4\u003e\n      临时问题不应该意味着手动重新运行。AI网关可以自动重试失败的请求多达5次。我们采用指数退避策略，间隔重试尝试以防止网络过载。\n      \u003cbr\u003e\u003cbr\u003e\n      \u003cimg src=\"https://github.com/roh26it/Rubeus/assets/971978/8a6e653c-94b2-4ba7-95c7-93544ee476b1\" height=200 /\u003e\n    \u003c/td\u003e\n    \u003ctd\u003e\n      \u003ch4\u003e\u003ca href=\"https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/load-balancing\"\u003e负载均衡\u003c/a\u003e\u003c/h4\u003e\n      根据自定义权重在多个API密钥或提供商之间有效分配负载。这确保了您的生成式AI应用程序的高可用性和最佳性能，防止任何单一LLM成为性能瓶颈。\n      \u003cbr\u003e\u003cbr\u003e\n      \u003cimg src=\"https://framerusercontent.com/images/6EWuq3FWhqrPe3kKLqVspevi4.png\" height=200 /\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003cbr\u003e\n\n## 配置 AI 网关\nAI 网关支持[配置](https://portkey.ai/docs/api-reference/config-object)，以实现如**后备（fallbacks）**、**负载均衡（load balancing）**、**重试（retries）**等多样化的路由策略。\n\u003cbr\u003e\u003cbr\u003e\n您可以在通过 `x-portkey-config` 头部进行 OpenAI 调用时使用这些配置\n```js\n// 使用 OpenAI JS SDK\nconst client = new OpenAI({\n  baseURL: \"http://127.0.0.1:8787\", // 网关 URL\n  defaultHeaders: {\n    'x-portkey-config': {.. 你的配置在这里 ..}, \n  }\n});\n```\n\u003cbr\u003e\n\u003cdetails\u003e\u003csummary\u003e这里有一个示例配置，在回退到 Gemini Pro 之前会重试 OpenAI 请求 5 次\u003c/summary\u003e\n\n```js\n{\n  \"retry\": { \"count\": 5 },\n  \"strategy\": { \"mode\": \"fallback\" },\n  \"targets\": [{\n      \"provider\": \"openai\",\n      \"api_key\": \"sk-***\"\n    },{\n      \"provider\": \"google\",\n      \"api_key\": \"gt5***\",\n      \"override_params\": {\"model\": \"gemini-pro\"}\n  }]\n}\n```\n\u003c/details\u003e \u003cdetails\u003e \u003csummary\u003e此配置将使得在 2 个 OpenAI 密钥之间实现等量的负载均衡\u003c/summary\u003e\n\n```js\n{\n  \"strategy\": { \"mode\": \"loadbalance\" },\n  \"targets\": [{\n      \"provider\": \"openai\",\n      \"api_key\": \"sk-***\",\n      \"weight\": \"0.5\"\n    },{\n      \"provider\": \"openai\",\n      \"api_key\": \"sk-***\",\n      \"weight\": \"0.5\"\n    }\n  ]\n}\n```\n\u003c/details\u003e\n了解更多关于配置对象。\n\u003cbr\u003e\n\n## 支持的SDKs\n\n| 语言 | 支持的SDKs |\n|---|---|\n| Node.js / JS / TS | [Portkey SDK](https://www.npmjs.com/package/portkey-ai) \u003cbr\u003e [OpenAI SDK](https://www.npmjs.com/package/openai) \u003cbr\u003e [LangchainJS](https://www.npmjs.com/package/langchain) \u003cbr\u003e [LlamaIndex.TS](https://www.npmjs.com/package/llamaindex) |\n| Python | [Portkey SDK](https://pypi.org/project/portkey-ai/) \u003cbr\u003e [OpenAI SDK](https://pypi.org/project/openai/) \u003cbr\u003e [Langchain](https://pypi.org/project/langchain/) \u003cbr\u003e [LlamaIndex](https://pypi.org/project/llama-index/) |\n| Go | [go-openai](https://github.com/sashabaranov/go-openai) |\n| Java | [openai-java](https://github.com/TheoKanning/openai-java) |\n| Rust | [async-openai](https://docs.rs/async-openai/latest/async_openai/) |\n| Ruby | [ruby-openai](https://github.com/alexrudall/ruby-openai) |\n\n\u003cbr\u003e\n\n\n\n## 部署 AI 网关\n\n[查看文档](docs/installation-deployments.md)了解如何在本地安装 AI 网关或者在流行的平台上部署它。\n\n\u003cbr\u003e\n\n## 路线图\n\n1. 支持更多的提供商。如果缺少某个提供商或 LLM 平台，请[提出功能请求](https://github.com/Portkey-AI/gateway/issues)。\n2. 增强的负载均衡功能，以优化不同模型和提供商之间的资源使用。\n3. 更加健壮的后备和重试策略，以进一步提高请求的可靠性。\n4. 增加统一 API 签名的可定制性，以满足更多样化的使用案例。\n\n[💬 在这里参与路线图讨论。](https://github.com/Portkey-AI/gateway/projects/)\n\n\u003cbr\u003e\n\n## 贡献\n\n最简单的贡献方式是选择任何带有 `good first issue` 标签的问题 💪。在[这里](./CONTRIBUTING.md)阅读贡献指南。\n\n发现 Bug？[在这里提交](https://github.com/Portkey-AI/gateway/issues) | 有功能请求？[在这里提交](https://github.com/Portkey-AI/gateway/issues)\n\n\u003cbr\u003e\n\n## 社区\n\n加入我们不断增长的全球社区，寻求帮助，分享想法，讨论 AI。\n\n- 查看我们的官方[博客](https://portkey.ai/blog)\n- 在 [Discord](https://portkey.ai/community) 上与我们实时交流\n- 在 [Twitter](https://twitter.com/PortkeyAI) 上关注我们\n- 在 [LinkedIn](https://www.linkedin.com/company/portkey-ai/) 上与我们建立联系\n- 阅读日文版文档 [日本語](./README.jp.md)\n\n\u003c!-- - 在 [YouTube](https://www.youtube.com/channel/UCZph50gLNXAh1DpmeX8sBdw) 上访问我们 --\u003e \u003c!-- - 加入我们的 [Dev 社区](https://dev.to/portkeyai) --\u003e \u003c!-- - 在 [Stack Overflow](https://stackoverflow.com/questions/tagged/portkey) 上查看标记为 #portkey 的问题 --\u003e\n\n![Rubeus Social Share (4)](https://github.com/Portkey-AI/gateway/assets/971978/89d6f0af-a95d-4402-b451-14764c40d03f)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fportkey-ai%2Fgateway","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fportkey-ai%2Fgateway","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fportkey-ai%2Fgateway/lists"}