{"id":13596680,"url":"https://github.com/KenyonY/openai-forward","last_synced_at":"2025-04-09T16:33:50.344Z","repository":{"id":149371340,"uuid":"620180763","full_name":"KenyonY/openai-forward","owner":"KenyonY","description":"🚀  大语言模型高效转发服务  · An efficient forwarding service designed for LLMs. ·  OpenAI API Reverse Proxy","archived":false,"fork":false,"pushed_at":"2024-10-07T16:46:09.000Z","size":1993,"stargazers_count":837,"open_issues_count":0,"forks_count":289,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-11-04T18:50:38.395Z","etag":null,"topics":["llm","openai","openai-api","openai-proxy","proxy","python","rate-limit"],"latest_commit_sha":null,"homepage":"https://api.openai-forward.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KenyonY.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-28T07:26:06.000Z","updated_at":"2024-11-04T09:26:33.000Z","dependencies_parsed_at":"2023-10-16T08:43:15.607Z","dependency_job_id":"8cc9f33a-c96a-4a53-9c89-99b6fe8a6ad1","html_url":"https://github.com/KenyonY/openai-forward","commit_stats":null,"previous_names":["kenyony/openai-forward","beidongjiedeguang/openai-forward"],"tags_count":47,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KenyonY%2Fopenai-forward","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KenyonY%2Fopenai-forward/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KenyonY%2Fopenai-forward/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KenyonY%2Fopenai-forward/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KenyonY","download_url":"https://codeload.github.com/KenyonY/openai-forward/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223403702,"owners_count":17139941,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llm","openai","openai-api","openai-proxy","proxy","python","rate-limit"],"created_at":"2024-08-01T16:02:39.365Z","updated_at":"2024-11-06T19:33:35.661Z","avatar_url":"https://github.com/KenyonY.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"**简体中文** | [**English**](https://github.com/KenyonY/openai-forward/blob/main/README_EN.md)\n\n\u003ch1 align=\"center\"\u003e\n    \u003ca href=\"https://github.com/KenyonY/openai-forward\"\u003e  OpenAI Forward \u003c/a\u003e\n    \u003cbr\u003e\n    \u003cbr\u003e\n\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://pypi.org/project/openai-forward/\"\u003e\n        \u003cimg src=\"https://img.shields.io/pypi/v/openai-forward?color=brightgreen\u0026style=flat-square\" alt=\"PyPI version\" \u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/KenyonY/openai-forward/blob/main/LICENSE\"\u003e\n        \u003cimg alt=\"License\" src=\"https://img.shields.io/github/license/KenyonY/openai-forward.svg?color=blue\u0026style=flat-square\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://hub.docker.com/r/beidongjiedeguang/openai-forward\"\u003e\n        \u003cimg alt=\"docker pull\" src=\"https://img.shields.io/docker/pulls/beidongjiedeguang/openai-forward?style=flat-square\u0026label=docker pulls\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/KenyonY/openai-forward/actions/workflows/ci.yml\"\u003e\n        \u003cimg alt=\"tests\" src=\"https://img.shields.io/github/actions/workflow/status/KenyonY/openai-forward/ci.yml?style=flat-square\u0026label=tests\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://pypistats.org/packages/openai-forward\"\u003e\n        \u003cimg alt=\"pypi downloads\" src=\"https://img.shields.io/pypi/dm/openai_forward?style=flat-square\"\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n[特点](#主要特性) |\n[部署指南](deploy.md) |\n[使用指南](#使用指南) |\n[配置](#配置) |\n[对话日志](#对话日志)\n\n\n\u003c/div\u003e\n\n\n\u003e [!IMPORTANT]\n\u003e\n\u003e 在v0.7.0以后在配置方面会有较大调整，并与之前版本不兼容。通过UI配置起来会更加方便，且提供了更强大的配置选项。\n\n\n**OpenAI-Forward** 是为大型语言模型实现的高效转发服务。其核心功能包括\n用户请求速率控制、Token速率限制、智能预测缓存、日志管理和API密钥管理等，旨在提供高效、便捷的模型转发服务。\n无论是代理本地语言模型还是云端语言模型，如 [LocalAI](https://github.com/go-skynet/LocalAI)\n或 [OpenAI](https://api.openai.com)，都可以由 OpenAI Forward 轻松实现。\n得益于 [uvicorn](https://github.com/encode/uvicorn), [aiohttp](https://github.com/aio-libs/aiohttp),\n和 [asyncio](https://docs.python.org/3/library/asyncio.html)\n等库支持，OpenAI-Forward 实现了出色的异步性能。\n\n\n### News\n\n- 🎉🎉🎉 v0.7.0版本后支持通过WebUI进行配置管理\n- gpt-1106版本已适配\n- 缓存后端切换为高性能数据库后端：[🗲 FlaxKV](https://github.com/KenyonY/flaxkv)\n\n\u003ca\u003e\n   \u003cimg src=\"https://raw.githubusercontent.com/KenyonY/openai-forward/main/.github/images/separators/aqua.png\" height=8px width=\"100%\"\u003e\n\u003c/a\u003e\n\n## 主要特性\n\n\n- **全能转发**：可转发几乎所有类型的请求\n- **性能优先**：出色的异步性能\n- **缓存AI预测**：对AI预测进行缓存，加速服务访问并节省费用\n- **用户流量控制**：自定义请求速率与Token速率\n- **实时响应日志**：提升LLMs可观察性\n- **自定义秘钥**：替代原始API密钥\n- **多目标路由**：转发多个服务地址至同一服务下的不同路由\n- **黑白名单**：可对指定IP进行黑白名单限制\n- **自动重试**：确保服务的稳定性，请求失败时将自动重试\n- **快速部署**：支持通过pip和docker在本地或云端进行快速部署\n\n**由本项目搭建的代理服务地址:**\n\n- 原始OpenAI 服务地址\n  \u003e https://api.openai-forward.com  \n  \u003e https://render.openai-forward.com\n\n- 开启缓存的服务地址（用户请求结果将被保存一段时间）\n  \u003e https://smart.openai-forward.com\n\n\u003csub\u003e\n注：此处部署的代理服务仅供个人学习和研究目的使用，勿用于任何商业用途。\n\u003c/sub\u003e\n\n## 部署指南\n\n👉 [部署文档](deploy.md)\n\n\n\u003ca\u003e\n   \u003cimg src=\"https://raw.githubusercontent.com/KenyonY/openai-forward/main/.github/images/separators/aqua.png\" height=8px width=\"100%\"\u003e\n\u003c/a\u003e\n\n## 使用指南\n\n### 快速入门\n\n**安装**\n\n```bash\npip install openai-forward \n\n# 或安装webui版本：\npip install openai-forward[webui]\n```\n\n**启动服务**\n\n```bash\naifd run\n# 或启动带webui的服务\naifd run --webui\n```\n\n如果读入了根路径的`.env`的配置, 将会看到以下启动信息\n\n```bash\n❯ aifd run\n╭────── 🤗 openai-forward is ready to serve!  ───────╮\n│                                                    │\n│  base url         https://api.openai.com           │\n│  route prefix     /                                │\n│  api keys         False                            │\n│  forward keys     False                            │\n│  cache_backend    MEMORY                           │\n╰────────────────────────────────────────────────────╯\n╭──────────── ⏱️ Rate Limit configuration ───────────╮\n│                                                    │\n│  backend                memory                     │\n│  strategy               moving-window              │\n│  global rate limit      100/minute (req)           │\n│  /v1/chat/completions   100/2minutes (req)         │\n│  /v1/completions        60/minute;600/hour (req)   │\n│  /v1/chat/completions   60/second (token)          │\n│  /v1/completions        60/second (token)          │\n╰────────────────────────────────────────────────────╯\nINFO:     Started server process [191471]\nINFO:     Waiting for application startup.\nINFO:     Application startup complete.\nINFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)\n```\n\n### 代理OpenAI模型:\n\n`aifd run`的默认选项便是代理`https://api.openai.com`\n\n下面以搭建好的服务地址`https://api.openai-forward.com` 为例\n\n**Python**\n\n```diff\n  from openai import OpenAI  # pip install openai\u003e=1.0.0\n  client = OpenAI(\n+     base_url=\"https://api.openai-forward.com/v1\", \n      api_key=\"sk-******\"\n  )\n```\n\n\n### 代理本地模型\n\n- **适用场景：** 与 [LocalAI](https://github.com/go-skynet/LocalAI)，\n  [api-for-open-llm](https://github.com/xusenlinzy/api-for-open-llm)等项目一起使用\n\n- **如何操作：**\n  以LocalAI为例，如果已在 http://localhost:8080 部署了LocalAI服务，仅需在环境变量或 .env\n  文件中设置 `FORWARD_CONFIG=[{\"base_url\":\"http://localhost:8080\",\"route\":\"/localai\",\"type\":\"openai\"}]`。\n  然后即可通过访问 http://localhost:8000/localai 使用LocalAI。\n\n(更多)\n\n### 代理任意云端模型\n\n#### 代理[gemini pro](https://ai.google.dev/)\n配置环境变量或 .env 文件如下：\n```env\nFORWARD_CONFIG=[{\"base_url\":\"https://generativelanguage.googleapis.com\",\"route\":\"/gemini\",\"type\":\"general\"}]\n```\n说明：`aidf run`启动后，即可通过访问 http://localhost:8000/gemini 使用gemini pro。\n\n\n- **场景1:**\n  使用通用转发,可对任意来源服务进行转发，\n  可获得请求速率控制与token速率控制；但通用转发不支持自定义秘钥.\n\n- **场景2：**\n  可通过 [LiteLLM](https://github.com/BerriAI/litellm) 可以将 众多云模型的 API 格式转换为 openai\n  的api格式，然后使用openai风格转发\n\n(更多)\n\n\n\u003ca\u003e\n   \u003cimg src=\"https://raw.githubusercontent.com/KenyonY/openai-forward/main/.github/images/separators/aqua.png\" height=8px width=\"100%\"\u003e\n\u003c/a\u003e\n\n## 配置\n\n\n执行 `aifd run --webui` 进入配置页面 (默认服务地址 http://localhost:8001)\n\n\n你可以在项目的运行目录下创建 .env 文件来定制各项配置。参考配置可见根目录下的\n[.env.example](.env.example)文件\n\n\n### 智能缓存\n\n开启缓存后，将会对指定路由的内容进行缓存，其中转发类型分别为`openai`与`general`两者行为略有不同，\n使用`general`转发时，默认会将相同的请求一律使用缓存返回，  \n使用`openai`转发时，在开启缓存后，可以通过OpenAI 的`extra_body`参数来控制缓存的行为，如\n\n**Python**\n\n```diff\n  from openai import OpenAI \n  client = OpenAI(\n+     base_url=\"https://smart.openai-forward.com/v1\", \n      api_key=\"sk-******\"\n  )\n  completion = client.chat.completions.create(\n    model=\"gpt-3.5-turbo\",\n    messages=[\n      {\"role\": \"user\", \"content\": \"Hello!\"}\n    ],\n+   extra_body={\"caching\": True}\n)\n```\n\n**Curl**\n\n```bash\ncurl https://smart.openai.com/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer sk-******\" \\\n  -d '{\n    \"model\": \"gpt-3.5-turbo\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}],\n    \"caching\": true\n  }'\n\n```\n\n### 自定义秘钥\n\n\u003cdetails open\u003e\n  \u003csummary\u003eClick for more details\u003c/summary\u003e\n\n见.env文件\n\n**用例:**\n\n```diff\n  import openai\n+ openai.api_base = \"https://api.openai-forward.com/v1\"\n- openai.api_key = \"sk-******\"\n+ openai.api_key = \"fk-******\"\n```\n\n\u003c/details\u003e\n\n### 多目标服务转发\n\n支持转发不同地址的服务至同一端口的不同路由下\n用例见  `.env.example`\n\n### 对话日志\n\n\u003cdetails open\u003e\n  \u003csummary\u003eClick for more details\u003c/summary\u003e\n\n保存路径在当前目录下的`Log/openai/chat/chat.log`路径中。  \n记录格式为\n\n```text\n{'messages': [{'role': 'user', 'content': 'hi'}], 'model': 'gpt-3.5-turbo', 'stream': True, 'max_tokens': None, 'n': 1, 'temperature': 1, 'top_p': 1, 'logit_bias': None, 'frequency_penalty': 0, 'presence_penalty': 0, 'stop': None, 'user': None, 'ip': '127.0.0.1', 'uid': '2155fe1580e6aed626aa1ad74c1ce54e', 'datetime': '2023-10-17 15:27:12'}\n{'assistant': 'Hello! How can I assist you today?', 'is_tool_calls': False, 'uid': '2155fe1580e6aed626aa1ad74c1ce54e'}\n```\n\n转换为`json`格式：\n\n```bash\naifd convert\n```\n\n得到`chat_openai.json`：\n\n```json\n[\n    {\n        \"datetime\": \"2023-10-17 15:27:12\",\n        \"ip\": \"127.0.0.1\",\n        \"model\": \"gpt-3.5-turbo\",\n        \"temperature\": 1,\n        \"messages\": [\n            {\n                \"user\": \"hi\"\n            }\n        ],\n        \"tools\": null,\n        \"is_tool_calls\": false,\n        \"assistant\": \"Hello! How can I assist you today?\"\n    }\n]\n```\n\n\u003c/details\u003e\n\n## 贡献\n\n欢迎通过提交拉取请求或在仓库中提出问题来为此项目做出贡献。\n\n\n## 许可证\n\nOpenAI-Forward 采用 [MIT](https://opensource.org/license/mit/) 许可证。\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKenyonY%2Fopenai-forward","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FKenyonY%2Fopenai-forward","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKenyonY%2Fopenai-forward/lists"}