{"id":13584964,"url":"https://github.com/xusenlinzy/api-for-open-llm","last_synced_at":"2025-04-11T09:29:24.041Z","repository":{"id":168545430,"uuid":"644259800","full_name":"xusenlinzy/api-for-open-llm","owner":"xusenlinzy","description":"Openai style api for open large language models, using LLMs just as chatgpt! Support for LLaMA, LLaMA-2, BLOOM, Falcon, Baichuan, Qwen, Xverse, SqlCoder, CodeLLaMA, ChatGLM, ChatGLM2, ChatGLM3 etc. 开源大模型的统一后端接口","archived":false,"fork":false,"pushed_at":"2024-09-26T07:47:57.000Z","size":18696,"stargazers_count":2427,"open_issues_count":23,"forks_count":278,"subscribers_count":23,"default_branch":"master","last_synced_at":"2025-04-03T10:41:12.361Z","etag":null,"topics":["baichuan","chatglm","code-llama","docker","internlm","langchain","llama","llama2","llms","nlp","openai","qwen","sqlcoder","xverse"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xusenlinzy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-23T06:43:51.000Z","updated_at":"2025-04-02T04:41:02.000Z","dependencies_parsed_at":"2023-06-09T07:46:19.715Z","dependency_job_id":"f0b32f1e-67b7-42af-8360-2eb18e12bd54","html_url":"https://github.com/xusenlinzy/api-for-open-llm","commit_stats":{"total_commits":429,"total_committers":15,"mean_commits":28.6,"dds":0.1398601398601399,"last_synced_commit":"1410ac39e2dcae4462cfff3e8a7e00d2789f2816"},"previous_names":["xusenlinzy/api-for-open-llm"],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xusenlinzy%2Fapi-for-open-llm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xusenlinzy%2Fapi-for-open-llm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xusenlinzy%2Fapi-for-open-llm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xusenlinzy%2Fapi-for-open-llm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xusenlinzy","download_url":"https://codeload.github.com/xusenlinzy/api-for-open-llm/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247824139,"owners_count":21002212,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["baichuan","chatglm","code-llama","docker","internlm","langchain","llama","llama2","llms","nlp","openai","qwen","sqlcoder","xverse"],"created_at":"2024-08-01T15:04:37.874Z","updated_at":"2025-04-11T09:29:24.023Z","avatar_url":"https://github.com/xusenlinzy.png","language":"Python","readme":"# API for Open LLMs\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/xusenlinzy/api-for-open-llm\"\u003e\u003cimg src=\"https://img.shields.io/github/license/xusenlinzy/api-for-open-llm\"\u003e\u003c/a\u003e\n    \u003ca href=\"\"\u003e\u003cimg src=\"https://img.shields.io/badge/python-3.8+-aff.svg\"\u003e\u003c/a\u003e\n    \u003ca href=\"\"\u003e\u003cimg src=\"https://img.shields.io/badge/pytorch-%3E=1.14-red?logo=pytorch\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/xusenlinzy/api-for-open-llm\"\u003e\u003cimg src=\"https://img.shields.io/github/last-commit/xusenlinzy/api-for-open-llm\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/xusenlinzy/api-for-open-llm\"\u003e\u003cimg src=\"https://img.shields.io/github/issues/xusenlinzy/api-for-open-llm?color=9cc\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/xusenlinzy/api-for-open-llm\"\u003e\u003cimg src=\"https://img.shields.io/github/stars/xusenlinzy/api-for-open-llm?color=ccf\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/xusenlinzy/api-for-open-llm\"\u003e\u003cimg src=\"https://img.shields.io/badge/langurage-py-brightgreen?style=flat\u0026color=blue\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\n![llm.png](assets/images/llm.png)\n\u003cdiv align=\"center\"\u003e 图片来自于论文: [A Survey of Large Language Models](https://arxiv.org/pdf/2303.18223.pdf) \u003c/div\u003e\n\n\n## 📢 新闻\n\n+ 【2024.06.13】 支持 `MiniCPM-Llama3-V-2_5` 模型，修改环境变量 `MODEL_NAME=minicpm-v`  `PROMPT_NAME=minicpm-v`  `DTYPE=bfloat16`\n\n\n+ 【2024.06.12】 支持 `GLM-4V` 模型，修改环境变量 `MODEL_NAME=glm-4v`  `PROMPT_NAME=glm-4v`  `DTYPE=bfloat16`， 测试示例见 [glm4v](./tests/glm4v.py)\n\n\n+ 【2024.06.08】 已支持 `QWEN2` 模型，修改环境变量 `MODEL_NAME=qwen2`  `PROMPT_NAME=qwen2`\n\n\n+ 【2024.06.05】 支持 `GLM4` 模型，修改环境变量 `MODEL_NAME=chatglm4`  `PROMPT_NAME=chatglm4`\n\n\n+ 【2024.04.18】 支持 `Code Qwen` 模型，[sql问答demo](https://github.com/xusenlinzy/api-for-open-llm/tree/master/streamlit-demo/streamlit_gallery/components/sql_chat)\n\n\n+ 【2024.04.16】 支持 `Rerank` 重排序模型，[使用方式](./docs/RAG.md)\n\n\n+ 【2024.02.26】 `QWEN1.5` 模型需要修改环境变量 `MODEL_NAME=qwen2`  `PROMPT_NAME=qwen2`\n\n\n更多新闻和历史请转至 [此处](https://github.com/xusenlinzy/api-for-open-llm/blob/master/docs/NEWS.md)\n\n---\n\n**此项目主要内容**\n\n此项目为开源大模型的推理实现统一的后端接口，与 `OpenAI` 的响应保持一致，具有以下特性：\n\n+ ✨ 以 `OpenAI ChatGPT API` 的方式调用各类开源大模型\n\n\n+ 🖨️ 支持流式响应，实现打印机效果\n\n\n+ 📖 实现文本嵌入模型，为文档知识问答提供支持\n\n\n+ 🦜️ 支持大规模语言模型开发工具 [`langchain` ](https://github.com/hwchase17/langchain) 的各类功能\n \n\n+ 🙌 只需要简单的修改环境变量即可将开源模型作为 `chatgpt` 的替代模型，为各类应用提供后端支持\n\n\n+ 🚀 支持加载经过自行训练过的 `lora` 模型\n\n\n+ ⚡ 支持 [vLLM](https://github.com/vllm-project/vllm) 推理加速和处理并发请求\n\n\n## 内容导引\n\n|                                               章节                                                |              描述               |\n|:-----------------------------------------------------------------------------------------------:|:-----------------------------:|\n|               [💁🏻‍♂支持模型](https://github.com/xusenlinzy/api-for-open-llm#-支持模型)                |       此项目支持的开源模型以及简要信息        |\n|       [🚄启动方式](https://github.com/xusenlinzy/api-for-open-llm/blob/master/docs/SCRIPT.md)       |        启动模型的环境配置和启动命令         |\n|   [⚡vLLM启动方式](https://github.com/xusenlinzy/api-for-open-llm/blob/master/docs/VLLM_SCRIPT.md)   |   使用 `vLLM` 启动模型的环境配置和启动命令    |\n|                 [💻调用方式](https://github.com/xusenlinzy/api-for-open-llm#-使用方式)                  |          启动模型之后的调用方式          |\n|         [❓常见问题](https://github.com/xusenlinzy/api-for-open-llm/blob/master/docs/FAQ.md)         |           一些常见问题的回复           |\n\n\n## 🐼 支持模型\n\n**语言模型**\n\n| 模型                                                 | 模型参数大小                           |\n|----------------------------------------------------|----------------------------------|\n| [Baichuan](https://huggingface.co/baichuan-inc)    | 7B/13B                           |\n| [ChatGLM](https://huggingface.co/THUDM)            | 6B                               | \n| [DeepSeek](https://huggingface.co/deepseek-ai)     | 7B/16B/67B/236B                  | \n| [InternLM](https://huggingface.co/internlm)        | 7B/20B                           |\n| [LLaMA](https://github.com/facebookresearch/llama) | 7B/13B/33B/65B                   | \n| [LLaMA-2](https://huggingface.co/meta-llama)       | 7B/13B/70B                       |\n| [LLaMA-3](https://huggingface.co/meta-llama)       | 8B/70B                           | \n| [Qwen](https://huggingface.co/Qwen)                | 1.8B/7B/14B/72B                  | \n| [Qwen1.5](https://huggingface.co/Qwen)             | 0.5B/1.8B/4B/7B/14B/32B/72B/110B | \n| [Qwen2](https://huggingface.co/Qwen)               | 0.5B/1.5B/7B/57B/72B             |\n| [Yi (1/1.5)](https://huggingface.co/01-ai)         | 6B/9B/34B                        |\n\n启动方式详见 [vLLM启动方式](https://github.com/xusenlinzy/api-for-open-llm/blob/master/docs/VLLM_SCRIPT.md)、[transformers启动方式](https://github.com/xusenlinzy/api-for-open-llm/blob/master/docs/SCRIPT.md)\n\n**嵌入模型**\n\n|            模型             |  维度  |                                        权重链接                                         |\n|:-------------------------:|:----:|:-----------------------------------------------------------------------------------:|\n|       bge-large-zh        | 1024 |              [bge-large-zh](https://huggingface.co/BAAI/bge-large-zh)               |\n|         m3e-large         | 1024 |            [moka-ai/m3e-large](https://huggingface.co/moka-ai/m3e-large)            |\n|  text2vec-large-chinese   | 1024 | [text2vec-large-chinese](https://huggingface.co/GanymedeNil/text2vec-large-chinese) |\n| bce-embedding-base_v1（推荐） | 768  | [bce-embedding-base_v1](https://huggingface.co/maidalun1020/bce-embedding-base_v1)  |\n\n\n## 🤖 使用方式\n\n### 环境变量\n\n+ `OPENAI_API_KEY`: 此处随意填一个字符串即可\n\n+ `OPENAI_API_BASE`: 后端启动的接口地址，如：http://192.168.0.xx:80/v1\n\n\n### [聊天界面](./applications)\n\n```shell\ncd streamlit-demo\npip install -r requirements.txt\nstreamlit run streamlit_app.py\n```\n\n![img.png](assets/images/demo.png)\n\n### [openai v1.1.0](https://github.com/openai/openai-python)\n\n\u003cdetails\u003e\n\u003csummary\u003e👉 Chat Completions\u003c/summary\u003e\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    api_key=\"EMPTY\",\n    base_url=\"http://192.168.20.59:7891/v1/\",\n)\n\n# Chat completion API\nchat_completion = client.chat.completions.create(\n    messages=[\n        {\n            \"role\": \"user\",\n            \"content\": \"你好\",\n        }\n    ],\n    model=\"gpt-3.5-turbo\",\n)\nprint(chat_completion)\n# 你好👋！我是人工智能助手 ChatGLM3-6B，很高兴见到你，欢迎问我任何问题。\n\n\n# stream = client.chat.completions.create(\n#     messages=[\n#         {\n#             \"role\": \"user\",\n#             \"content\": \"感冒了怎么办\",\n#         }\n#     ],\n#     model=\"gpt-3.5-turbo\",\n#     stream=True,\n# )\n# for part in stream:\n#     print(part.choices[0].delta.content or \"\", end=\"\", flush=True)\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e👉 Completions\u003c/summary\u003e\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    api_key=\"EMPTY\",\n    base_url=\"http://192.168.20.59:7891/v1/\",\n)\n\n\n# Chat completion API\ncompletion = client.completions.create(\n    model=\"gpt-3.5-turbo\",\n    prompt=\"你好\",\n)\nprint(completion)\n# 你好👋！我是人工智能助手 ChatGLM-6B，很高兴见到你，欢迎问我任何问题。\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e👉 Embeddings\u003c/summary\u003e\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    api_key=\"EMPTY\",\n    base_url=\"http://192.168.20.59:7891/v1/\",\n)\n\n\n# compute the embedding of the text\nembedding = client.embeddings.create(\n    input=\"你好\",\n    model=\"text-embedding-ada-002\"\n)\nprint(embedding)\n\n```\n\n\u003c/details\u003e\n\n\n### 可接入的项目\n\n**通过修改 `OPENAI_API_BASE` 环境变量，大部分的 `chatgpt` 应用和前后端项目都可以无缝衔接！**\n\n+ [ChatGPT-Next-Web: One-Click to deploy well-designed ChatGPT web UI on Vercel](https://github.com/Yidadaa/ChatGPT-Next-Web)\n\n```shell\ndocker run -d -p 3000:3000 \\\n   -e OPENAI_API_KEY=\"sk-xxxx\" \\\n   -e BASE_URL=\"http://192.168.0.xx:80\" \\\n   yidadaa/chatgpt-next-web\n```\n\n![web](assets/images/web.png)\n\n+ [dify: An easy-to-use LLMOps platform designed to empower more people to create sustainable, AI-native applications](https://github.com/langgenius/dify)\n\n```shell\n# 在docker-compose.yml中的api和worker服务中添加以下环境变量\nOPENAI_API_BASE: http://192.168.0.xx:80/v1\nDISABLE_PROVIDER_CONFIG_VALIDATION: 'true'\n```\n\n![dify](assets/images/dify.png)\n\n\n## 📜 License\n\n此项目为 `Apache 2.0` 许可证授权，有关详细信息，请参阅 [LICENSE](LICENSE) 文件。\n\n\n## 🚧 References\n\n+ [ChatGLM: An Open Bilingual Dialogue Language Model](https://github.com/THUDM/ChatGLM-6B)\n\n+ [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://arxiv.org/abs/2211.05100)\n\n+ [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971v1)\n\n+ [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://github.com/ymcui/Chinese-LLaMA-Alpaca)\n\n+ [Phoenix: Democratizing ChatGPT across Languages](https://github.com/FreedomIntelligence/LLMZoo)\n\n+ [MOSS: An open-sourced plugin-augmented conversational language model](https://github.com/OpenLMLab/MOSS)\n\n+ [FastChat: An open platform for training, serving, and evaluating large language model based chatbots](https://github.com/lm-sys/FastChat)\n\n+ [LangChain: Building applications with LLMs through composability](https://github.com/hwchase17/langchain)\n\n+ [ChuanhuChatgpt](https://github.com/GaiZhenbiao/ChuanhuChatGPT)\n\n\n## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=xusenlinzy/api-for-open-llm\u0026type=Date)](https://star-history.com/#xusenlinzy/api-for-open-llm\u0026Date)\n","funding_links":[],"categories":["Langchain","A01_文本生成_文本对话","Python","Building"],"sub_categories":["大语言对话模型及数据","LLM Models"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxusenlinzy%2Fapi-for-open-llm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxusenlinzy%2Fapi-for-open-llm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxusenlinzy%2Fapi-for-open-llm/lists"}