{"id":13456701,"url":"https://github.com/ninehills/LLM-Fusion-API","last_synced_at":"2025-03-24T11:31:03.005Z","repository":{"id":184489934,"uuid":"671929056","full_name":"ninehills/LLM-Fusion-API","owner":"ninehills","description":null,"archived":false,"fork":false,"pushed_at":"2024-01-30T03:52:45.000Z","size":32,"stargazers_count":32,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-18T01:40:35.164Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ninehills.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-28T13:21:43.000Z","updated_at":"2024-04-30T01:51:55.000Z","dependencies_parsed_at":"2024-01-30T04:49:41.463Z","dependency_job_id":null,"html_url":"https://github.com/ninehills/LLM-Fusion-API","commit_stats":null,"previous_names":["ninehills/llm-fusion-api"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ninehills%2FLLM-Fusion-API","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ninehills%2FLLM-Fusion-API/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ninehills%2FLLM-Fusion-API/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ninehills%2FLLM-Fusion-API/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ninehills","download_url":"https://codeload.github.com/ninehills/LLM-Fusion-API/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245260771,"owners_count":20586462,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T08:01:26.323Z","updated_at":"2025-03-24T11:31:02.701Z","avatar_url":"https://github.com/ninehills.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# LLM-Fusion-API\n\n## Supported API\n\n- [OpenAI](https://platform.openai.com/docs/api-reference/introduction)\n- [Wenxin](https://cloud.baidu.com/doc/WENXINWORKSHOP/s/flfmc9do2)\n- [FastChat](https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md)\n- [MiniMax](https://api.minimax.chat/)\n- [Zhipu](https://open.bigmodel.cn/doc/api#overview)\n    - Also known as ChatGLM.\n\n## OpenAI API Compatibility\n\n### Chat Completion\n\n| API | system message | function | stream | temperature | top_p | n | stop | max_tokens | presence_penalty | frequency_penalty | logit_bias |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| OpenAI | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |\n| Wenxin | ❌* |❌ | ✔️ | ✔️ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |\n| FastChat | ✔️ | ❌ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ❌ | ❌ | ❌ |\n| MiniMax | ✔️ | ❌ | ✔️ | ✔️ | ✔️ | ✔️ | ❌ | ✔️ | ❌ | ❌ | ❌ |\n| Zhipu | ❌ | ❌ | ✔️ | ✔️ | ✔️ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |\n\n* System messages will be converted into user/assistant message pairs.\n\n### Emebeddings\n\n| API | model | max_tokens |\n| --- | --- | --- |\n| OpenAI | text-embedding-ada-002 | 8191  |\n| Wenxin | embedding-v1 | 384* |\n\n- If the input is longer than 384 tokens, it will be truncated.\n\n## Running the API\n\n```bash\npython3 -m venv .venv\nsource .venv/bin/activate\n\npip install -r requirements.txt\n\ncp .env.example .env\n# edit .env to set OPENAI_API_KEY etc.\n\n# run the API\n./.venv/bin/uvicorn llm_fusion_api:app --reload\n```\n\n## Deploy\n\nMake docker image\n\n```bash\ndocker build -t ninehills/llm-fusion-api:latest .\n```\n\n### Test the API\n\n```txt\n$ curl localhost:8000/v1/models 2\u003e/dev/null| jq \".data[].id\"\n\"gpt-3.5-turbo-16k-0613\"\n\"gpt-3.5-turbo-0301\"\n\"gpt-3.5-turbo-16k\"\n\"gpt-4-0613\"\n\"gpt-4-0314\"\n\"text-embedding-ada-002\"\n\"gpt-4\"\n\"gpt-3.5-turbo-0613\"\n\"gpt-3.5-turbo\"\n\"wenxin/ernie-bot\"\n\"wenxin/ernie-bot-turbo\"\n\"wenxin/bloomz_7b1\"\n\"wenxin/embedding-v1\"\n\"minimax/abab5.5-chat\"\n\"minimax/embo-01\"\n\n$ curl http://localhost:8000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer xxx\" \\\n  -d '{ \"stream\": true,\n    \"model\": \"minimax/abab5.5-chat\",\n    \"messages\": [\n      {\n        \"role\": \"system\",\n        \"content\": \"You are a helpful assistant.\"\n      },\n      {\n        \"role\": \"user\",\n        \"content\": \"Hello!\"\n      }\n    ]\n  }'\ndata: {\"id\": \"abe9437a622c413abd157605efb6e228\", \"object\": \"chat.completion.chunk\", \"created\": 1690690250, \"model\": \"abab5.5-chat\", \"choices\": [{\"index\": 0, \"delta\": {\"role\": \"assistant\", \"content\": \"\"}, \"finish_reason\": null}]}\n\ndata: {\"id\": \"abe9437a622c413abd157605efb6e228\", \"object\": \"chat.completion.chunk\", \"created\": 1690690250, \"model\": \"abab5.5-chat\", \"choices\": [{\"index\": 0, \"delta\": {\"content\": \"Hello! How can I assist you today?\"}, \"finish_reason\": null}]}\n\ndata: {\"id\": \"abe9437a622c413abd157605efb6e228\", \"object\": \"chat.completion.chunk\", \"created\": 1690690250, \"model\": \"abab5.5-chat\", \"choices\": [{\"index\": 0, \"delta\": {\"content\": \"\"}, \"finish_reason\": \"stop\"}]}\n\ndata: [DONE]\n\n$ curl http://localhost:8000/v1/embeddings \\\n  -H \"Authorization: Bearer xxxx\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"input\": \"The food was delicious and the waiter...\",\n    \"model\": \"text-embedding-ada-002\"\n  }'\n\n$ curl http://localhost:8000/v1/engines/text-embedding-ada-002/embeddings \\\n  -H \"Authorization: Bearer xxxx\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"input\": \"The food was delicious and the waiter...\"\n  }'\n\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fninehills%2FLLM-Fusion-API","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fninehills%2FLLM-Fusion-API","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fninehills%2FLLM-Fusion-API/lists"}