{"id":22072400,"url":"https://github.com/vtuber-plan/langport","last_synced_at":"2025-06-30T02:36:35.155Z","repository":{"id":163667227,"uuid":"638989840","full_name":"vtuber-plan/langport","owner":"vtuber-plan","description":"Langport is a language model inference service","archived":false,"fork":false,"pushed_at":"2024-09-09T23:52:24.000Z","size":893,"stargazers_count":93,"open_issues_count":11,"forks_count":12,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-06-23T21:07:37.336Z","etag":null,"topics":["api","chatgpt","chatgpt-api","fauxpilot","langchain","language-model","llama","llama-cpp","llm","openai","tabby"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vtuber-plan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-10T14:22:13.000Z","updated_at":"2025-06-13T04:00:38.000Z","dependencies_parsed_at":"2023-06-09T09:45:50.048Z","dependency_job_id":"6038ad7b-0dde-4d98-bd7d-8f0aedb7b099","html_url":"https://github.com/vtuber-plan/langport","commit_stats":{"total_commits":213,"total_committers":3,"mean_commits":71.0,"dds":0.2300469483568075,"last_synced_commit":"f8d4b6c11a1bafae8601f11b0f5acb4470b3feb6"},"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/vtuber-plan/langport","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vtuber-plan%2Flangport","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vtuber-plan%2Flangport/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vtuber-plan%2Flangport/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vtuber-plan%2Flangport/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vtuber-plan","download_url":"https://codeload.github.com/vtuber-plan/langport/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vtuber-plan%2Flangport/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261573565,"owners_count":23179151,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","chatgpt","chatgpt-api","fauxpilot","langchain","language-model","llama","llama-cpp","llm","openai","tabby"],"created_at":"2024-11-30T21:12:50.693Z","updated_at":"2025-06-30T02:36:35.127Z","avatar_url":"https://github.com/vtuber-plan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# LangPort\n\n\u003ca href=\"https://github.com/vtuber-plan/langport\"\u003e\n  \u003cimg alt=\"GitHub Repo stars\" src=\"https://img.shields.io/github/stars/vtuber-plan/langport?style=social\"\u003e\n\u003c/a\u003e\n\u003ca href=\"https://github.com/vtuber-plan/langport/blob/main/LICENSE\"\u003e\n  \u003cimg alt=\"License\" src=\"https://img.shields.io/github/license/vtuber-plan/langport\"\u003e\n\u003c/a\u003e\n\n![architecture](assets/architecture.jpg)\n\n\u003c/div\u003e\n\nLangPort is a open-source large language model serving platform.\nOur goal is to build a super fast LLM inference service.\n\nThis project is inspired by [lmsys/fastchat](https://github.com/lm-sys/FastChat), we hope that the serving platform is lightweight and fast, but fastchat includes other features such as training and evaluation make it complicated.\n\nThe core features include:\n- Huggingface transformers support.\n- ggml (llama.cpp) support.\n- A distributed serving system for state-of-the-art models.\n- Streaming generation support with various decoding strategies.\n- Batch inference for higher throughput.\n- Support for encoder-only, decoder-only and encoder-decoder models.\n- OpenAI-compatible RESTful APIs.\n- FauxPilot-compatible RESTful APIs.\n- HuggingFace-compatible RESTful APIs.\n- Tabby-compatible RESTful APIs.\n\n## Support Model Architectures\n* LLaMa, LLaMa2, GLM, Bloom, OPT, GPT2, GPT Neo, GPT Big Code and so on.\n\n## Tested Models\n* NingYu, LLaMa, LLaMa2, Vicuna, ChatGLM, ChatGLM2, Falcon, Starcoder, WizardLM, InternLM, OpenBuddy, FireFly, CodeGen, Phoenix, RWKV, StableLM and so on.\n\n## News\n- [2024/01/13] Introduce the `ChatProto`.\n- [2023/08/04] Dynamic batch inference.\n- [2023/07/16] Support int4 quantization.\n- [2023/07/13] Support generation logprobs parameter.\n- [2023/06/18] Add ggml (llama.cpp gpt.cpp starcoder.cpp etc.) worker support.\n- [2023/06/09] Add LLama.cpp worker support.\n- [2023/06/01] Add HuggingFace Bert embedding worker support.\n- [2023/06/01] Add HuggingFace text generation API support.\n- [2023/06/01] Add tabby API support.\n- [2023/05/23] Add chat throughput test script.\n- [2023/05/22] New distributed architecture.\n- [2023/05/14] Batch inference supported.\n- [2023/05/10] Langport project started.\n\n\n## Install\n\n### Method 1: With pip\n\n```bash\npip install langport\n```\n\nor:\n\n```bash\npip install git+https://github.com/vtuber-plan/langport.git \n```\n\nIf you need ggml generation worker, use this command:\n```bash\npip install langport[ggml]\n```\n\nIf you want to use GPU:\n```bash\nCT_CUBLAS=1 pip install langport[ggml]\n```\n\n### Method 2: From source\n\n1. Clone this repository\n```bash\ngit clone https://github.com/vtuber-plan/langport.git\ncd langport\n```\n\n2. Install the Package\n```bash\npip install --upgrade pip\npip install -e .\n```\n\n## Quick start\nIt is simple to start a local chat API service:\n\nFirst, start a worker process in the terminal:\n``` bash\npython -m langport.service.server.generation_worker --port 21001 --model-path \u003cyour model path\u003e\n```\n\nThen, start a API service in another terminal:\n``` bash\npython -m langport.service.gateway.openai_api\n```\nNow, you can use the inference API by openai protocol.\n\n## Start the server\n\nIt is simple to start a single node chat API service:\n``` bash\npython -m langport.service.server.generation_worker --port 21001 --model-path \u003cyour model path\u003e\npython -m langport.service.gateway.openai_api\n```\n\nIf you need a single node embeddings API server:\n```bash\npython -m langport.service.server.embedding_worker --port 21002 --model-path bert-base-chinese --gpus 0 --num-gpus 1\npython -m langport.service.gateway.openai_api --port 8000 --controller-address http://localhost:21002\n```\n\nIf you need the embeddings API or other features, you can deploy a distributed inference cluster:\n``` bash\npython -m langport.service.server.dummy_worker --port 21001\npython -m langport.service.server.generation_worker --model-path \u003cyour model path\u003e --neighbors http://localhost:21001\npython -m langport.service.server.embedding_worker --model-path \u003cyour model path\u003e --neighbors http://localhost:21001\npython -m langport.service.gateway.openai_api --controller-address http://localhost:21001\n```\n\nIn practice, the gateway can connect to any node to distribute inference tasks:\n\n``` bash\npython -m langport.service.server.dummy_worker --port 21001\npython -m langport.service.server.generation_worker --port 21002 --model-path \u003cyour model path\u003e --neighbors http://localhost:21001\npython -m langport.service.server.generation_worker --port 21003 --model-path \u003cyour model path\u003e --neighbors http://localhost:21001 http://localhost:21002\npython -m langport.service.server.generation_worker --port 21004 --model-path \u003cyour model path\u003e --neighbors http://localhost:21001 http://localhost:21003\npython -m langport.service.server.generation_worker --port 21005 --model-path \u003cyour model path\u003e --neighbors http://localhost:21001 http://localhost:21004\npython -m langport.service.gateway.openai_api --controller-address http://localhost:21003 # 21003 is OK!\npython -m langport.service.gateway.openai_api --controller-address http://localhost:21002 # Any worker is also OK!\n```\n\nRun text generation with multi GPUs:\n\n``` bash\npython -m langport.service.server.generation_worker --port 21001 --model-path \u003cyour model path\u003e --gpus 0,1 --num-gpus 2\npython -m langport.service.gateway.openai_api\n```\n\nRun text generation with ggml worker:\n\n```bash\npython -m langport.service.server.ggml_generation_worker --port 21001 --model-path \u003cyour model path\u003e --gpu-layers \u003cnum layer to gpu (resize this for your VRAM)\u003e\n```\n\nRun OpenAI forward server: \n```bash\npython -m langport.service.server.chatgpt_generation_worker --port 21001 --api-url \u003curl\u003e --api-key \u003ckey\u003e\n```\n\n\n## License\n\nlangport is released under the Apache Software License.\n\n\n## See also\n\n- [langport-docs](https://github.com/vtuber-plan/langport/tree/main/docs)\n- [langport-source](https://github.com/vtuber-plan/langport)\n\n\n## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=vtuber-plan/langport\u0026type=Date)](https://star-history.com/#vtuber-plan/langport\u0026Date)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvtuber-plan%2Flangport","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvtuber-plan%2Flangport","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvtuber-plan%2Flangport/lists"}