{"id":26726912,"url":"https://github.com/AlibabaPAI/llumnix","last_synced_at":"2025-03-27T22:04:54.334Z","repository":{"id":247054089,"uuid":"803133426","full_name":"AlibabaPAI/llumnix","owner":"AlibabaPAI","description":"Efficient and easy multi-instance LLM serving","archived":false,"fork":false,"pushed_at":"2025-03-25T12:58:04.000Z","size":3727,"stargazers_count":343,"open_issues_count":4,"forks_count":27,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-03-25T13:44:10.244Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AlibabaPAI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":"docs/Supported_Models.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-20T06:13:59.000Z","updated_at":"2025-03-25T12:49:55.000Z","dependencies_parsed_at":"2024-07-06T10:30:54.882Z","dependency_job_id":"b62b597d-ebc7-442f-a168-482185201bd1","html_url":"https://github.com/AlibabaPAI/llumnix","commit_stats":null,"previous_names":["alibabapai/llumnix"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlibabaPAI%2Fllumnix","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlibabaPAI%2Fllumnix/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlibabaPAI%2Fllumnix/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlibabaPAI%2Fllumnix/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AlibabaPAI","download_url":"https://codeload.github.com/AlibabaPAI/llumnix/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245931863,"owners_count":20695963,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-27T22:02:11.478Z","updated_at":"2025-03-27T22:04:54.293Z","avatar_url":"https://github.com/AlibabaPAI.png","language":"Python","funding_links":[],"categories":["Inference"],"sub_categories":["Inference Engine"],"readme":"\u003ch1 align=\"center\"\u003e\nLlumnix\n\u003c/h1\u003e\n\n\u003ch3 align=\"center\"\u003e\nEfficient and easy \u003ci\u003emulti-instance\u003c/i\u003e LLM serving\n\u003c/h3\u003e\n\n---\n\n## 🔥 Latest News\n\n- [2025.1] We updated vLLM to version v0.6.3.post1.\n- [2024.11] Llumnix v0.1.0 launched!\n- [2024.7] We officially released the first version of Llumnix.\n- [2024.6] We released our OSDI '24 [research paper](https://arxiv.org/abs/2406.03243) on arxiv.\n\n## 🚀 Why Llumnix\n\nLlumnix is a cross-instance request scheduling layer built on top of LLM inference engines such as [vLLM](https://github.com/vllm-project/vllm).\n\nLlumnix provides optimized multi-instance serving performance in terms of:\n\n- *Low latency*\n  - **Reduced time-to-first-token** (TTFT) and queuing delays with less memory fragmentation\n  - **Reduced time-between-tokens** (TBT) and preemption stalls with better load balancing\n- *High throughput*\n  - Integration with state-of-the-art inference engines\n  - Support for techniques like prefill-decode disaggregation\n\nLlumnix achieves this with:\n\n- Dynamic, fine-grained, KV-cache-aware scheduling\n- Continuous **rescheduling** across instances\n  - Enabled by a KV cache migration mechanism with near-zero overhead\n  - Exploited for continuous load balancing, de-fragmentation, and prefill-decode disaggregation\n\nLlumnix is easy to use with:\n\n- Minimal code changes required for vanilla vLLM deployments\n\n- Seamless integration with existing multi-instance deployment platforms\n\n- Fault tolerance, elasticity, and high service availability\n\n- Extensibility to more inference engines and scheduling policies\n\n\n## Getting Started\n\nLlumnix provides two entrypoints `api_server` and `serve` for deploying Llumnix. The `api_server` entrypoint provides a compatible deployment method with the default single-instance vLLM. By contrast, using the `serve` entrypoint, user can easily deploy Llumnix via Ray job submission API.\n\nIf you are already utilizing vLLM for multi-instance LLM serving deployments, simply replace the vLLM serving deployment command `python -m entrypoints.vllm.api_server ...` for each instance with the command provided below:\n```\npython -m llumnix.entrypoints.vllm.api_server \\\n    --host $HOST \\\n    --port $PORT \\\n    ...\n```\nDuring the serving deployment execution, Llumnix will automatically configure itself and serve as the request scheduling layer on top of the multiple vLLM engine instances.\n\nFor deploying Llumnix using the `serve` module, please refer to [QuickStart](./docs/Quickstart.md).\n\n\nVisit our [documentation](./docs/) to get started:\n- [Quick Start](./docs/Quickstart.md)\n- [Supported Models](./docs/Supported_Models.md)\n- [Fault Tolerance](./docs/Fault_Tolerance.md)\n- [Simulator](./docs/Simulator.md)\n- [Prefill-decode Disaggregation](./docs/Prefill-decode_Disaggregation.md)\n\n## Performance\nWe evaluate the performance of the KV-cache-aware load-balancing scheduler and migration mechanism of Llumnix with 16 Qwen2.5-7B instances (each using an A10-24GB GPU) and 16 Llama2-13B instances (each using an A800-80GB GPU).\n\nWe use Poisson distributions with different request rates to generate request arrivals. For the input/output lengths of requests, we use ShareGPT dataset.\n\n\u003cdiv align=center\u003e\n\u003cimg src=\"./docs/v0.1.0_benchmark.png\" align=\"center\" width=80%/\u003e\n\u003c/div\u003e\n\nLlumnix outperforms a simple round-robin scheduler in TTFT (prefill) by up to 6.4x and 12.1x for mean and P99, and 12% for P99 TBT (decode). Llumnix also shows significantly shorter average preemption stalls (by two orders of magnitude).\n\nWith the KV-cache-aware load-balancing scheduler and the migration mechanism, Llumnix also outperforms a simple load balancing scheduler based on queue sizes in TTFT (prefill) by up to 4.6x and 9.1x for mean and P99, and 15% for P99 TBT (decode).\n\n## Roadmap\n\nLlumnix is currently in an alpha stage. Moving forward, we have work items planned including but not limited to:\n\n- Architectural improvement: improving the scalability and efficiency of distributed serving and coordination;\n- Policy optimization: better dispatching, migration, auto-scaling policies;\n- New features: incorporating more inference engine features;\n- Engineering: testing, CI/CD, etc.\n\n## Publication\n\nPlease cite our paper if you use Llumnix in your research:\n\n```bibtex\n@inproceedings{sun2024llumnix,\n  title={Llumnix: Dynamic Scheduling for Large Language Model Serving},\n  author={Biao Sun and Ziming Huang and Hanyu Zhao and Wencong Xiao and Xinyi Zhang and Yong Li and Wei Lin},\n  booktitle={18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24)},\n  year={2024}\n}\n```\n\n## License\n\nLlumnix is licensed under the Apache 2.0 License.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAlibabaPAI%2Fllumnix","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAlibabaPAI%2Fllumnix","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAlibabaPAI%2Fllumnix/lists"}