https://github.com/xLLM-AI/xllm
A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
https://github.com/xLLM-AI/xllm
deepseek glm inference inference-engine large-language-models llm-inference qwen
Last synced: about 4 hours ago
JSON representation
A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
- Host: GitHub
- URL: https://github.com/xLLM-AI/xllm
- Owner: xLLM-AI
- License: apache-2.0
- Created: 2025-08-12T13:16:07.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2026-08-03T09:51:30.000Z (3 days ago)
- Last Synced: 2026-08-03T10:05:21.533Z (3 days ago)
- Topics: deepseek, glm, inference, inference-engine, large-language-models, llm-inference, qwen
- Language: C++
- Homepage: https://xllm-ai.com/
- Size: 25.8 MB
- Stars: 1,504
- Watchers: 18
- Forks: 275
- Open Issues: 193
-
Metadata Files:
- Readme: README.md
- Contributing: .github/CONTRIBUTING.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
- Agents: AGENTS.md
- Claude: CLAUDE.md
Awesome Lists containing this project
- awesome-infra-for-ai - xLLM-AI/xllm - xLLM is a high-performance LLM inference engine optimized for diverse AI accelerators, particularly Chinese hardware, focusing on efficient, low-latency, and high-throughput model serving. (Model Serving & Inference / Model Serving Frameworks)
README
[English](./README.md) | [δΈζ](./README_zh.md)
[](https://docs.xllm-ai.com/) [](https://quay.io/repository/jd_xllm/xllm-ai?tab=tags) [](https://opensource.org/licenses/Apache-2.0) [](https://arxiv.org/abs/2510.14686) [](https://deepwiki.com/jd-opensource/xllm)
---------------------
### π’ News
- 2026-07-06: π xLLM is officially donated to the OpenAtom Foundation!
- 2026-06-13: π We day-0 support the [MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3) model, please refer to the [Deployment Document](https://github.com/jd-opensource/xllm/blob/preview/minimax-m3/testspace/run_minimax_m3.sh) for deployment.
- 2026-04-24: π We day-0 support the [DeepSeek-V4](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) model, please refer to the [Deployment Document](https://github.com/jd-opensource/xllm/blob/preview/deepseek-v4-mlu/testspace/run_deepseek_v4.sh) for deployment.
More News
- 2026-02-12: π We day-0 support high-performance inference for the [GLM-5](https://github.com/zai-org/GLM-5) model, please refer to the [Deployment Document](https://github.com/zai-org/GLM-5/blob/main/example/ascend.md) for deployment.
- 2025-12-21: π We day-0 support high-performance inference for the [GLM-4.7](https://github.com/zai-org) model.
- 2025-12-08: π We day-0 support high-performance inference for the [GLM-4.6V](https://github.com/zai-org/GLM-V) model.
- 2025-12-05: π We now support high-performance inference for the [GLM-4.5/GLM-4.6](https://github.com/zai-org/GLM-4.5/blob/main/README_zh.md) series models.
- 2025-12-05: π We now support high-performance inference for the [VLM-R1](https://github.com/om-ai-lab/VLM-R1) model.
- 2025-12-05: π We build hybrid KV cache management based on [Mooncake](https://github.com/kvcache-ai/Mooncake), supporting global KV cache management with intelligent offloading and prefetching.
- 2025-10-16: π We recently have released our [xLLM Technical Report](https://arxiv.org/abs/2510.14686) on arXiv, providing comprehensive technical blueprints and implementation insights.
## Overview
**xLLM** is an **efficient LLM inference framework**, specifically optimized for **Chinese AI accelerators**, enabling enterprise-grade deployment with enhanced efficiency and reduced cost.
## Highlights
* **Top-tier Performance**: Delivers high-throughput, low-latency inference through many advanced features.
* **Mainstream Hardware Support**: Purpose-built and deeply optimized for Chinese AI accelerators.
* **Service-Engine Decoupled Architecture**: Service layer handles scheduling and availability; engine layer handles computation.
* **Enterprise-grade Deployment**: Battle-tested at scale across JD.com's core retail business.
## Hardware Support
| Hardware | Abbreviation | Example | Remark |
| ------------------ | ------------ | ------- | ------------------- |
| Ascend NPU | NPU | A2, A3 | HDK Driver 25.2.0 + |
| Cambricon MLU | MLU | MLU | |
| Moore Threads GPU | MUSA | S5000 | |
| Hygon DCU | DCU | BW1000 | |
| MetaX MACA | MACA | MXC500 | |
| Iluvatar CoreX GPU | ILU | BI150 | |
## Getting Started
* [Quick Start](https://docs.xllm-ai.com/en/getting_started/quick_start/)
* [Launch xLLM](https://docs.xllm-ai.com/en/getting_started/launch_xllm/)
* [Online Service](https://docs.xllm-ai.com/en/getting_started/online_service/)
* [Offline Inference](https://docs.xllm-ai.com/en/getting_started/offline_service/)
* [Supported Models](https://docs.xllm-ai.com/en/supported_models/)
## Community & Support
## Acknowledgment
This project was made possible thanks to the following open-source projects:
- [ScaleLLM](https://github.com/vectorch-ai/ScaleLLM) - xLLM draws inspiration from ScaleLLM's graph construction method and references its runtime execution.
- [Mooncake](https://github.com/kvcache-ai/Mooncake) - Build xLLM hybrid KV cache management based on Mooncake.
- [brpc](https://github.com/apache/brpc) - Build high-performance http service based on brpc.
- [tokenizers-cpp](https://github.com/mlc-ai/tokenizers-cpp) - Build C++ tokenizer based on tokenizers-cpp.
- [safetensors](https://github.com/huggingface/safetensors) - xLLM relies on the C binding safetensors capability.
- [Partial JSON Parser](https://github.com/promplate/partial-json-parser) - Implement xLLM's C++ JSON parser with insights from Python and Go implementations.
- [concurrentqueue](https://github.com/cameron314/concurrentqueue) - A fast multi-producer, multi-consumer lock-free concurrent queue for C++11.
Thanks to the following collaborating university laboratories:
- [THU-MIG](https://ise.thss.tsinghua.edu.cn/mig/projects.html) (School of Software, BNRist, Tsinghua University)
- USTC-Cloudlab (Cloud Computing Lab, University of Science and Technology of China)
- [Beihang-HiPO](https://github.com/buaa-hipo) (Beihang HiPO research group)
- PKU-DS-LAB (Data Structure Laboratory, Peking University)
- PKU-NetSys-LAB (NetSys Lab, Peking University)
- [TJU-TANKLab](https://flashserve.org/) (TANK Lab, Tianjin University)
Thanks to all the following [developers](https://github.com/jd-opensource/xllm/graphs/contributors) who have contributed to xLLM.
## Citation
If you think this repository is helpful to you, welcome to cite us:
```
@article{liu2025xllm,
title={xLLM Technical Report},
author={Liu, Tongxuan and Peng, Tao and Yang, Peijun and Zhao, Xiaoyang and Lu, Xiusheng and Huang, Weizhe and Liu, Zirui and Chen, Xiaoyu and Liang, Zhiwei and Xiong, Jun and others},
journal={arXiv preprint arXiv:2510.14686},
year={2025}
}
```