https://github.com/internlm/xtuner
An efficient, flexible and full-featured toolkit for fine-tuning LLM (InternLM2, Llama3, Phi3, Qwen, Mistral, ...)
https://github.com/internlm/xtuner
agent baichuan chatbot chatglm2 chatglm3 conversational-ai internlm large-language-models llama2 llama3 llava llm llm-training mixtral msagent peft phi3 qwen supervised-finetuning
Last synced: 3 months ago
JSON representation
An efficient, flexible and full-featured toolkit for fine-tuning LLM (InternLM2, Llama3, Phi3, Qwen, Mistral, ...)
- Host: GitHub
- URL: https://github.com/internlm/xtuner
- Owner: InternLM
- License: apache-2.0
- Created: 2023-07-11T03:18:13.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2025-05-07T10:15:05.000Z (9 months ago)
- Last Synced: 2025-05-08T21:15:09.507Z (9 months ago)
- Topics: agent, baichuan, chatbot, chatglm2, chatglm3, conversational-ai, internlm, large-language-models, llama2, llama3, llava, llm, llm-training, mixtral, msagent, peft, phi3, qwen, supervised-finetuning
- Language: Python
- Homepage: https://xtuner.readthedocs.io/zh-cn/latest/
- Size: 2.12 MB
- Stars: 4,532
- Watchers: 35
- Forks: 341
- Open Issues: 249
-
Metadata Files:
- Readme: README.md
- Contributing: .github/CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://github.com/InternLM/xtuner/stargazers)
[](https://github.com/InternLM/xtuner/blob/main/LICENSE)
[](https://pypi.org/project/xtuner/)
[](https://pypi.org/project/xtuner/)
[](https://github.com/InternLM/xtuner/issues)
[](https://github.com/InternLM/xtuner/issues)
đ join us on [](https://cdn.vansin.top/internlm/xtuner.jpg)
[](https://twitter.com/intern_lm)
[](https://discord.gg/xa29JuW87d)
đ Explore our models on
[](https://huggingface.co/xtuner)
[](https://www.modelscope.cn/organization/xtuner)
[](https://openxlab.org.cn/usercenter/xtuner)
[](https://www.wisemodel.cn/organization/xtuner)
English | [įŽäŊä¸æ](README_zh-CN.md)
## đ Speed Benchmark
## đ News
- **\[2025/09\]** XTuner V1 Released! A Next-Generation Training Engine Built for Ultra-Large MoE Models
## đ XTuner V1
XTuner V1 is a next-generation LLM training engine specifically designed for ultra-large-scale MoE models. Unlike traditional 3D parallel training architectures, XTuner V1 is optimized for the mainstream MoE training scenarios prevalent in today's academic research.
### Key Features
**đ Dropless Training**
- **Scalable without complexity:** Train 200B-scale MoE models without expert parallelism; 600B models require only intra-node expert parallelism
- **Optimized parallelism strategy:** Smaller expert parallelism dimension compared to traditional 3D approaches, enabling more efficient Dropless training
**đ Long Sequence Support**
- **Memory-efficient design:** Train 200B MoE models on 64k sequence lengths without sequence parallelism through advanced memory optimization techniques
- **Flexible scaling:** Full support for DeepSpeed Ulysses sequence parallelism with linearly scalable maximum sequence length
- **Robust performance:** Maintains stability despite expert load imbalance during long sequence training
**⥠Superior Efficiency**
- **Massive scale:** Supports MoE training up to 1T parameters
- **Breakthrough performance:** First to achieve FSDP training throughput that surpasses traditional 3D parallel schemes for MoE models above 200B scale
- **Hardware optimization:** Achieves training efficiency on Ascend A3 Supernode that exceeds NVIDIA H800
## đĨ Roadmap
XTuner V1 is committed to continuously improving training efficiency for pre-training, instruction fine-tuning, and reinforcement learning of ultra-large MoE models, with special focus on Ascend NPU optimization.
### đ Training Engine
Our vision is to establish XTuner V1 as a versatile training backend that seamlessly integrates with the broader open-source ecosystem.
| Model | GPU(FP8) | GPU(BF16)| NPU(BF16) |
|------------|-----------|----------|-----------|
| Intern S1 | â
| â
| â
|
| Intern VL | â
| â
| â
|
| Qwen3 Dense| â
| â
| â
|
| Qwen3 MoE | â
| â
| â
|
| GPT OSS | â
| â
| đ§ |
| Deepseek V3| â
| â
| đ§ |
| KIMI K2 | â
| â
| đ§ |
### đ§ Algorithm
The algorithm component is actively evolving. We welcome community contributions - with XTuner V1, scale your algorithms to unprecedented sizes!
**Implemented**
- â
**Multimodal Pre-training** - Full support for vision-language model training
- â
**Multimodal Supervised Fine-tuning** - Optimized for instruction following
- â
[GRPO](https://arxiv.org/pdf/2402.03300) - Group Relative Policy Optimization
**Coming Soon**
- đ [MPO](https://arxiv.org/pdf/2411.10442) - Mixed Preference Optimization
- đ [DAPO](https://arxiv.org/pdf/2503.14476) - Dynamic Sampling Policy Optimization
- đ **Multi-turn Agentic RL** - Advanced agent training capabilities
### ⥠Inference Engine Integration
Seamless deployment with leading inference frameworks:
- [x] LMDeploy
- [ ] vLLM
- [ ] SGLang
### Data Preparation
- You can use [GraphGen](https://github.com/open-sciencelab/GraphGen) to create synthetic data for fine-tuning.
## đ¤ Contributing
We appreciate all contributions to XTuner. Please refer to [CONTRIBUTING.md](.github/CONTRIBUTING.md) for the contributing guideline.
## đ Acknowledgement
The development of XTuner V1's training engine has been greatly inspired by and built upon the excellent work of the open-source community. We extend our sincere gratitude to the following pioneering projects:
**Training Engine:**
- [Torchtitan](https://github.com/pytorch/torchtitan) - A PyTorch native platform for training generative AI models
- [Deepspeed](https://github.com/deepspeedai/DeepSpeed) - Microsoft's deep learning optimization library
- [MindSpeed](https://gitee.com/ascend/MindSpeed) - Ascend's high-performance training acceleration library
- [Megatron](https://github.com/NVIDIA/Megatron-LM) - NVIDIA's large-scale transformer training framework
**Reinforcement Learning:**
XTuner V1's reinforcement learning capabilities have been enhanced through insights and best practices from:
- [veRL](https://github.com/volcengine/verl) - Volcano Engine Reinforcement Learning for LLMs
- [SLIME](https://github.com/THUDM/slime) - THU's scalable RLHF implementation
- [AReal](https://github.com/inclusionAI/AReaL) - Ant Reasoning Reinforcement Learning for LLMs
- [OpenRLHF](https://github.com/OpenRLHF/OpenRLHF) - An Easy-to-use, Scalable and High-performance RLHF Framework based on Ray
We are deeply grateful to all contributors and maintainers of these projects for advancing the field of large-scale model training.
## đī¸ Citation
```bibtex
@misc{2023xtuner,
title={XTuner: A Toolkit for Efficiently Fine-tuning LLM},
author={XTuner Contributors},
howpublished = {\url{https://github.com/InternLM/xtuner}},
year={2023}
}
```
## License
This project is released under the [Apache License 2.0](LICENSE). Please also adhere to the Licenses of models and datasets being used.