https://github.com/deepseek-ai/DualPipe
https://github.com/deepseek-ai/DualPipe
Last synced: over 1 year ago
JSON representation
- Host: GitHub
- URL: https://github.com/deepseek-ai/DualPipe
- Owner: deepseek-ai
- License: mit
- Created: 2025-02-26T13:29:57.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-02-27T02:42:34.000Z (over 1 year ago)
- Last Synced: 2025-02-27T03:20:52.981Z (over 1 year ago)
- Language: Python
- Size: 85.9 KB
- Stars: 690
- Watchers: 6
- Forks: 35
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- Awesome-LLM-Inference - **DualPipe** - ai/DualPipe.svg?style=social) |โญ๏ธโญ๏ธ | (๐Contents / ๐DeepSeek/Multi-head Latent Attention(MLA) ([ยฉ๏ธback๐๐ป](#paperlist)))
- awesome-deep-reasoning - 2025.02.27 - DualPipe achieves full overlap of forward and backward computation-communication phases, also reducing pipeline bubbles. (News)
README
# DualPipe
DualPipe is an innovative bidirectional pipeline parallelism algorithm introduced in the [DeepSeek-V3 Technical Report](https://arxiv.org/pdf/2412.19437). It achieves full overlap of forward and backward computation-communication phases, also reducing pipeline bubbles. For detailed information on computation-communication overlap, please refer to the [profile data](https://github.com/deepseek-ai/profile-data).
### Schedules
Example DualPipe scheduling for 8 PP ranks and 20 micro-batches in two directions.
The micro-batches in the reverse direction are symmetric to those in the forward direction, so
we omit their batch ID for illustration simplicity. Two cells enclosed by a shared black border
have mutually overlapped computation and communication
### Pipeline Bubbles and Memory Usage Comparison
| Method | Bubble | Parameter | Activation |
|-------------|---------------------------------|-----------|------------|
| 1F1B | (PP-1)(๐น+๐ต) | 1ร | PP |
| ZB1P | (PP-1)(๐น+๐ต-2๐) | 1ร | PP |
| DualPipe | (PP/2-1)(๐น&๐ต+๐ต-3๐) | 2ร | PP+1 |
๐น denotes the execution time of a forward chunk, ๐ต denotes the execution time of a
full backward chunk, ๐ denotes the execution time of a "backward for weights" chunk, and ๐น&๐ต
denotes the execution time of two mutually overlapped forward and backward chunks.
## Quick Start
The usage is shown in the following example:
```bash
python example.py
```
Note: For real-world applications, you will need to implement a custom `overlapped_forward_backward` method tailored to your specific module.
## Requirements
- PyTorch 2.0 and above
## Developers
DualPipe was created and developed by Jiashi Li and Chengqi Deng and Wenfeng Liang.
## Citation
```bibtex
@misc{deepseekai2024deepseekv3technicalreport,
title={DeepSeek-V3 Technical Report},
author={DeepSeek-AI},
year={2024},
eprint={2412.19437},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2412.19437},
}
```