https://github.com/sustech-data/optimus
https://github.com/sustech-data/optimus
Last synced: 10 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/sustech-data/optimus
- Owner: SUSTech-data
- Created: 2023-09-05T01:51:49.000Z (almost 3 years ago)
- Default Branch: master
- Last Pushed: 2023-10-14T13:46:21.000Z (over 2 years ago)
- Last Synced: 2025-08-18T10:29:23.895Z (11 months ago)
- Language: Python
- Size: 219 KB
- Stars: 5
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Optimus: HuggingFace-Aligned 3D-parallel backend
- flash attention 2 support on training
- flash attention 2 support on left-padding generation with kv cache
- fmha on GQA & MQA
- multi model topology support by mpu context
- more model type for experiment (PPL,RM,...)
TODO:
- GQA & MQA generation (left-padding)
- less model control option
- generator based on non-batch flash attention and self-design cuda fused kernel
- Fixed pipeline model
- KV Cache management by pre-malloc and reuse (pre-calculate)