Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/linkedin/Liger-Kernel
Efficient Triton Kernels for LLM Training
https://github.com/linkedin/Liger-Kernel
Last synced: 2 months ago
JSON representation
Efficient Triton Kernels for LLM Training
- Host: GitHub
- URL: https://github.com/linkedin/Liger-Kernel
- Owner: linkedin
- License: bsd-2-clause
- Created: 2024-08-06T17:47:52.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2024-08-26T20:05:45.000Z (2 months ago)
- Last Synced: 2024-08-26T23:54:16.069Z (2 months ago)
- Language: Python
- Homepage:
- Size: 10.6 MB
- Stars: 1,932
- Watchers: 30
- Forks: 81
- Open Issues: 30
Awesome Lists containing this project
- StarryDivineSky - linkedin/Liger-Kernel - Attn 相同的精神，但适用于 RMSNorm、RoPE、SwiGLU 和 CrossEntropy！通过内核融合、就地替换和分块技术，将多 GPU 训练吞吐量提高 20%，并将内存使用量降低 60%。确切：计算是精确的 - 没有近似值！前向和后向传递均通过严格的单元测试实现，并针对没有 Liger 内核的训练运行进行收敛测试，以确保准确性。轻：Liger Kernel 的依赖项最少，只需要 Torch 和 Triton，不需要额外的库！告别依赖性头痛！支持多 GPU：与多 GPU 设置（PyTorch FSDP、DeepSpeed、DDP 等）兼容。Trainer 框架集成：Axolotl、LLaMa-Factory、SFTTrainer、Hugging Face Trainer、SWIFT (文本生成、文本对话 / 大语言对话模型及数据)
- awesome-production-machine-learning - Liger Kernel - Kernel.svg?style=social) - Liger Kernel is a collection of Triton kernels designed specifically for LLM training. (Computation Load Distribution)
- awesome-LLM-resourses - Liger-Kernel