Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/66RING/tiny-flash-attention
flash attention tutorial written in python, triton, cuda, cutlass
https://github.com/66RING/tiny-flash-attention
Last synced: 24 days ago
JSON representation
flash attention tutorial written in python, triton, cuda, cutlass
- Host: GitHub
- URL: https://github.com/66RING/tiny-flash-attention
- Owner: 66RING
- License: mit
- Created: 2023-12-07T01:49:40.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-01-03T02:48:34.000Z (30 days ago)
- Last Synced: 2025-01-03T03:32:33.042Z (30 days ago)
- Language: Cuda
- Homepage:
- Size: 94.7 KB
- Stars: 228
- Watchers: 4
- Forks: 18
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-cuda-triton-hpc - 66RING/tiny-flash-attention - flash-attention?style=social"/> : [flash attention](https://github.com/Dao-AILab/flash-attention) tutorial written in python, triton, cuda, cutlass. (Frameworks)
README
# Tiny FlashAttention
WIP
A tiny [flash attention](https://github.com/Dao-AILab/flash-attention) implement in python, rust, cuda and c for learning purpose.
- [python version](#flash-attention-2)
* [x] [naive pure python code](./flash_attention_py/tiny_flash_attn.py)
- [triton version](#triton-flash-attention-2)
* [x] [triton code](./flash_attention_py/tiny_flash_attn_triton.py)
- [c version]
* [x] [naive pure c code](./flash_attention_c/csrc/attn.cpp)
* [x] [naive cuda code standalone](./flash_attention_cuda/standalone_src)
* [x] [naive cuda code python binding](./flash_attention_cutlass/csrc/flash_attention.cu)
* [x] [cutlass cuda code](./flash_attention_cutlass/csrc/flash_attention.cu)
- [rust version]## cutlass cute flash attention in action
my env: cutlass v3.4, torch 1.14, cuda 12.4
- [en tutorial](./cutlass_cute_tutorial_en.md)
- [zh tutorial](./cutlass_cute_tutorial_zh.md)