https://github.com/sandeepnmenon/flashattention_tests

Benchmarking different FlashAttention and FlashAttention-v2 implementations
https://github.com/sandeepnmenon/flashattention_tests

Last synced: 24 days ago
JSON representation

Benchmarking different FlashAttention and FlashAttention-v2 implementations

• Host: GitHub
• URL: https://github.com/sandeepnmenon/flashattention_tests
• Owner: sandeepnmenon
• License: mit
• Created: 2023-11-19T02:03:54.000Z (12 months ago)
• Default Branch: master
• Last Pushed: 2023-11-20T22:07:20.000Z (12 months ago)
• Last Synced: 2023-11-21T03:19:58.438Z (12 months ago)
• Language: Jupyter Notebook
• Size: 157 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# FlashAttention_tests

Benchmarking different FlashAttention and FlashAttention-v2 implementations

# Flash-Attention Repo links

Standard Attention: [https://github.com/Dao-AILab/flash-attention/blob/main/flash_attn/modules/mha.py](https://github.com/Dao-AILab/flash-attention/blob/main/flash_attn/modules/mha.py#L175)

Flash Self Attention: [https://github.com/Dao-AILab/flash-attention/blob/main/flash_attn/modules/mha.py](https://github.com/Dao-AILab/flash-attention/blob/main/flash_attn/modules/mha.py#L36)

[Kernel code](https://github.com/Dao-AILab/flash-attention/blob/2c3baba4a63c4007c8a132c5380edc9430f88a22/flash_attn/flash_attn_triton.py#L66C5-L66C16)