https://github.com/proger/hippogriff
Griffin MQA + Hawk Linear RNN Hybrid
https://github.com/proger/hippogriff
Last synced: 5 months ago
JSON representation
Griffin MQA + Hawk Linear RNN Hybrid
- Host: GitHub
- URL: https://github.com/proger/hippogriff
- Owner: proger
- License: mit
- Created: 2024-03-01T11:25:07.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-04-26T15:48:44.000Z (about 2 years ago)
- Last Synced: 2024-04-26T20:40:56.518Z (about 2 years ago)
- Language: Python
- Homepage: https://pypi.org/project/hippogriff
- Size: 1.17 MB
- Stars: 72
- Watchers: 4
- Forks: 5
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Hippogriff
[](https://pypi.python.org/pypi/hippogriff)
This repository implements Hawk and Griffin blocks from [Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models](https://arxiv.org/abs/2402.19427v1) using [Accelerated Scan](https://github.com/proger/accelerated-scan) and [Flash Attention](https://github.com/Dao-AILab/flash-attention) for PyTorch.
```
pip install hippogriff
```
