https://github.com/proger/hippogriff

Griffin MQA + Hawk Linear RNN Hybrid
https://github.com/proger/hippogriff

Last synced: 5 months ago
JSON representation

Griffin MQA + Hawk Linear RNN Hybrid

• Host: GitHub
• URL: https://github.com/proger/hippogriff
• Owner: proger
• License: mit
• Created: 2024-03-01T11:25:07.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2024-04-26T15:48:44.000Z (about 2 years ago)
• Last Synced: 2024-04-26T20:40:56.518Z (about 2 years ago)
• Language: Python
• Homepage: https://pypi.org/project/hippogriff
• Size: 1.17 MB
• Stars: 72
• Watchers: 4
• Forks: 5
• Open Issues: 4
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Hippogriff

[![PyPI Version](https://img.shields.io/pypi/v/hippogriff.svg)](https://pypi.python.org/pypi/hippogriff)

This repository implements Hawk and Griffin blocks from [Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models](https://arxiv.org/abs/2402.19427v1) using [Accelerated Scan](https://github.com/proger/accelerated-scan) and [Flash Attention](https://github.com/Dao-AILab/flash-attention) for PyTorch.

```

pip install hippogriff

```

![hippogriff](hippogriff.png)