https://github.com/lucidrains/remixer-pytorch
Implementation of the Remixer Block from the Remixer paper, in Pytorch
https://github.com/lucidrains/remixer-pytorch
artificial-intelligence deep-learning transformers
Last synced: about 1 year ago
JSON representation
Implementation of the Remixer Block from the Remixer paper, in Pytorch
- Host: GitHub
- URL: https://github.com/lucidrains/remixer-pytorch
- Owner: lucidrains
- License: mit
- Created: 2021-09-24T14:58:29.000Z (almost 5 years ago)
- Default Branch: main
- Last Pushed: 2021-09-27T21:52:26.000Z (almost 5 years ago)
- Last Synced: 2025-04-19T07:47:16.949Z (about 1 year ago)
- Topics: artificial-intelligence, deep-learning, transformers
- Language: Python
- Homepage:
- Size: 81.1 KB
- Stars: 35
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## Remixer - Pytorch
Implementation of the Remixer Block from the Remixer paper, in Pytorch. It claims that substituting the feedforwards in transformers with sequence wide mixing followed by multiplication and subtraction leads to better language understanding results.
## Install
```bash
$ pip install remixer-pytorch
```
## Usage
```python
import torch
from remixer_pytorch import RemixerBlock
block = RemixerBlock(
dim = 512,
seq_len = 1024
)
x = torch.randn(1, 1024, 512)
block(x) # (1, 1024, 512)
```
## Citations
```bibtex
@inproceedings{anonymous,
title = {Remixers: A Mixer-Transformer Architecture with Compositional Operators for Natural Language Understanding },
author = {Anonymous},
year = {2021},
url = {https://openreview.net/forum?id=9FHQHJnRtfL}
}
```