https://github.com/lucidrains/metaformer-gpt
Implementation of Metaformer, but in an autoregressive manner
https://github.com/lucidrains/metaformer-gpt
artificial-intelligence autoregressive deep-learning token-mixing
Last synced: 5 months ago
JSON representation
Implementation of Metaformer, but in an autoregressive manner
- Host: GitHub
- URL: https://github.com/lucidrains/metaformer-gpt
- Owner: lucidrains
- License: mit
- Created: 2022-06-18T16:32:13.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2022-06-21T03:36:34.000Z (over 3 years ago)
- Last Synced: 2025-04-12T13:45:27.480Z (6 months ago)
- Topics: artificial-intelligence, autoregressive, deep-learning, token-mixing
- Language: Python
- Homepage:
- Size: 34.3 MB
- Stars: 23
- Watchers: 3
- Forks: 5
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## Metaformer - GPT (wip)
Implementation of Metaformer, but in an autoregressive manner. In particular, they propose simply using mean centering as a way to do token mixing in a parameter-less fashion, alternating with feedforwards.
## Install
```bash
$ pip install metaformer-gpt
```
## Usage
```python
import torch
from metaformer_gpt import MetaformerGPT
gpt = MetaformerGPT(
num_tokens = 256,
dim = 512,
depth = 8
)
ids = torch.randint(0, 256, (1, 1024))
logits = gpt(ids) # (1, 1024, 256)
```
## Citations
```bibtex
@article{Yu2021MetaFormerIA,
title = {MetaFormer is Actually What You Need for Vision},
author = {Weihao Yu and Mi Luo and Pan Zhou and Chenyang Si and Yichen Zhou and Xinchao Wang and Jiashi Feng and Shuicheng Yan},
journal = {ArXiv},
year = {2021},
volume = {abs/2111.11418}
}
```
```bibtex
@misc{woo2022etsformer,
title = {ETSformer: Exponential Smoothing Transformers for Time-series Forecasting},
author = {Gerald Woo and Chenghao Liu and Doyen Sahoo and Akshat Kumar and Steven Hoi},
year = {2022},
eprint = {2202.01381},
archivePrefix = {arXiv},
primaryClass = {cs.LG}
}
```