https://github.com/lucidrains/fast-transformer-pytorch

Implementation of Fast Transformer in Pytorch
https://github.com/lucidrains/fast-transformer-pytorch

artificial-intelligence attention-mechanism deep-learning transformers

Last synced: over 1 year ago
JSON representation

Implementation of Fast Transformer in Pytorch

• Host: GitHub
• URL: https://github.com/lucidrains/fast-transformer-pytorch
• Owner: lucidrains
• License: mit
• Created: 2021-08-23T14:33:59.000Z (almost 5 years ago)
• Default Branch: main
• Last Pushed: 2021-08-26T14:18:34.000Z (almost 5 years ago)
• Last Synced: 2025-03-27T01:09:38.327Z (over 1 year ago)
• Topics: artificial-intelligence, attention-mechanism, deep-learning, transformers
• Language: Python
• Homepage:
• Size: 63.5 KB
• Stars: 173
• Watchers: 9
• Forks: 16
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          


## Fast Transformer - Pytorch

Implementation of Fast Transformer in Pytorch. This only work as an encoder.

Yannic video

AI Epiphany

## Install

```bash

$ pip install fast-transformer-pytorch

```

## Usage

```python

import torch

from fast_transformer_pytorch import FastTransformer

model = FastTransformer(

    num_tokens = 20000,

    dim = 512,

    depth = 2,

    max_seq_len = 4096,

    absolute_pos_emb = True   # default uses relative positional encoding, but if that isn't working, then turn on absolute positional embedding by setting this to True

)

x = torch.randint(0, 20000, (1, 4096))

mask = torch.ones(1, 4096).bool()

logits = model(x, mask = mask) # (1, 4096, 20000)

```

## Citations

```bibtex

@misc{wu2021fastformer,

    title   = {Fastformer: Additive Attention is All You Need}, 

    author  = {Chuhan Wu and Fangzhao Wu and Tao Qi and Yongfeng Huang},

    year    = {2021},

    eprint  = {2108.09084},

    archivePrefix = {arXiv},

    primaryClass = {cs.CL}

}

```