https://github.com/willguimont/transformers

Flexible transformer implementation for research
https://github.com/willguimont/transformers

deep-learning deep-neural-networks learning machine-learning pytorch transformer transformer-architecture

Last synced: about 12 hours ago
JSON representation

Flexible transformer implementation for research

• Host: GitHub
• URL: https://github.com/willguimont/transformers
• Owner: willGuimont
• License: mit
• Created: 2023-03-23T15:51:47.000Z (almost 2 years ago)
• Default Branch: master
• Last Pushed: 2025-02-12T19:41:57.000Z (3 days ago)
• Last Synced: 2025-02-12T20:25:29.727Z (3 days ago)
• Topics: deep-learning, deep-neural-networks, learning, machine-learning, pytorch, transformer, transformer-architecture
• Language: Python
• Homepage:
• Size: 107 KB
• Stars: 5
• Watchers: 1
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE
  • Citation: CITATION.cff

Awesome Lists containing this project

README

        
# transformers

Collection of easy to understand transformer-based models in PyTorch.

The implementation is heavily commented and should be easy to follow.

## Installation

```bash

pip install git+https://github.com/willGuimont/transformers

```

## Implemented models

General:

- Transformer (Vaswani et al., 2017)

- Parallel Transformer (Dehghani et al., 2023)

- PerceiverIO (Jaegle et al., 2022)

Positional encoding:

- Sinusoidal positional encoding

- Relative positional encoding

- Learnable positional encoding

- Learnable Fourier positional encoding (Li, 2021)

Vision:

- VisionTransformer (Dosovitskiy et al., 2021)

NLP:

- Simple character-level Transformer language model

## Next steps

- VICReg

- Rotary positional encoding https://arxiv.org/pdf/2104.09864.pdf

- Optimizing Deeper Transformers on Small Datasets (Xu et al., 2021)

- Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau et al., 2016)

- Universal Transformers [Paper](https://arxiv.org/abs/2310.07096)

- Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles (Ryali et al., 2023)

- Swin Transformer (Liu et al., 2021)

- DINO: Emerging Properties in Self-Supervised Vision Transformers (Caron et al., 2021)

- FlashAttention (Dao et al., 2022)

- DETR (Carion et al., 2020)

- Unlimiformer: Long-Range Transformers with Unlimited Length Input (Bertsch et al., 2023)

- PointBERT (Yu et al., 2022)

- Hydra Attention: Efficient Attention with Many Heads (Bolya et al., 2022)

- Hyena Hierarchy: Towards Larger Convolutional Language Models (Poli et al., 2023)

- Thinking Like Transformers (Weiss et al., 2021)

- Long short-term memory (Schmidhuber, 1997)

- Rethinking Positional Encoding in Language Pre-training (Ke et al., 2021)

## Cite this repository

```

@software{Guimont-Martin_transformer_flexible_and_2023,

    author = {Guimont-Martin, William},

    month = {2},

    title = {{transformer: flexible and easy to understand transformer models}},

    version = {0.1.0},

    year = {2023}

}

```