https://github.com/udbhavprasad072300/transformer-implementations

Library - Vanilla, ViT, DeiT, BERT, GPT
https://github.com/udbhavprasad072300/transformer-implementations

pytorch transformer-implementations transformers vision-transformers

Last synced: about 1 month ago
JSON representation

Library - Vanilla, ViT, DeiT, BERT, GPT

• Host: GitHub
• URL: https://github.com/udbhavprasad072300/transformer-implementations
• Owner: UdbhavPrasad072300
• License: mit
• Created: 2020-12-07T06:31:26.000Z (almost 4 years ago)
• Default Branch: main
• Last Pushed: 2021-09-30T22:25:00.000Z (about 3 years ago)
• Last Synced: 2024-09-26T14:24:09.364Z (about 2 months ago)
• Topics: pytorch, transformer-implementations, transformers, vision-transformers
• Language: Jupyter Notebook
• Homepage:
• Size: 3.29 MB
• Stars: 63
• Watchers: 3
• Forks: 18
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

        
# Transformer Implementations



  

        

  

  

        

  

  

        

  

  

        

  



Transformer Implementations and some examples with them

Implemented:



  
Vanilla Transformer

  
ViT - Vision Transformers

  
DeiT - Data efficient image Transformers

  
BERT - Bidirectional Encoder Representations from Transformers

  
GPT - Generative Pre-trained Transformer



## Installation

PyPi

```bash

$ pip install transformer-implementations

```

or

```bash

python setup.py build

python setup.py install

```

## Example

In notebooks directory there is a notebook on how to use each of these models for their intented use; such as image classification for Vision Transformer (ViT) and others.

Check them out!

```python

from transformer_package.models import ViT

image_size = 28 # Model Parameters

channel_size = 1

patch_size = 7

embed_size = 512

num_heads = 8

classes = 10

num_layers = 3

hidden_size = 256

dropout = 0.2

model = ViT(image_size, 

            channel_size, 

            patch_size, 

            embed_size, 

            num_heads, 

            classes, 

            num_layers, 

            hidden_size, 

            dropout=dropout).to(DEVICE)

            

prediction = model(image_tensor)

```

## Language Translation

from "Attention is All You Need": https://arxiv.org/pdf/1706.03762.pdf

Models trained with Implementation:



  
Multi30k - German to English



## Multi-class Image Classification with Vision Transformers (ViT)

from "An Image is Worth 16x16 words: Transformers for image recognition at scale": https://arxiv.org/pdf/2010.11929v1.pdf

Models trained with Implementation:



 
MNIST - Grayscale Images

  
CIFAR10 - MultiChannel Images



Note: ViT will not perform great on small datasets

## Multi-class Image Classification with Data-efficient image Transformers (DeiT)

from "Training data-efficient image transformers & distillation through attention": https://arxiv.org/pdf/2012.12877v1.pdf

Models trained with Implementation:



 
Pretraining Teacher model for Distillation

 
CIFAR10 - Low Res Images