https://github.com/pbloem/former

Simple transformer implementation from scratch in pytorch.
https://github.com/pbloem/former

machine-learning pytorch transformer

Last synced: 26 days ago
JSON representation

Simple transformer implementation from scratch in pytorch.

• Host: GitHub
• URL: https://github.com/pbloem/former
• Owner: pbloem
• License: mit
• Created: 2019-08-15T09:04:24.000Z (about 5 years ago)
• Default Branch: master
• Last Pushed: 2024-05-14T09:30:27.000Z (6 months ago)
• Last Synced: 2024-10-01T22:04:52.197Z (about 1 month ago)
• Topics: machine-learning, pytorch, transformer
• Language: Python
• Homepage: http://peterbloem.nl/blog/transformers
• Size: 34.1 MB
• Stars: 1,036
• Watchers: 23
• Forks: 170
• Open Issues: 7
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# former

Simple transformer implementation from scratch in pytorch. See http://peterbloem.nl/blog/transformers for an in-depth explanation.

# Limitations

The models implemented here are designed to show the simplicity of transformer models and self-attention. As such they will not scale as far as the bigger transformers. For that you'll need a number of tricks that complicate the code (see the blog post for details).

All models in the repository consist of a single stack of transformer blocks (that is, no encoder/decoder structures). It turns out that this simple configuration often works best. 

# Installation and use

First, download or clone the repository. Then, in the directory that contains setup.py, run

```

pip install -e . 

```

The switch `-e` ensures that when you edit the code, the installed packaged is also changed. This means that you can, for instance, add print statements to the code to see how it works. 

Then, from the same directory, run:

```

python experiments/classify.py

```

This will run a simple classification experiment on the IMDb dataset.

Hyperparameters are passed as command line arguments. The defaults should work well. The classification data is 

automatically downloaded, and the wikipedia data is included in the repository.

### Requirements

Python 3.6+ is required. The pip command above should install all required packages. You may also need

```pip install future```

depending on the exact python version.

### conda environment

The file ```environment.yml``` describes a complete conda environment with all dependencies. After cloning or downloading the project, you create the environment as follows:

```

conda env create -f environment.yml --name former

conda activate former

```