https://github.com/armandgiraud/transformerressources

A Repo containing various Ressources to Understand the Transformer model
https://github.com/armandgiraud/transformerressources

Last synced: 6 days ago
JSON representation

A Repo containing various Ressources to Understand the Transformer model

• Host: GitHub
• URL: https://github.com/armandgiraud/transformerressources
• Owner: ArmandGiraud
• Created: 2018-10-14T20:06:30.000Z (about 6 years ago)
• Default Branch: master
• Last Pushed: 2019-04-10T19:52:34.000Z (over 5 years ago)
• Last Synced: 2024-01-17T20:34:20.388Z (10 months ago)
• Language: Jupyter Notebook
• Size: 144 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# TransformerRessources

A Repo containing various Ressources to Understand the Transformer model

## Basics:

* The [Google Research Blog Article](https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html), describing the architecture + some good intuition tips

* [The Tensor2tensor lib ](https://github.com/tensorflow/tensor2tensor#language-modeling) to see the "official" implementation and experiment.

* [The explanation of Attention layer](http://nlp.seas.harvard.edu/2018/04/03/attention.html#Attention) wich are at the core of the model

* [Google Paper](https://arxiv.org/pdf/1803.02155.pdf) explaining the Attention & positional Embeddings.

## Advanced:

* Let's start with [Attention is all you need](https://arxiv.org/abs/1706.03762) The original Paper.

* A great Blog article By Harvard Nlp [Annotated Transformer](nlp.seas.harvard.edu/2018/04/03/attention.html). 

This paper give great Details (with code on the implementation)

* [an excellent explannation of the mathematics](https://staff.fnwi.uva.nl/s.abnar/?p=108) (multi-head attention, self-attention) and where the weights matrices really are

* [illustrated transformer](http://jalammar.github.io/illustrated-transformer/)

## Practical tips & tricks

* A great Article [Training Tips for the Transformer Model](https://arxiv.org/abs/1804.00247)

* a PyTorch implementation of the Transformer for NMT [Git here](https://github.com/huggingface/pytorch-openai-transformer-lm)

*