awesome-transformer
A collection of transformer's guides, implementations and variants.
https://github.com/ictnlp/awesome-transformer
Last synced: 5 days ago
JSON representation
-
Papers
-
NMT Basic
- Japanese and Korean Voice Search
- DL4MT
- Sequence to Sequence Learning with Neural Networks
- Neural Machine Translation by Jointly Learning to Align and Translate
- Effective Approaches to Attention-based Neural Machine Translation
- Neural Machine Translation in Linear Time
- Convolutional Sequence to Sequence Learning
- Neural Machine Translation of Rare Words with Subword Units
- Japanese and Korean Voice Search
- A Structured Self-attentive Sentence Embedding
- Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
-
Transformer original paper
-
-
Implementations & How to reproduce paper's result
-
Minimal, paper-equavalent but not certainly performance-reproducable implementations(both *PyTorch* implementations)
-
Complex, performance-reproducable implementations
- tensor2tensor
- OpenNMT-py
- sacrebleu - v13a.pl` but more convenient, to calculate bleu score and report the signature as `BLEU+case.mixed+lang.de-en+test.wmt17 = 32.97 66.1/40.2/26.6/18.1 (BP = 0.980 ratio = 0.980 hyp_len = 63134 ref_len = 64399)` for easy reproduction.
- tensor2tensor transformer.py
- OpenNMT-tf
- Tensor2Tensor for Neural Machine Translation
- OpenNMT-py doc - py opts.py](https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/opts.py)
- corpus preprocessed by OpenNMT - trainingdata/wmt_ende_sp_model.tar.gz). Note that the preprocess procedure includes tokenization, bpe/word-piece operation(here using [sentencepiece](https://github.com/google/sentencepiece) powered by Google which implements word-piece algorithm), see [OpenNMT-tf script](https://github.com/OpenNMT/OpenNMT-tf/blob/master/scripts/wmt/prepare_data.sh) for more details.
- OpenNMT-py FAQ
- OpenNMT-py issue
- OpenNMT: Open-Source Toolkit for Neural Machine Translation
- doc - -update-freq` when training to accumulate every `N` batches loss to backward, so it's `8` for 1 GPU, `2` for 4 GPUs and so on.
- fairseq-py example
- fairseq-py issue
- get_ende_bleu.sh
- transformer_base_multistep8
- t2t issue 539
- t2t issue 444
- t2t issue 317
- the preprocessed WMT'16 EN-DE data provided by Google
- “变形金刚”为何强大：从模型到代码全面解析Google Tensor2Tensor系统
-
Transformer original paper
-
Complex, not certainly performance-reproducable implementations
-
-
Training tips
-
Complex, not certainly performance-reproducable implementations
-
-
Further
-
Complex, not certainly performance-reproducable implementations
- The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation
- Scaling Neural Machine Translation
- Universal Transformer
- Self-Attention with Relative Position Representations
- Improving Language Understanding by Generative Pre-Training
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
-
-
Uncategorized
-
Uncategorized
-
Programming Languages
Categories
Sub Categories