{"id":2340,"url":"https://github.com/SkyAndCloud/awesome-transformer","name":"awesome-transformer","description":"This repo is not maintained. For latest version, please visit https://github.com/ictnlp. A collection of transformer's guides, implementations and variants.","projects_count":48,"last_synced_at":"2026-08-11T11:00:21.297Z","repository":{"id":50065695,"uuid":"147050686","full_name":"SkyAndCloud/awesome-transformer","owner":"SkyAndCloud","description":"This repo is not maintained. For latest version, please visit https://github.com/ictnlp. A collection of transformer's guides, implementations and variants.","archived":false,"fork":false,"pushed_at":"2018-12-05T14:40:19.000Z","size":49,"stargazers_count":44,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2026-07-23T01:03:50.513Z","etag":null,"topics":["awesome","deep-learning","encoder-decoder","nmt","transformer"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SkyAndCloud.png","metadata":{},"created_at":"2018-09-02T03:18:01.000Z","updated_at":"2025-08-30T22:40:20.000Z","dependencies_parsed_at":"2022-09-16T16:33:43.908Z","dependency_job_id":null,"html_url":"https://github.com/SkyAndCloud/awesome-transformer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SkyAndCloud/awesome-transformer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SkyAndCloud%2Fawesome-transformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SkyAndCloud%2Fawesome-transformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SkyAndCloud%2Fawesome-transformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SkyAndCloud%2Fawesome-transformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SkyAndCloud","download_url":"https://codeload.github.com/SkyAndCloud/awesome-transformer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SkyAndCloud%2Fawesome-transformer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36517983,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-08-06T04:43:03.162Z","status":"online","status_checked_at":"2026-08-11T02:00:06.871Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2024-01-04T20:07:33.054Z","updated_at":"2026-08-11T11:00:21.298Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":["Papers","Further","Implementations \u0026 How to reproduce paper's result?","Training tips","Uncategorized"],"sub_categories":["Transformer original paper","NMT Basic","Complex, not certainly performance-reproducable implementations","Minimal, paper-equavalent but not certainly performance-reproducable implementations(both *PyTorch* implementations)","Complex, performance-reproducable implementations","Uncategorized"],"readme":"\u003cp align=\"center\"\u003e\u003cimg src=\"https://i.loli.net/2018/11/03/5bdd66a85e6d3.jpg\"\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://raw.githubusercontent.com/SkyAndCloud/awesome-transformer/master/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/cocoapods/l/Kingfisher.svg?style=flat\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n**This repo is not maintained. For latest version, please visit [ICTNLP official repo](https://github.com/ictnlp/awesome-transformer)**\n\nA collection of transformer's guides, implementations and so on(For those who want to do some research using transformer as a baseline or simply reproduce paper's performance).\n\nPlease feel free to pull requests or report issues.\n\n* [Why this project?](#why-this-project)\n* [Papers](#papers)\n    * [NMT Basic](#nmt-basic)\n    * [Transformer original paper](#transformer-original-paper)\n* [Implementations \u0026amp; How to reproduce paper's result?](#implementations--how-to-reproduce-papers-result)\n    * [Minimal, paper-equavalent but not certainly performance-reproducable implementations(both \u003cem\u003ePyTorch\u003c/em\u003e implementations)](#minimal-paper-equavalent-but-not-certainly-performance-reproducable-implementationsboth-pytorch-implementations)\n    * [Complex, performance-reproducable implementations](#complex-performance-reproducable-implementations)\n        * \u003ca href=\"#t2t\"\u003ePaper's original implementation: tensor2tensor(using \u003cem\u003eTensorFlow\u003c/em\u003e)]\u003c/a\u003e\n            * [Code](#code)\n            * [Code annotation](#code-annotation)\n            * [Steps to reproduce WMT14 English-German result:](#steps-to-reproduce-wmt14-english-german-result)\n            * [Resources](#resources)\n        * [Harvard NLP Group's implementation: OpenNMT-py(using \u003cem\u003ePyTorch\u003c/em\u003e)](#harvard-nlp-groups-implementation-opennmt-pyusing-pytorch)\n             * [Code](#code-1)\n             * [Steps to reproduce WMT14 English-German result:](#steps-to-reproduce-wmt14-english-german-result-1)\n             * [Resources](#resources-1)\n        * [FAIR's implementation: fairseq-py(using \u003cem\u003ePyTorch\u003c/em\u003e)](#fairs-implementation-fairseq-pyusing-pytorch)\n             * [Code](#code-2)\n             * [Steps to reproduce WMT14 English-German result:](#steps-to-reproduce-wmt14-english-german-result-2)\n             * [Resources](#resources-2)\n    * [Complex, not certainly performance-reproducable implementations](#complex-not-certainly-performance-reproducable-implementations)\n* [Training tips](#training-tips)\n* [Further](#further)\n* [Contributors](#contributors)\n      \n## Why this project?\n\nTransformer is a powerful model applied in sequence to sequence learning. However, when we were using transformer as our baseline in NMT research we found no good \u0026 reliable guide to reproduce approximate result as reported in original paper(even official \u003ca href=\"#t2t\"\u003etensor2tensor\u003c/a\u003e implementation), which means our research would be unauthentic. We collected some implementations, obtained corresponding performance-reproducable approaches and other materials, which eventually formed this project.\n\n## Papers\n\n### NMT Basic\n- seq2seq model: [Sequence to Sequence Learning with Neural Networks](https://arxiv.org/abs/1409.3215)\n- seq2seq \u0026 attention: [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473)\n- refined attention: [Effective Approaches to Attention-based Neural Machine Translation](http://arxiv.org/abs/1508.04025)\n- seq2seq using CGRU: [DL4MT](https://github.com/nyu-dl/dl4mt-tutorial)\n- GNMT: [Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://arxiv.org/abs/1609.08144)\n- bytenet: [Neural Machine Translation in Linear Time](https://arxiv.org/abs/1610.10099)\n- convolutional NMT: [Convolutional Sequence to Sequence Learning](https://arxiv.org/abs/1705.03122)\n- bpe: [Neural Machine Translation of Rare Words with Subword Units](https://arxiv.org/abs/1508.07909)\n- word piece: [Japanese and Korean Voice Search](https://ieeexplore.ieee.org/document/6289079/)\n- self attention paper: [A Structured Self-attentive Sentence Embedding](https://arxiv.org/abs/1703.03130)\n\n### Transformer original paper\n\n\u003cp align=\"center\"\u003e\n\u003ca\u003e\u003cimg width=50% src=\"https://i.loli.net/2018/11/03/5bdd7052d8b63.png\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n- [Attention is All You Need](https://arxiv.org/abs/1706.03762)\n\n## Implementations \u0026 How to reproduce paper's result?\n\nIndeed there are lots of transformer implementations on the Internet, in order to simplify learning curve, here we only include **the most valuable** projects.\n\n\u003e**[Note]**: In transformer original paper, there are *WMT14 English-German*, *WMT14 English-French* two results\n   ![transformer result](https://i.loli.net/2018/11/03/5bdd701614ba1.png)\nHere we regard a implementation as performance-reproducable **if there exists approaches to reproduce WMT14 English-German BLEU score**. Therefore, we'll also support corresponding approach to reproduce *WMT14 English-German* result.\n\n### Minimal, paper-equavalent but not certainly performance-reproducable implementations(both *PyTorch* implementations)\n\n1. attention-is-all-you-need-pytorch\n\n    - [code](https://github.com/jadore801120/attention-is-all-you-need-pytorch)\n\n2. Harvard NLP Group's annotation   \n\n    - [code](http://nlp.seas.harvard.edu/2018/04/03/attention.html)\n\n### Complex, performance-reproducable implementations\n\nBecause transformer's original implementation should run on **8 GPU** to replicate corresponding result, where each GPU loads one batch and after forward propagation 8 batch's loss is summed to execute backward operation, so we can **accumulate every 8 batch's loss** to execute backward operation if we **only have 1 GPU** to imitate this process. **You'd better assemble `gpu_count`, `tokens_on_each_gpu` and `gradient_accumulation_count` to satisfy `gpu_count * tokens_on_each_gpu * gradient_accumulation_count = 4096 * 8`**. See each implementation's guide for details.\n    \nAlthough original paper used `multi-bleu.perl` to evaluate bleu score, we recommend using [sacrebleu](https://github.com/awslabs/sockeye/tree/master/contrib/sacrebleu), which should be equivalent to `mteval-v13a.pl` but more convenient, to calculate bleu score and report the signature as `BLEU+case.mixed+lang.de-en+test.wmt17 = 32.97 66.1/40.2/26.6/18.1 (BP = 0.980 ratio = 0.980 hyp_len = 63134 ref_len = 64399)` for easy reproduction.\n**Note that sacrebleu already has an inner-tokenizer, so the text should be untokenized version.**\n\nThe transformer paper's original model settings can be found in [tensor2tensor transformer.py](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/transformer.py). For example, You can find `base model configs` in`transformer_base` function.\n\nAs you can see, [OpenNMT-tf](https://github.com/OpenNMT/OpenNMT-tf/tree/master/scripts/wmt) also has a replicable instruction but we prefer \u003ca href=\"#t2t\"\u003etensor2tensor\u003c/a\u003e as a baseline to reproduce paper's result if we have to use TensorFlow since it is official.\n\n#### \u003ca id=\"t2t\"/\u003ePaper's original implementation: tensor2tensor(using *TensorFlow*)\n\n##### Code\n\n- [tensor2tensor](https://github.com/tensorflow/tensor2tensor)\n\n##### Code annotation\n\n- [“变形金刚”为何强大：从模型到代码全面解析Google Tensor2Tensor系统](https://cloud.tencent.com/developer/article/1153079)(only Chinese version, corresponding to tensor2tensor v1.6.3)\n\n##### Steps to reproduce WMT14 English-German result: \n\n**updated on v1.10.0**\n\n```shell\n# 1. Install tensor2tensor toolkit\npip install tensor2tensor\n\n# 2. Basic config\n# For BPE model use this problem\nPROBLEM=translate_ende_wmt_bpe32k\nMODEL=transformer\nHPARAMS=transformer_base\n# or use transformer_large to reproduce large model\n# HPARAMS=transformer_large\nDATA_DIR=$HOME/t2t_data\nTMP_DIR=/tmp/t2t_datagen\nTRAIN_DIR=$HOME/t2t_train/$PROBLEM/$MODEL-$HPARAMS\n\nmkdir -p $DATA_DIR $TMP_DIR $TRAIN_DIR\n\n# 3. Download and preprocess corpus\n# Note that tensor2tensor has an inner tokenizer\nt2t-datagen \\\n  --data_dir=$DATA_DIR \\\n  --tmp_dir=$TMP_DIR \\\n  --problem=$PROBLEM\n\n# 4. Train on 8 GPUs. You'll get nearly expected performance after ~250k steps and certainly expected performance after ~500k steps.\nt2t-trainer \\\n  --data_dir=$DATA_DIR \\\n  --problem=$PROBLEM \\\n  --model=$MODEL \\\n  --hparams_set=$HPARAMS \\\n  --output_dir=$TRAIN_DIR \\ \n  --train_steps=600000\n\n# 5. Translate\nDECODE_FILE=$TMP_DIR/newstest2014.tok.bpe.32000.en\nBEAM_SIZE=4\nALPHA=0.6\n\nt2t-decoder \\\n  --data_dir=$DATA_DIR \\\n  --problem=$PROBLEM \\\n  --model=$MODEL \\\n  --hparams_set=$HPARAMS \\\n  --output_dir=$TRAIN_DIR \\\n  --decode_hparams=\"beam_size=$BEAM_SIZE,alpha=$ALPHA\" \\\n  --decode_from_file=$DECODE_FILE \\\n  --decode_to_file=$TMP_DIR/newstest2014.en.tok.32kbpe.transformer_base.beam5.alpha0.6.decode\n\n# 6. Debpe\ncat $TMP_DIR/newstest2014.en.tok.32kbpe.transformer_base.beam5.alpha0.6.decode | sed 's/@@ //g' \u003e $TMP_DIR/newstest2014.en.tok.32kbpe.transformer_base.beam5.alpha0.6.decode.debpe\n#Do compound splitting on the translation\nperl -ple 's{(\\S)-(\\S)}{$1 ##AT##-##AT## $2}g' \u003c $TMP_DIR/newstest2014.en.tok.32kbpe.transformer_base.beam5.alpha0.6.decode.debpe \u003e $TMP_DIR/newstest2014.en.tok.32kbpe.transformer_base.beam5.alpha0.6.decode.debpe.atat\n```\n\n\u003ca id=\"compound_split\"/\u003e**Note that step 6 remains a postprocessing**. For some historical reasons, Google split compound words before getting the final BLEU results which will bring moderate increase. see [get_ende_bleu.sh](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/get_ende_bleu.sh) for more details.\n\nIf you have only 1 GPU, you can use `transformer_base_multistep8` hparams to imitate 8 GPU.\n\n![transformer_base_multistep8](https://i.loli.net/2018/11/03/5bdd6a22ae29a.png)\n \n You can also modify `transformer_base_multistep8` function to accumulate gradient times you want. Here is an example using 4 GPU to run transformer big model. Note that `hparams.optimizer_multistep_accumulate_steps = 2` since we only need to accumulate gradient twice for 4 GPU.\n\n ```python\n@registry.register_hparams\ndef transformer_base_multistep8():\n  \"\"\"HParams for simulating 8 GPUs with MultistepAdam optimizer.\"\"\"\n  hparams = transformer_big()\n  hparams.optimizer = \"MultistepAdam\"\n  hparams.optimizer_multistep_accumulate_steps = 2\n  return hparams\n ```\n\n##### Resources\n- [t2t issue 539](https://github.com/tensorflow/tensor2tensor/issues/539)\n- [t2t issue 444](https://github.com/tensorflow/tensor2tensor/issues/444)\n- [t2t issue 317](https://github.com/tensorflow/tensor2tensor/issues/317)\n- [Tensor2Tensor for Neural Machine Translation](https://arxiv.org/abs/1803.07416)\n\n#### Harvard NLP Group's implementation: OpenNMT-py(using *PyTorch*)\n\n##### Code\n\n- [OpenNMT-py](https://github.com/OpenNMT/OpenNMT-py)\n        \n##### Steps to reproduce WMT14 English-German result:\n\n**updated on v0.5.0**\n\nFor command arguments meaning, see [OpenNMT-py doc](http://opennmt.net/OpenNMT-py/main.html) or [OpenNMT-py opts.py](https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/opts.py)\n\n1. Download [corpus preprocessed by OpenNMT](https://s3.amazonaws.com/opennmt-trainingdata/wmt_ende_sp.tar.gz), [sentencepiece model preprocessed by OpenNMT](https://s3.amazonaws.com/opennmt-trainingdata/wmt_ende_sp_model.tar.gz). Note that the preprocess procedure includes tokenization, bpe/word-piece operation(here using [sentencepiece](https://github.com/google/sentencepiece) powered by Google which implements word-piece algorithm), see [OpenNMT-tf script](https://github.com/OpenNMT/OpenNMT-tf/blob/master/scripts/wmt/prepare_data.sh) for more details.\n        \n2. Preprocess. Because English and German are similar languages here we use `-share_vocab` to share vocabulary between source language and target language, which means you don't need to set this flag for distant language pairs such as Chinese-English. Meanwhile, we use a max sequence length of `100` to cover almostly all sentences on the basis of sentence length distribution of corpus.\n   For example:\n    \n    ```shell\n    python preprocess.py \\\n        -train_src ../wmt-en-de/train.en.shuf \\\n        -train_tgt ../wmt-en-de/train.de.shuf \\\n        -valid_src ../wmt-en-de/valid.en \\\n        -valid_tgt ../wmt-en-de/valid.de \\\n        -save_data ../wmt-en-de/processed \\\n        -src_seq_length 100 \\\n        -tgt_seq_length 100 \\\n        -max_shard_size 200000000 \\\n        -share_vocab\n    ```\n        \n3. Train. For example, if you only have 4 GPU:\n    ```shell\n    python  train.py -data /tmp/de2/data -save_model /tmp/extra \\\n        -layers 6 -rnn_size 512 -word_vec_size 512 -transformer_ff 2048 -heads 8  \\\n        -encoder_type transformer -decoder_type transformer -position_encoding \\\n        -train_steps 200000  -max_generator_batches 2 -dropout 0.1 \\\n        -batch_size 4096 -batch_type tokens -normalization tokens  -accum_count 2 \\\n        -optim adam -adam_beta2 0.998 -decay_method noam -warmup_steps 8000 -learning_rate 2 \\\n        -max_grad_norm 0 -param_init 0  -param_init_glorot \\\n        -label_smoothing 0.1 -valid_steps 10000 -save_checkpoint_steps 10000 \\\n        -world_size 4 -gpu_ranks 0 1 2 3 \n    ```\n    \n    \u003ca id=\"accum_count\"/\u003eNote that here `-accum_count` means every `N` batches accumulating loss to backward, so it's 2 for 4 GPUs and so on.\n        \n4. Translate. For example:  \n   You can set `-batch_size`(default `30`) larger to boost the translation.\n   \n    ```shell\n    python translate.py -gpu 0 -replace_unk -alpha 0.6 -beta 0.0 -beam_size 5 -length_penalty wu -coverage_penalty wu \\\n         -share_vocab vocab_file -max_length 200 -model model_file -src newstest2014.en.32kspe -output model.pred -verbose\n    ```\n        \n    Note that testset in corpus preprocessed by OpenNMT is newstest2017 while it is newstest2014 in original paper, which may be a mistake. To obtain newstest2014 testset as in paper, here we can use sentencepiece to encode `newstest2014.en` manually. You can find `\u003cmodel_file\u003e`in step 1's downloaded archive.\n    \n    ```shell\n    spm_encode --model=\u003cmodel_file\u003e --output_format=piece \u003c newstest2014.en \u003e newstest2014.en.32kspe\n    ```\n        \n5. Detokenization. Since training data is processed by [sentencepiece](https://github.com/google/sentencepiece), step 4's translation should be sentencepiece-encoded style, so we need a decoding procedure to obtain a detokenized plain prediction. \n    For example: \n    \n    ```shell\n    spm_decode --model=\u003cmodel_file\u003e --input_format=piece \u003c input \u003e output\n    ```\n\n6. \u003ca href=\"#compound_split\"\u003ePostprocess\u003c/a\u003e\n\nThere is also a [bpe-version](https://drive.google.com/uc?export=download\u0026id=0B_bZck-ksdkpM25jRUN2X2UxMm8) WMT'16 ENDE corpus preprocessed by Google. See [subword-nmt](https://github.com/rsennrich/subword-nmt) for bpe encoding and decoding.\n\n##### Resources\n\n- [OpenNMT-py FAQ](http://opennmt.net/OpenNMT-py/FAQ.html)\n- ~~[OpenNMT-py issue](https://github.com/OpenNMT/OpenNMT-py/issues/637)(deprecated)~~\n- [OpenNMT: Open-Source Toolkit for Neural Machine Translation](https://arxiv.org/abs/1701.02810)\n\n#### FAIR's implementation: fairseq-py(using *PyTorch*)\n        \n##### Code\n\n- [fairseq-py](https://github.com/pytorch/fairseq/)\n        \n##### Steps to reproduce WMT14 English-German result:\n\n**updated on commit `7e60d45`**\n\n\u003ca id=\"update_freq\"/\u003eFor arguments meaning, see [doc](https://fairseq.readthedocs.io/en/latest/command_line_tools.html). Note that we can use `--update-freq` when training to accumulate every `N` batches loss to backward, so it's `8` for 1 GPU, `2` for 4 GPUs and so on.\n\n1. Download [the preprocessed WMT'16 EN-DE data provided by Google](https://drive.google.com/uc?export=download\u0026id=0B_bZck-ksdkpM25jRUN2X2UxMm8) and extract it.\n\n    ```\n    TEXT=wmt16_en_de_bpe32k\n    mkdir $TEXT\n    tar -xzvf wmt16_en_de.tar.gz -C $TEXT\n    ```\n\n2. Preprocess the dataset with a joined dictionary\n\n    ```\n    python preprocess.py --source-lang en --target-lang de \\\n            --trainpref $TEXT/train.tok.clean.bpe.32000 \\\n            --validpref $TEXT/newstest2013.tok.bpe.32000 \\\n            --testpref $TEXT/newstest2014.tok.bpe.32000 \\\n            --destdir data-bin/wmt16_en_de_bpe32k \\\n            --nwordssrc 32768 --nwordstgt 32768 \\\n            --joined-dictionary\n    ```\n\n3. Train. For a base model.\n\n    ```\n    # train about 180k steps\n    python train.py data-bin/wmt16_en_de_bpe32k \\\n        --arch transformer_vaswani_wmt_en_de --share-all-embeddings \\\n        --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \\\n        --lr-scheduler inverse_sqrt --warmup-init-lr 1e-07 --warmup-updates 4000 \\\n        --lr 0.0007 --min-lr 1e-09 \\\n        --dropout 0.3 --weight-decay 0.0 --criterion label_smoothed_cross_entropy \\ \n        --label-smoothing 0.1 --max-tokens 4096 --update-freq 2 \\\n        --no-progress-bar --log-format json --log-interval 10 --save-interval-updates 1000 \\\n        --keep-interval-updates 5\n    # average last 5 checkpoints\n    modelfile=checkpoints\n    python scripts/average_checkpoints.py --inputs $modelfile --num-update-checkpoints 5 \\\n        --output $modelfile/average-model.pt\n    ```\n\n    For a big model.\n    ```\n    # train about 270k steps\n    python train.py data-bin/wmt16_en_de_bpe32k \\\n        --arch transformer_vaswani_wmt_en_de_big --share-all-embeddings \\\n        --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \\\n        --lr-scheduler inverse_sqrt --warmup-init-lr 1e-07 --warmup-updates 4000 \\\n        --lr 0.0005 --min-lr 1e-09 \\\n        --dropout 0.3 --weight-decay 0.0 --criterion label_smoothed_cross_entropy \\\t\t\n        --label-smoothing 0.1 --max-tokens 4096 --update-freq 2\\\n        --no-progress-bar --log-format json --log-interval 10 --save-interval-updates 1000\\\n        --keep-interval-updates 20\n    # average last 20 checkpoints\n    modelfile=checkpoints\n    python scripts/average_checkpoints.py --inputs $modelfile --num-update-checkpoints 20 \\ \n        --output $modelfile/average-model.pt\n    ```\n\n4. Inference\n    ```\n    model=average-model.pt\n    subset=test\n    python generate.py data-bin/wmt16_en_de_bpe32k --path $modelfile/$model \\\n        --gen-subset $subset --beam 4 --batch-size 128 --remove-bpe --lenpen 0.6 \u003e pred.de\n    # because fairseq's output is unordered, we need to recover its order\n    grep ^H pred.de | cut -f1,3- | cut -c3- | sort -k1n | cut -f2- | tr -d ' ' \u003e pred.de\n    ```\n\n5. \u003ca href=\"#compound_split\"\u003ePostprocess\u003c/a\u003e\n\n##### Resources\n\n- [fairseq-py example](https://github.com/pytorch/fairseq/tree/master/examples/translation)\n- [fairseq-py issue](https://github.com/pytorch/fairseq/issues/202). The corpus problem described in the issue has been fixed now, so we can directly follow the instruction above.\n\n### Complex, not certainly performance-reproducable implementations\n\n- [Marian](https://github.com/marian-nmt/marian-examples/tree/master/transformer)(purely c++ implementation without any deep learning framework)\n\n## Training tips\n\n- [Training Tips for the Transformer Model](https://arxiv.org/abs/1804.00247)\n\n## Further\n\n- RNMT+: [The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation](https://arxiv.org/abs/1804.09849)\n- [Scaling Neural Machine Translation](https://arxiv.org/abs/1806.00187)\n- Turing-complete Transformer: [Universal Transformer](https://arxiv.org/abs/1807.03819)\n- [Self-Attention with Relative Position Representations](https://arxiv.org/abs/1803.02155)\n- [Improving Language Understanding by Generative Pre-Training](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf)\n- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805)\n\n## Contributors\n\nThis project is developed and maintained by Natural Language Processing Group, ICT/CAS.\n\n- [Yong Shan](https://github.com/SkyAndCloud)\n- [Jinchao Zhang](https://github.com/zhangjcqq)\n- Shuhao Gu\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/skyandcloud%2Fawesome-transformer/projects"}