{"id":14959000,"url":"https://github.com/lilianweng/transformer-tensorflow","last_synced_at":"2025-04-05T16:06:33.194Z","repository":{"id":34318012,"uuid":"154189397","full_name":"lilianweng/transformer-tensorflow","owner":"lilianweng","description":"Implementation of Transformer Model in Tensorflow","archived":false,"fork":false,"pushed_at":"2023-03-25T00:52:16.000Z","size":23663,"stargazers_count":468,"open_issues_count":6,"forks_count":90,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-03-29T15:05:56.423Z","etag":null,"topics":["attention-is","tensorflow-models","transformer","transformer-encoder"],"latest_commit_sha":null,"homepage":"https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lilianweng.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-22T17:49:18.000Z","updated_at":"2025-03-27T01:59:05.000Z","dependencies_parsed_at":"2024-09-02T16:30:46.760Z","dependency_job_id":null,"html_url":"https://github.com/lilianweng/transformer-tensorflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lilianweng%2Ftransformer-tensorflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lilianweng%2Ftransformer-tensorflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lilianweng%2Ftransformer-tensorflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lilianweng%2Ftransformer-tensorflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lilianweng","download_url":"https://codeload.github.com/lilianweng/transformer-tensorflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247361686,"owners_count":20926643,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention-is","tensorflow-models","transformer","transformer-encoder"],"created_at":"2024-09-24T13:18:40.512Z","updated_at":"2025-04-05T16:06:33.172Z","avatar_url":"https://github.com/lilianweng.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Transformer\nImplementation of the *Transformer* model in the paper:\n\n\u003e Ashish Vaswani, et al. [\"Attention is all you need.\"](http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf) NIPS 2017.\n\n![Transformer model](https://lilianweng.github.io/lil-log/assets/images/transformer.png)\n\nCheck my blog post on attention and transformer:\n* [Attention? Attention!](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html)\n\nImplementations that helped me:\n* https://github.com/Kyubyong/transformer/\n* https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/transformer.py\n* http://nlp.seas.harvard.edu/2018/04/01/attention.html\n\n\n### Setup\n\n```bash\n$ git clone https://github.com/lilianweng/transformer-tensorflow.git\n$ cd transformer-tensorflow\n$ pip install -r requirements.txt\n```\n\n### Train a Model\n\n```bash\n# Check the help message:\n\n$ python train.py --help\n\nUsage: train.py [OPTIONS]\n\nOptions:\n  --seq-len INTEGER               Input sequence length.  [default: 20]\n  --d-model INTEGER               d_model  [default: 512]\n  --d-ff INTEGER                  d_ff  [default: 2048]\n  --n-head INTEGER                n_head  [default: 8]\n  --batch-size INTEGER            Batch size  [default: 128]\n  --max-steps INTEGER             Max train steps.  [default: 300000]\n  --dataset [iwslt15|wmt14|wmt15]\n                                  Which translation dataset to use.  [default:\n                                  iwslt15]\n  --help                          Show this message and exit.\n\n# Train a model on dataset WMT14:\n\n$ python train.py --dataset wmt14\n```\n\n### Evaluate a Trained Model\n\nLet's say, the model is saved in folder `transformer-wmt14-seq20-d512-head8-1541573730` in `checkpoints` folder.\n\n```bash\n$ python eval.py transformer-wmt14-seq20-d512-head8-1541573730\n```\nWith the default config, this implementation gets BLEU ~ 20 on wmt14 test set.\n\n\n### Implementation Notes\n\\[WIP\\] A couple of tricking points in the implementation.\n\n* How to construct the mask correctly?\n* How to correctly shift decoder input (as training input) and decoder target (as ground truth in the loss function)?\n* How to make the prediction in an autoregressive way?\n* Keeping the embedding of `\u003cpad\u003e` as a constant zero vector is sorta important.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flilianweng%2Ftransformer-tensorflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flilianweng%2Ftransformer-tensorflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flilianweng%2Ftransformer-tensorflow/lists"}