{"id":13486416,"url":"https://github.com/jadore801120/attention-is-all-you-need-pytorch","last_synced_at":"2026-10-27T20:20:58.765Z","repository":{"id":37689637,"uuid":"94317993","full_name":"jadore801120/attention-is-all-you-need-pytorch","owner":"jadore801120","description":"A PyTorch implementation of the Transformer model in \"Attention is All You Need\".","archived":false,"fork":false,"pushed_at":"2024-04-16T07:27:13.000Z","size":166,"stargazers_count":9821,"open_issues_count":83,"forks_count":2098,"subscribers_count":92,"default_branch":"master","last_synced_at":"2026-10-24T20:16:53.745Z","etag":null,"topics":["attention","attention-is-all-you-need","deep-learning","natural-language-processing","nlp","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jadore801120.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-14T10:15:20.000Z","updated_at":"2026-10-24T18:01:09.000Z","dependencies_parsed_at":"2024-09-22T06:02:09.441Z","dependency_job_id":null,"html_url":"https://github.com/jadore801120/attention-is-all-you-need-pytorch","commit_stats":{"total_commits":183,"total_committers":8,"mean_commits":22.875,"dds":"0.28415300546448086","last_synced_commit":"132907dd272e2cc92e3c10e6c4e783a87ff8893d"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jadore801120/attention-is-all-you-need-pytorch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jadore801120%2Fattention-is-all-you-need-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jadore801120%2Fattention-is-all-you-need-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jadore801120%2Fattention-is-all-you-need-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jadore801120%2Fattention-is-all-you-need-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jadore801120","download_url":"https://codeload.github.com/jadore801120/attention-is-all-you-need-pytorch/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jadore801120%2Fattention-is-all-you-need-pytorch/sbom","scorecard":null,"funding_links":[],"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":345608443,"owners_count":38558909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-10-03T21:59:58.778Z","status":"online","status_checked_at":"2026-10-26T02:00:06.194Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","attention-is-all-you-need","deep-learning","natural-language-processing","nlp","pytorch"],"created_at":"2024-07-31T18:00:45.422Z","updated_at":"2026-10-26T20:19:03.146Z","avatar_url":"https://github.com/jadore801120.png","language":"Python","funding_links":[],"categories":["\u003ca name=\"for-beginner\"\u003eFor Beginners\u003c/a\u003e","Implementations","Implementations \u0026 How to reproduce paper's result","Implementations \u0026 How to reproduce paper's result?","Python","Paper implementations","Paper Implementations","Paper implementations｜论文实现","Research \u0026 Data Analysis"],"sub_categories":["Minimal, paper-equavalent but not certainly performance-reproducable implementations(both *PyTorch* implementations)","Other libraries｜其他库:"],"readme":"# Attention is all you need: A Pytorch Implementation\n\nThis is a PyTorch implementation of the Transformer model in \"[Attention is All You Need](https://arxiv.org/abs/1706.03762)\" (Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, arxiv, 2017). \n\n\nA novel sequence to sequence framework utilizes the **self-attention mechanism**, instead of Convolution operation or Recurrent structure, and achieve the state-of-the-art performance on **WMT 2014 English-to-German translation task**. (2017/06/12)\n\n\u003e The official Tensorflow Implementation can be found in: [tensorflow/tensor2tensor](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/transformer.py).\n\n\u003e To learn more about self-attention mechanism, you could read \"[A Structured Self-attentive Sentence Embedding](https://arxiv.org/abs/1703.03130)\".\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"http://imgur.com/1krF2R6.png\" width=\"250\"\u003e\n\u003c/p\u003e\n\n\nThe project support training and translation with trained model now.\n\nNote that this project is still a work in progress.\n\n**BPE related parts are not yet fully tested.**\n\n\nIf there is any suggestion or error, feel free to fire an issue to let me know. :)\n\n\n# Usage\n\n## WMT'16 Multimodal Translation: de-en\n\nAn example of training for the WMT'16 Multimodal Translation task (http://www.statmt.org/wmt16/multimodal-task.html).\n\n### 0) Download the spacy language model.\n```bash\n# conda install -c conda-forge spacy \npython -m spacy download en\npython -m spacy download de\n```\n\n### 1) Preprocess the data with torchtext and spacy.\n```bash\npython preprocess.py -lang_src de -lang_trg en -share_vocab -save_data m30k_deen_shr.pkl\n```\n\n### 2) Train the model\n```bash\npython train.py -data_pkl m30k_deen_shr.pkl -log m30k_deen_shr -embs_share_weight -proj_share_weight -label_smoothing -output_dir output -b 256 -warmup 128000 -epoch 400\n```\n\n### 3) Test the model\n```bash\npython translate.py -data_pkl m30k_deen_shr.pkl -model trained.chkpt -output prediction.txt\n```\n\n## [(WIP)] WMT'17 Multimodal Translation: de-en w/ BPE \n### 1) Download and preprocess the data with bpe:\n\n\u003e Since the interfaces is not unified, you need to switch the main function call from `main_wo_bpe` to `main`.\n\n```bash\npython preprocess.py -raw_dir /tmp/raw_deen -data_dir ./bpe_deen -save_data bpe_vocab.pkl -codes codes.txt -prefix deen\n```\n\n### 2) Train the model\n```bash\npython train.py -data_pkl ./bpe_deen/bpe_vocab.pkl -train_path ./bpe_deen/deen-train -val_path ./bpe_deen/deen-val -log deen_bpe -embs_share_weight -proj_share_weight -label_smoothing -output_dir output -b 256 -warmup 128000 -epoch 400\n```\n\n### 3) Test the model (not ready)\n- TODO:\n\t- Load vocabulary.\n\t- Perform decoding after the translation.\n---\n# Performance\n## Training\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://i.imgur.com/S2EVtJx.png\" width=\"400\"\u003e\n\u003cimg src=\"https://i.imgur.com/IZQmUKO.png\" width=\"400\"\u003e\n\u003c/p\u003e\n\n- Parameter settings:\n  - batch size 256 \n  - warmup step 4000 \n  - epoch 200 \n  - lr_mul 0.5\n  - label smoothing \n  - do not apply BPE and shared vocabulary\n  - target embedding / pre-softmax linear layer weight sharing. \n \n  \n## Testing \n- coming soon.\n---\n# TODO\n  - Evaluation on the generated text.\n  - Attention weight plot.\n---\n# Acknowledgement\n- The byte pair encoding parts are borrowed from [subword-nmt](https://github.com/rsennrich/subword-nmt/).\n- The project structure, some scripts and the dataset preprocessing steps are heavily borrowed from [OpenNMT/OpenNMT-py](https://github.com/OpenNMT/OpenNMT-py).\n- Thanks for the suggestions from @srush, @iamalbert, @Zessay, @JulesGM, @ZiJianZhao, and @huanghoujing.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjadore801120%2Fattention-is-all-you-need-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjadore801120%2Fattention-is-all-you-need-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjadore801120%2Fattention-is-all-you-need-pytorch/lists"}