{"id":13652984,"url":"https://github.com/soskek/attention_is_all_you_need","last_synced_at":"2025-04-09T23:16:37.364Z","repository":{"id":113372111,"uuid":"94322176","full_name":"soskek/attention_is_all_you_need","owner":"soskek","description":"Transformer of \"Attention Is All You Need\" (Vaswani et al. 2017) by Chainer. ","archived":false,"fork":false,"pushed_at":"2017-10-03T00:00:15.000Z","size":354,"stargazers_count":320,"open_issues_count":1,"forks_count":70,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-04-09T23:16:14.745Z","etag":null,"topics":["attention-mechanism","chainer","deep-learning","deep-neural-networks","google","neural-network"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/soskek.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-06-14T11:09:49.000Z","updated_at":"2025-04-08T08:31:00.000Z","dependencies_parsed_at":"2023-06-26T02:00:13.005Z","dependency_job_id":null,"html_url":"https://github.com/soskek/attention_is_all_you_need","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soskek%2Fattention_is_all_you_need","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soskek%2Fattention_is_all_you_need/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soskek%2Fattention_is_all_you_need/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soskek%2Fattention_is_all_you_need/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/soskek","download_url":"https://codeload.github.com/soskek/attention_is_all_you_need/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248125593,"owners_count":21051771,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention-mechanism","chainer","deep-learning","deep-neural-networks","google","neural-network"],"created_at":"2024-08-02T02:01:04.632Z","updated_at":"2025-04-09T23:16:37.346Z","avatar_url":"https://github.com/soskek.png","language":"Jupyter Notebook","funding_links":[],"categories":["Implementations","Jupyter Notebook"],"sub_categories":[],"readme":"# Transformer - Attention Is All You Need\n[Chainer](https://github.com/chainer/chainer/)-based Python implementation of Transformer, an attention-based seq2seq model without convolution and recurrence.  \nIf you want to see the architecture, please see [net.py](https://github.com/soskek/attention_is_all_you_need/blob/master/net.py).\n\nSee \"[Attention Is All You Need](https://arxiv.org/abs/1706.03762)\", Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, arxiv, 2017.\n\nThis repository is partly derived from my [convolutional seq2seq](https://github.com/soskek/convolutional_seq2seq) repo, which is also derived from Chainer's official [seq2seq example](https://github.com/chainer/chainer/tree/seq2seq-europal/examples/seq2seq).\n\n## Requirement\n\n- Python 3.6.0+\n- [Chainer](https://github.com/chainer/chainer/) 2.0.0+\n- [numpy](https://github.com/numpy/numpy) 1.12.1+\n- [cupy](https://github.com/cupy/cupy) 1.0.0+ (if using gpu)\n- nltk\n- progressbar\n- (You can install all through `pip`)\n- and their dependencies\n\n## Prepare Dataset\nYou can use any parallel corpus.  \nFor example, run\n```\nsh download_wmt.sh\n```\nwhich downloads and decompresses [training dataset](http://www.statmt.org/europarl/v7/fr-en.tgz) and [development dataset](http://www.statmt.org/wmt15/dev-v2.tgz) from [WMT](http://www.statmt.org/wmt15/translation-task.html#download)/[europal](http://www.statmt.org/europarl/) into your current directory. These files and their paths are set in training script `train.py` as default.\n\n## How to Run\n```\nPYTHONIOENCODING=utf-8 python -u train.py -g=0 -i DATA_DIR -o SAVE_DIR\n```\n\nDuring training, logs for loss, perplexity, word accuracy and time are printed at a certain internval, in addition to validation tests (perplexity and BLEU for generation) every half epoch. And also, generation test is performed and printed for checking training progress.\n\n### Arguments\n\nSome of them is as follows:\n- `-g`: your gpu id. If cpu, set `-1`.\n- `-i DATA_DIR`, `-s SOURCE`, `-t TARGET`, `-svalid SVALID`, `-tvalid TVALID`:  \n  `DATA_DIR` directory needs to include a pair of training dataset `SOURCE` and `TARGET` with a pair of validation dataset `SVALID` and `TVALID`. Each pair should be parallell corpus with line-by-line sentence alignment.\n- `-o SAVE_DIR`: JSON log report file and a model snapshot will be saved in `SAVE_DIR` directory (if it does not exist, it will be automatically made).\n- `-e`: max epochs of training corpus.\n- `-b`: minibatch size.\n- `-u`: size of units and word embeddings.\n- `-l`: number of layers in both the encoder and the decoder.\n- `--source-vocab`: max size of vocabulary set of source language\n- `--target-vocab`: max size of vocabulary set of target language\n\nPlease see the others by `python train.py -h`.\n\n\n## Note\n\nThis repository does not aim for complete validation of results in the paper, so I have not eagerly confirmed validity of performance. But, I expect my implementation is almost compatible with a model described in the paper. Some differences where I am aware are as follows:  \n- Optimization/training strategy. Detailed information about batchsize, parameter initialization, etc. is unclear in the paper. Additionally, the learning rate proposed in the paper may work only with a large batchsize (e.g. 4000) for deep layer nets. I changed warmup_step to 32000 from 4000, though there is room for improvement. I also changed `relu` into `leaky relu` in feedforward net layers for easy gradient propagation.\n- Vocabulary set, dataset, preprocessing and evaluation. This repo uses a common word-based tokenization, although the paper uses byte-pair encoding. Size of token set also differs. Evaluation (validation) is little unfair and incompatible with one in the paper, e.g., even validation set replaces unknown words to a single \"unk\" token.\n- Beam search is unused in BLEU calculation.\n- Model size. The setting of a model in this repo is one of \"base model\" in the paper, although you can modify some lines for using \"big model\".\n- This code follows some settings used in [tensor2tensor repository](https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/models), which includes a Transformer model. For example, positional encoding used in the repository seems to differ from one in the paper. This code follows the former one.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoskek%2Fattention_is_all_you_need","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoskek%2Fattention_is_all_you_need","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoskek%2Fattention_is_all_you_need/lists"}