{"id":19074337,"url":"https://github.com/howardyclo/pytorch-seq2seq-example","last_synced_at":"2025-04-28T20:00:17.418Z","repository":{"id":104756654,"uuid":"120180876","full_name":"howardyclo/pytorch-seq2seq-example","owner":"howardyclo","description":"Fully batched seq2seq example based on practical-pytorch, and more extra features.","archived":false,"fork":false,"pushed_at":"2018-03-11T10:31:02.000Z","size":155,"stargazers_count":76,"open_issues_count":4,"forks_count":17,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-18T19:41:01.860Z","etag":null,"topics":["attention-seq2seq","checkpoint","fixed-embedding","glove-embeddings","jupyter-notebook","nlp","nlp-machine-learning","pretrained-embedding","pytorch","pytorch-nlp-tutorial","pytorch-tutorial","seq2seq","shared-embedding","tensorboard","tensorboard-visualization","tie-embedding"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/howardyclo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-04T11:58:23.000Z","updated_at":"2024-01-04T16:20:29.000Z","dependencies_parsed_at":"2023-03-26T17:02:52.616Z","dependency_job_id":null,"html_url":"https://github.com/howardyclo/pytorch-seq2seq-example","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/howardyclo%2Fpytorch-seq2seq-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/howardyclo%2Fpytorch-seq2seq-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/howardyclo%2Fpytorch-seq2seq-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/howardyclo%2Fpytorch-seq2seq-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/howardyclo","download_url":"https://codeload.github.com/howardyclo/pytorch-seq2seq-example/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251380914,"owners_count":21580338,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention-seq2seq","checkpoint","fixed-embedding","glove-embeddings","jupyter-notebook","nlp","nlp-machine-learning","pretrained-embedding","pytorch","pytorch-nlp-tutorial","pytorch-tutorial","seq2seq","shared-embedding","tensorboard","tensorboard-visualization","tie-embedding"],"created_at":"2024-11-09T01:50:36.629Z","updated_at":"2025-04-28T20:00:17.392Z","avatar_url":"https://github.com/howardyclo.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"Batched Seq2Seq Example\nBased on the [`seq2seq-translation-batched.ipynb`](https://github.com/spro/practical-pytorch/blob/master/seq2seq-translation/seq2seq-translation-batched.ipynb) from *practical-pytorch*, but more extra features.\n\nThis example runs grammatical error correction task where the source sequence is a grammatically erroneuous English sentence and the target sequence is an grammatically correct English sentence. The corpus and evaluation script can be download at: https://github.com/keisks/jfleg.\n\n### Extra features\n- Cleaner codebase\n- Very detailed comments for learners\n- Implement Pytorch native dataset and dataloader for batching\n- Correctly handle the hidden state from bidirectional encoder and past to the decoder as initial hidden state.\n- Fully batched attention mechanism computation (only implement `general attention` but it's sufficient). Note: The original code still uses for-loop to compute, which is very slow.\n- Support LSTM instead of only GRU\n- Shared embeddings (encoder's input embedding and decoder's input embedding)\n- Pretrained Glove embedding\n- Fixed embedding\n- Tie embeddings (decoder's input embedding and decoder's output embedding)\n- Tensorboard visualization\n- Load and save checkpoint\n- Replace unknown words by selecting the source token with the highest attention score. (Translation)\n\n### Cons\nComparing to the state-of-the-art seq2seq library, OpenNMT-py, there are some stuffs that aren't optimized in this codebase:\n- Use CuDNN when possible (always on encoder, on decoder when `input_feed`=0)\n- Always avoid indexing / loops and use torch primitives.\n- When possible, batch softmax operations across time. (this is the second complicated part of the code)\n- Batch inference and beam search for translation (this is the most complicated part of the code)\n\n### How to speed up RNN training?\nSeveral ways to speed up RNN training:\n- Batching\n- Static padding\n- Dynamic padding\n- Bucketing\n- Truncated BPTT \n\nSee [\"Sequence Models and the RNN API (TensorFlow Dev Summit 2017)\"](https://www.youtube.com/watch?v=RIR_-Xlbp7s\u0026t=490s) for understanding those techniques.\n\nYou can use [torchtext](http://torchtext.readthedocs.io/en/latest/index.html) or OpenNMT's data iterator for speeding up the training. It can be 7x faster! (ex: 7 hours for an epoch -\u003e 1 hour!)\n\n### Acknowledgement\nThanks to the author of OpenNMT-py @srush for answering the questions for me! See https://github.com/OpenNMT/OpenNMT-py/issues/552\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhowardyclo%2Fpytorch-seq2seq-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhowardyclo%2Fpytorch-seq2seq-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhowardyclo%2Fpytorch-seq2seq-example/lists"}