{"id":18746776,"url":"https://github.com/phohenecker/pytorch-transformer","last_synced_at":"2025-04-12T22:15:22.673Z","repository":{"id":121811332,"uuid":"155567123","full_name":"phohenecker/pytorch-transformer","owner":"phohenecker","description":"A PyTorch implementation of the Transformer model from \"Attention Is All You Need\".","archived":false,"fork":false,"pushed_at":"2019-07-13T19:56:46.000Z","size":61,"stargazers_count":59,"open_issues_count":2,"forks_count":10,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-12T22:15:21.259Z","etag":null,"topics":["attention-is-all-you-need","deep-learning","python3","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/phohenecker.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-31T14:01:58.000Z","updated_at":"2024-09-30T18:59:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"054124fb-8135-4852-a646-83a1ac292739","html_url":"https://github.com/phohenecker/pytorch-transformer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phohenecker%2Fpytorch-transformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phohenecker%2Fpytorch-transformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phohenecker%2Fpytorch-transformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phohenecker%2Fpytorch-transformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/phohenecker","download_url":"https://codeload.github.com/phohenecker/pytorch-transformer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248637796,"owners_count":21137538,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention-is-all-you-need","deep-learning","python3","pytorch"],"created_at":"2024-11-07T16:26:52.752Z","updated_at":"2025-04-12T22:15:22.666Z","avatar_url":"https://github.com/phohenecker.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"pytorch-transformer\n===================\n\n\nThis repository provides a PyTorch implementation of the *Transformer* model that has been introduced in the paper\n*Attention Is All You Need* (Vaswani et al. 2017).\n\n\nInstallation\n------------\n\nThe easiest way to install this package is via pip:\n\n```bash\npip install git+https://github.com/phohenecker/pytorch-transformer\n```\n\n\nUsage\n-----\n\n```python\nimport transformer\nmodel = transformer.Transformer(...)\n```\n\n##### 1. Computing Predictions given a Target Sequence\n\nThis is the default behaviour of a\n[`Transformer`](src/main/python/transformer/transformer.py),\nand is implemented in its\n[`forward`](src/main/python/transformer/transformer.py#L205)\nmethod:\n```python\npredictions = model(input_seq, target_seq)\n```\n\n\n##### 2. Evaluating the Probability of a Target Sequence\n\nThe probability of an output sequence given an input sequence under an already trained model can be evaluated by means\nof the function\n[`eval_probability`](src/main/python/transformer/transformer_tools.py#L46):\n```python\nprobabilities = transformer.eval_probability(model, input_seq, target_seq, pad_index=...)\n```\n\n##### 3. Sampling an Output Sequence\n\nSampling a random output given an input sequence under the distribution computed by a model is realized by the function\n[`sample_output`](src/main/python/transformer/transformer_tools.py#L115):\n\n```python\noutput_seq = transformer.sample_output(model, input_seq, eos_index, pad_index, max_len)\n```\n\n\nPretraining Encoders with BERT\n------------------------------\n\nFor pretraining the encoder part of the transformer\n(i.e.,[`transformer.Encoder`](src/main/python/transformer/encoder.py))\nwith BERT (Devlin et al., 2018), the class [`MLMLoss`](src/main/python/transformer/bert/mlm_loss.py) provides an\nimplementation of the masked language-model loss function.\nA full example of how to implement pretraining with BERT can be found in\n[`examples/bert_pretraining.py`](examples/bert_pretraining.py).\n\n\nReferences\n----------\n\n\u003e Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., Polosukhin, I. (2017).\n\u003e Attention Is All You Need.  \n\u003e Preprint at http://arxiv.org/abs/1706.03762.\n\n\u003e Devlin, J., Chang, M.-W., Lee, K., \u0026 Toutanova, K. (2018).  \n\u003e BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  \n\u003e Preprint at http://arxiv.org/abs/1810.04805.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphohenecker%2Fpytorch-transformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphohenecker%2Fpytorch-transformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphohenecker%2Fpytorch-transformer/lists"}