{"id":13674176,"url":"https://github.com/dreamgonfly/Transformer-pytorch","last_synced_at":"2025-04-28T14:30:52.000Z","repository":{"id":62585176,"uuid":"149839228","full_name":"dreamgonfly/transformer-pytorch","owner":"dreamgonfly","description":"A PyTorch implementation of Transformer in \"Attention is All You Need\"","archived":false,"fork":false,"pushed_at":"2020-12-06T07:55:18.000Z","size":2898,"stargazers_count":103,"open_issues_count":3,"forks_count":28,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-10-02T18:47:16.852Z","etag":null,"topics":["deep-learning","machine-translation","natural-language-processing"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1706.03762","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dreamgonfly.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-09-22T02:43:29.000Z","updated_at":"2024-07-17T04:50:30.000Z","dependencies_parsed_at":"2022-11-03T22:03:26.139Z","dependency_job_id":null,"html_url":"https://github.com/dreamgonfly/transformer-pytorch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dreamgonfly%2Ftransformer-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dreamgonfly%2Ftransformer-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dreamgonfly%2Ftransformer-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dreamgonfly%2Ftransformer-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dreamgonfly","download_url":"https://codeload.github.com/dreamgonfly/transformer-pytorch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224120047,"owners_count":17259004,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","machine-translation","natural-language-processing"],"created_at":"2024-08-02T11:00:42.449Z","updated_at":"2024-11-11T14:30:41.068Z","avatar_url":"https://github.com/dreamgonfly.png","language":"Python","funding_links":[],"categories":["GitHub"],"sub_categories":[],"readme":"# Transformer-pytorch\nA PyTorch implementation of Transformer in \"Attention is All You Need\" (https://arxiv.org/abs/1706.03762)\n\nThis repo focuses on clean, readable, and modular implementation of the paper.\n\n\u003cimg width=\"559\" alt=\"screen shot 2018-09-27 at 1 49 14 pm\" src=\"https://user-images.githubusercontent.com/2340721/46123973-44b08900-c25c-11e8-9468-7aef9e4e3f18.png\"\u003e\n\n## Requirements\n- Python 3.6+\n- [PyTorch 4.1+](http://pytorch.org/)\n- [NumPy](http://www.numpy.org/)\n- [NLTK](https://www.nltk.org/)\n- [tqdm](https://github.com/tqdm/tqdm)\n\n## Usage\n\n### Prepare datasets\nThis repo comes with example data in `data/` directory. To begin, you will need to prepare datasets with given data as follows:\n```\n$ python prepare_datasets.py --train_source=data/example/raw/src-train.txt --train_target=data/example/raw/tgt-train.txt --val_source=data/example/raw/src-val.txt --val_target=data/example/raw/tgt-val.txt --save_data_dir=data/example/processed\n```\n\nThe example data is brought from [OpenNMT-py](https://github.com/OpenNMT/OpenNMT-py).\nThe data consists of parallel source (src) and target (tgt) data for training and validation.\nA data file contains one sentence per line with tokens separated by a space.\nBelow are the provided example data files.\n\n- `src-train.txt`\n- `tgt-train.txt`\n- `src-val.txt`\n- `tgt-val.txt`\n\n### Train model\nTo train model, provide the train script with a path to processed data and save files as follows:\n\n```\n$ python train.py --data_dir=data/example/processed --save_config=checkpoints/example_config.json --save_checkpoint=checkpoints/example_model.pth --save_log=logs/example.log \n```\n\nThis saves model config and checkpoints to given files, respectively.\nYou can play around with hyperparameters of the model with command line arguments. \nFor example, add `--epochs=300` to set the number of epochs to 300. \n\n### Translate\nTo translate a sentence in source language to target language:\n```\n$ python predict.py --source=\"There is an imbalance here .\" --config=checkpoints/example_config.json --checkpoint=checkpoints/example_model.pth\n\nCandidate 0 : Hier fehlt das Gleichgewicht .\nCandidate 1 : Hier fehlt das das Gleichgewicht .\nCandidate 2 : Hier fehlt das das das Gleichgewicht .\n```\n\nIt will give you translation candidates of the given source sentence.\nYou can adjust the number of candidates with command line argument. \n\n### Evaluate\nTo calculate BLEU score of a trained model:\n```\n$ python evaluate.py --save_result=logs/example_eval.txt --config=checkpoints/example_config.json --checkpoint=checkpoints/example_model.pth\n\nBLEU score : 0.0007947\n```\n\n## File description\n- `models.py` includes Transformer's encoder, decoder, and multi-head attention.\n- `embeddings.py` contains positional encoding.\n- `losses.py` contains label smoothing loss.\n- `optimizers.py` contains Noam optimizer.\n- `metrics.py` contains accuracy metric.\n- `beam.py` contains beam search.\n- `datasets.py` has code for loading and processing data. \n- `trainer.py` has code for training model.\n- `prepare_datasets.py` processes data.\n- `train.py` trains model.\n- `predict.py` translates given source sentence with a trained model.\n- `evaluate.py` calculates BLEU score of a trained model.\n\n## Reference\n- [OpenNMT-py](https://github.com/OpenNMT/OpenNMT-py)\n\n## Author\n[@dreamgonfly](https://github.com/dreamgonfly)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdreamgonfly%2FTransformer-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdreamgonfly%2FTransformer-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdreamgonfly%2FTransformer-pytorch/lists"}