{"id":18786431,"url":"https://github.com/kayoyin/transformer-slt","last_synced_at":"2025-12-22T07:30:13.461Z","repository":{"id":50563403,"uuid":"252429009","full_name":"kayoyin/transformer-slt","owner":"kayoyin","description":"Sign Language Translation with Transformers (COLING'2020, ECCV'20 SLRTP Workshop)","archived":false,"fork":false,"pushed_at":"2021-02-02T20:24:15.000Z","size":7546,"stargazers_count":150,"open_issues_count":4,"forks_count":39,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-12-29T12:47:52.462Z","etag":null,"topics":["deep-learning","machine-learning","neural-machine-translation","sign-language-translation","transformer"],"latest_commit_sha":null,"homepage":"","language":"ASL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kayoyin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-02T10:52:25.000Z","updated_at":"2024-12-14T16:42:26.000Z","dependencies_parsed_at":"2022-09-23T02:00:32.690Z","dependency_job_id":null,"html_url":"https://github.com/kayoyin/transformer-slt","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayoyin%2Ftransformer-slt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayoyin%2Ftransformer-slt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayoyin%2Ftransformer-slt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayoyin%2Ftransformer-slt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kayoyin","download_url":"https://codeload.github.com/kayoyin/transformer-slt/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239703051,"owners_count":19683196,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","machine-learning","neural-machine-translation","sign-language-translation","transformer"],"created_at":"2024-11-07T20:51:33.021Z","updated_at":"2025-12-22T07:30:13.371Z","avatar_url":"https://github.com/kayoyin.png","language":"ASL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# transformer-slt\nThis repository gathers data and code supporting the experiments in the paper [Better Sign Language Translation with STMC-Transformer](https://www.aclweb.org/anthology/2020.coling-main.525/).\n\n## Installation\nThis code is based on [OpenNMT](https://github.com/OpenNMT/OpenNMT-py) v1.0.0 and requires all of its dependencies (`torch==1.6.0`). Additional requirements are [NLTK](https://www.nltk.org/) for NMT evaluation metrics.\n\nThe recommended way to install is shown below:\n```\n# create a new virtual environment\nvirtualenv --python=python3 venv\nsource venv/bin/activate\n\n# clone the repo\ngit clone https://github.com/kayoyin/transformer-slt.git\ncd transformer-slt\n\n# install python dependencies\npip install -r requirements.txt\n\n# install OpenNMT-py\npython setup.py install\n\n```\n\n## Sample Usage\n\n### Data processing\n\n```\nonmt_preprocess -train_src data/phoenix2014T.train.gloss -train_tgt data/phoenix2014T.train.de -valid_src data/phoenix2014T.dev.gloss -valid_tgt data/phoenix2014T.dev.de -save_data data/dgs -lower \n```\n\n### Training\n```\npython  train.py -data data/dgs -save_model model -keep_checkpoint 1 \\\n          -layers 2 -rnn_size 512 -word_vec_size 512 -transformer_ff 2048 -heads 8  \\\n          -encoder_type transformer -decoder_type transformer -position_encoding \\\n          -max_generator_batches 2 -dropout 0.1 \\\n          -early_stopping 3 -early_stopping_criteria accuracy ppl \\\n          -batch_size 2048 -accum_count 3 -batch_type tokens -normalization tokens \\\n          -optim adam -adam_beta2 0.998 -decay_method noam -warmup_steps 3000 -learning_rate 0.5 \\\n          -max_grad_norm 0 -param_init 0  -param_init_glorot \\\n          -label_smoothing 0.1 -valid_steps 100 -save_checkpoint_steps 100 \\\n          -world_size 1 -gpu_ranks 0\n```\n\n### Inference\n```\npython translate.py -model model [model2 model3 ...] -src data/phoenix2014T.test.gloss -output pred.txt -gpu 0 -replace_unk -beam_size 4\n```\n\n### Scoring\n```\n# BLEU-1,2,3,4\npython tools/bleu.py 1 pred.txt data/phoenix2014T.test.de\npython tools/bleu.py 2 pred.txt data/phoenix2014T.test.de\npython tools/bleu.py 3 pred.txt data/phoenix2014T.test.de\npython tools/bleu.py 4 pred.txt data/phoenix2014T.test.de\n\n# ROUGE\npython tools/rouge.py pred.txt data/phoenix2014T.test.de\n\n# METEOR\npython tools/meteor.py pred.txt data/phoenix2014T.test.de\n```\n\n# To dos:\n* Add configurations \u0026 steps to recreate paper results\n\n# Reference\nPlease cite the paper below if you found the resources in this repository useful:\n```\n@inproceedings{yin-read-2020-better,\n    title = \"Better Sign Language Translation with {STMC}-Transformer\",\n    author = \"Yin, Kayo  and\n      Read, Jesse\",\n    booktitle = \"Proceedings of the 28th International Conference on Computational Linguistics\",\n    month = dec,\n    year = \"2020\",\n    address = \"Barcelona, Spain (Online)\",\n    publisher = \"International Committee on Computational Linguistics\",\n    url = \"https://www.aclweb.org/anthology/2020.coling-main.525\",\n    doi = \"10.18653/v1/2020.coling-main.525\",\n    pages = \"5975--5989\",\n    abstract = \"Sign Language Translation (SLT) first uses a Sign Language Recognition (SLR) system to extract sign language glosses from videos. Then, a translation system generates spoken language translations from the sign language glosses. This paper focuses on the translation system and introduces the STMC-Transformer which improves on the current state-of-the-art by over 5 and 7 BLEU respectively on gloss-to-text and video-to-text translation of the PHOENIX-Weather 2014T dataset. On the ASLG-PC12 corpus, we report an increase of over 16 BLEU. We also demonstrate the problem in current methods that rely on gloss supervision. The video-to-text translation of our STMC-Transformer outperforms translation of GT glosses. This contradicts previous claims that GT gloss translation acts as an upper bound for SLT performance and reveals that glosses are an inefficient representation of sign language. For future SLT research, we therefore suggest an end-to-end training of the recognition and translation models, or using a different sign language annotation scheme.\",\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkayoyin%2Ftransformer-slt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkayoyin%2Ftransformer-slt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkayoyin%2Ftransformer-slt/lists"}