{"id":17897961,"url":"https://github.com/grypesc/tatoeba-mt-benchmark","last_synced_at":"2026-02-10T08:32:40.791Z","repository":{"id":77201139,"uuid":"309470674","full_name":"grypesc/tatoeba-mt-benchmark","owner":"grypesc","description":"Tatoeba machine translation benchmark and implementations of different seq2seq algorithms.","archived":false,"fork":false,"pushed_at":"2021-07-01T10:38:20.000Z","size":205,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-08T05:45:47.668Z","etag":null,"topics":["benchmark","dataset","machine-translation","natural-language-processing","neural-machine-translation","nlp","pytorch","reinforcement-learning","seq2seq","sequence-to-sequence","tatoeba"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grypesc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-02T19:13:57.000Z","updated_at":"2024-01-01T00:00:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"7990ea81-f5e6-4336-af45-c424fee094d5","html_url":"https://github.com/grypesc/tatoeba-mt-benchmark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/grypesc/tatoeba-mt-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grypesc%2Ftatoeba-mt-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grypesc%2Ftatoeba-mt-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grypesc%2Ftatoeba-mt-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grypesc%2Ftatoeba-mt-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grypesc","download_url":"https://codeload.github.com/grypesc/tatoeba-mt-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grypesc%2Ftatoeba-mt-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273121981,"owners_count":25049539,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","dataset","machine-translation","natural-language-processing","neural-machine-translation","nlp","pytorch","reinforcement-learning","seq2seq","sequence-to-sequence","tatoeba"],"created_at":"2024-10-28T15:22:02.127Z","updated_at":"2026-02-10T08:32:40.757Z","avatar_url":"https://github.com/grypesc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tatoeba-machine-translation-benchmark\nTatoeba machine translation benchmark supporting 30 pairs of languages. This benchmark is focused on delivering\nhigh flexibility for research purposes. Datasets are tokenized with Spacy tokenizers during dataset generation phase.\nDataPipeline objects deliver vocabs and Torch data loaders. FastText pretrained embeddings are also available. Main \nevaluation method is perplexity and BLEU score based on torchtext implementation. Currently implemented models:\n* enc_dec_attn.py - Bidirectional encoder-decoder with attention.\n* rlst.py - Reinforcement Learning for on-line Sequence Transformation.\n* transformer.py - A standard transformer architecture\n\nSetup:\n```bash\ngit clone https://github.com/grypesc/tatoeba-mt-benchmark\ncd tatoeba-mt-benchmark \u0026\u0026 mkdir data \u0026\u0026 mkdir checkpoints\n```\nOr\n```bash\ngit clone https://ben.ii.pw.edu.pl/gitlab/recurrent-graph-networks/mt\ncd mt \u0026\u0026 mkdir data \u0026\u0026 mkdir checkpoints\n```\nThen:\n```bash\npython3 -m venv venv\nsource venv/bin/activate\npip install pip -U\npip install -r requirements.txt\npython -m spacy download en_core_web_md\npython -m spacy download es_core_news_md\npython -m spacy download fr_core_news_md\npython -m spacy download de_core_news_md\npython -m spacy download zh_core_web_md\npython -m spacy download ru_core_news_md\n```\nDownload raw data from 21.04.2021 snapshot: \n```bash\nwget 'https://www.dropbox.com/s/kuiseuq2rn5540s/21-04-2021-tatoeba.zip?dl=1' -O data/21-04-2021-tatoeba.zip\nunzip data/21-04-2021-tatoeba.zip -d data/\n```\nAlternatively, you can specify ```--update``` argument for generate_datasets.py to download the newest data from Tatoeba.\n\nCreate desired datasets by running generate_dataset.py, currently there are 6 languages \nsupported: (en, es, fr, de, zh, ru), so that's 30 combinations:\n\n```python3\npython generate_datasets.py --src en --trg es\n```\n\nTo repeat paper results for en-es language pair, firstly train models using following commands:\n```python3\npython enc_dec_attn.py --src en --trg es --embed-dropout 0.2  --decoder-dropout 0.5 --checkpoint-dir  checkpoints/enc-en-es \\\n--enc-hid-dim 256 --dec-hid-dim 256 --weight-decay 1e-5 --teacher-forcing 1.0 --epochs 50 --attn-dim 64 --batch-size 128 --lr 3e-4 \n```\n```python3 \npython rlst.py --src en --trg es --checkpoint-dir checkpoints/rlst-en-es --rnn-hid-dim 512 --teacher-forcing 1.0 \\\n--epochs 50 --lr 3e-4 --weight-decay 1e-5 --rnn-num-layers 4 --rnn-dropout 0.5 --embed-dropout 0.2 --N 50000 \\\n--eta-min 0.02 --eta-max 0.2 --rho 0.99\n```\n```python3 \npython transformer.py --src en --trg es --lr 3e-4 --checkpoint-dir checkpoints/trans-en-es --num-heads 8 --epochs 50 \\\n--dropout 0.25 --embed-dropout 0.2 --weight-decay 1e-4 --d-ffn 512 --num-layers 6 --batch-size 128 --d-model 256\n```\nNow test models on test and long test sets:\n```python3\npython enc_dec_attn.py --src en --trg es --checkpoint-dir checkpoints/enc-en-es --enc-hid-dim 256 --dec-hid-dim 256 \\\n--attn-dim 64 --batch-size 32 --test --test-seq-max-len 400 --load-model-name enc_dec_attn_best.pth\n```\n```python3 \npython rlst.py --src en --trg es --checkpoint-dir checkpoints/rlst-en-es --rnn-hid-dim 512 --rnn-num-layers 4 --test \\\n--batch-size 32 --testing-episode-max-time 512 --load-model-name rlst_best.pth\n```\n```python3 \npython transformer.py --src en --trg es --lr 3e-4 --checkpoint-dir checkpoints/trans-en-es --num-heads 8 --epochs 50 \\\n--d-ffn 512 --num-layers 6 --test --test-seq-max-len 400 --batch-size 32 --d-model 256 --load-model-name transformer_best.pth\n```\nTo repeat results for other language pairs change value of ```--src``` and ```--trg``` parameters. \n\nModels are saved and evaluated on validation set after every epoch. The benchmark always saves last and the best model\naccording to its BLEU score on validation set. For more information and hyperparameters:\n```python3\npython rlst.py --help\n```\n\nFeel free to add new models, suggestions, issues and make PRs :smiling_face_with_three_hearts:.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrypesc%2Ftatoeba-mt-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrypesc%2Ftatoeba-mt-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrypesc%2Ftatoeba-mt-benchmark/lists"}