{"id":24330826,"url":"https://github.com/andreimoraru123/neural-machine-translation","last_synced_at":"2026-01-20T09:01:32.821Z","repository":{"id":180781465,"uuid":"665395537","full_name":"AndreiMoraru123/Neural-Machine-Translation","owner":"AndreiMoraru123","description":"Modern Eager TensorFlow implementation of Attention Is All You Need","archived":false,"fork":false,"pushed_at":"2024-10-07T10:16:45.000Z","size":1069,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-14T17:02:51.448Z","etag":null,"topics":["attention","beam-search","bleu-score","byte-pair-encoding","deep-learning","dot-product-attention","einops","embedding-projector","embeddings","encoder-decoder","keras","label-smoothing","language","language-model","nlp","self-attention","tensorflow","tokenization","transformers","translation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AndreiMoraru123.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-07-12T05:44:34.000Z","updated_at":"2024-10-07T10:16:49.000Z","dependencies_parsed_at":"2023-09-25T06:47:02.369Z","dependency_job_id":null,"html_url":"https://github.com/AndreiMoraru123/Neural-Machine-Translation","commit_stats":{"total_commits":80,"total_committers":2,"mean_commits":40.0,"dds":"0.36250000000000004","last_synced_commit":"5bb0eb7c301504b72e74cc8b013f6139a823e61d"},"previous_names":["andreimoraru123/machine-translation","andreimoraru123/neural-machine-translation"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AndreiMoraru123/Neural-Machine-Translation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndreiMoraru123%2FNeural-Machine-Translation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndreiMoraru123%2FNeural-Machine-Translation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndreiMoraru123%2FNeural-Machine-Translation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndreiMoraru123%2FNeural-Machine-Translation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AndreiMoraru123","download_url":"https://codeload.github.com/AndreiMoraru123/Neural-Machine-Translation/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AndreiMoraru123%2FNeural-Machine-Translation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28599828,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-20T08:51:33.170Z","status":"ssl_error","status_checked_at":"2026-01-20T08:51:10.855Z","response_time":117,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","beam-search","bleu-score","byte-pair-encoding","deep-learning","dot-product-attention","einops","embedding-projector","embeddings","encoder-decoder","keras","label-smoothing","language","language-model","nlp","self-attention","tensorflow","tokenization","transformers","translation"],"created_at":"2025-01-18T01:14:38.683Z","updated_at":"2026-01-20T09:01:32.803Z","avatar_url":"https://github.com/AndreiMoraru123.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Attention Is All You Need\n\n![TensorFlow](https://img.shields.io/badge/TensorFlow-%23FF6F00.svg?style=for-the-badge\u0026logo=TensorFlow\u0026logoColor=white)  ![Keras](https://img.shields.io/badge/Keras-%23D00000.svg?style=for-the-badge\u0026logo=Keras\u0026logoColor=white)  ![nVIDIA](https://img.shields.io/badge/nVIDIA-%2376B900.svg?style=for-the-badge\u0026logo=nVIDIA\u0026logoColor=white)\n\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![Checked with mypy](https://www.mypy-lang.org/static/mypy_badge.svg)](https://mypy-lang.org/) [![Build Status](https://github.com/AndreiMoraru123/Neural-Machine-Translation/actions/workflows/python-app.yml/badge.svg)](https://github.com/AndreiMoraru123/Neural-Machine-Translation/actions/workflows/python-app.yml)\n\n\n\u003e [!NOTE]\\\n\u003e I adapted the code from [this awesome PyTorch version](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Machine-Translation/tree/master). Please check it out as well.\n\n\u003e [!IMPORTANT]\\\n\u003e I am using python `3.9` with tensorflow `2.10` as this is their last available version for native-Windows on GPU.\n\n## Steps\n 1. `pip install -r requirements.txt`\n 2. [download.py](https://github.com/AndreiMoraru123/Neural-Machine-Translation/blob/main/download.py) downloads all the data (`en`-`de` file pairs from Europarl, Common Crawl and News Commentary) to the specified folder as argument.\n 3. [encode.py](https://github.com/AndreiMoraru123/Neural-Machine-Translation/blob/main/encode.py) filters the data based on the arguments (origin, maximum length etc.) and trains the BPE model, saving it to a file.\n 4. [train.py](https://github.com/AndreiMoraru123/Neural-Machine-Translation/blob/main/train.py) runs the whole training pipeline with top-down logic found in the file. Everything is managed by  the `Trainer` from [trainer.py](https://github.com/AndreiMoraru123/Neural-Machine-Translation/blob/main/trainer.py) (logging embeddings, checkpointing etc.).\n 5. [translate.py](https://github.com/AndreiMoraru123/Neural-Machine-Translation/blob/main/translate.py) runs the model inference and optionally evaluates it with `sacrebleu` using the `Evaluator` from [evaluator.py](https://github.com/AndreiMoraru123/Neural-Machine-Translation/blob/main/evaluator.py).\n 6. [docs](https://github.com/AndreiMoraru123/Neural-Machine-Translation/tree/main/docs) contains notes with svg drawings from [the original repo](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Machine-Translation/tree/master) and markdown files explaining the choices I had to make for adaptating from one framework to another.\n\nThe code itself is heavily commented and you can get a feel for how language models work by looking at the [tests](https://github.com/AndreiMoraru123/Neural-Machine-Translation/tree/main/test).\n\n## Overfitting on one sentence\n\nInput sequence:\n\n```\n\"I declare resumed the session of the European Parliament \"\n\"adjourned on Friday 17 December 1999, and I would like \"\n\"once again to wish you a happy new year in the hope that \"\n\"you enjoyed a pleasant festive period.\"\n```\n\nResults in the following generated hypotheses (all should to be the top one and the exact label for this sentence):\n\nTop generated sequence:\n```\n('Ich erkläre die am Freitag, dem 17. Dezember unterbrochene Sitzungsperiode '\n 'des Europäischen Parlaments für wiederaufgenommen, wünsche Ihnen nochmals '\n 'alles Gute zum Jahreswechsel und hoffe, daß Sie schöne Ferien hatten.')\n```\nAll generated sequences in the beam (k=5) search:\n```\n[{'hypothesis': 'Ich die am Freitag, dem 17. Dezember unterbrochene '\n                'Sitzungsperiode des Europäischen Parlaments für '\n                'wiederaufgenommen, wünsche Ihnen nochmals alles Gute zum '\n                'Jahreswechsel und hoffe, daß Sie schöne Ferien hatten.',\n  'score': -3.3601136207580566},\n {'hypothesis': 'Ich erkläre die am Freitag, dem 17. Dezember unterbrochene '\n                'Sitzungsperiode des Europäischen Parlaments für '\n                'wiederaufgenommen, wünsche Ihnen nochmals alles Gute zum '\n                'Jahreswechsel und hoffe, daß Sie schöne Ferien hatten.',\n  'score': -1.4448045492172241},\n {'hypothesis': 'Ich Ich erkläre die am Freitag, dem 17. Dezember '\n                'unterbrochene Sitzungsperiode des Europäischen Parlaments für '\n                'wiederaufgenommen, wünsche Ihnen nochmals alles Gute zum '\n                'Jahreswechsel und hoffe, daß Sie schöne Ferien hatten.',\n  'score': -3.1513545513153076},\n {'hypothesis': 'Ich erkläre die die am Freitag, dem 17. Dezember '\n                'unterbrochene Sitzungsperiode des Europäischen Parlaments für '\n                'wiederaufgenommen, wünsche Ihnen nochmals alles Gute zum '\n                'Jahreswechsel und hoffe, daß Sie schöne Ferien hatten.',\n  'score': -3.3080737590789795},\n {'hypothesis': 'Ich erkläre erkläre die am Freitag, dem 17. Dezember '\n                'unterbrochene Sitzungsperiode des Europäischen Parlaments für '\n                'wiederaufgenommen, wünsche Ihnen nochmals alles Gute zum '\n                'Jahreswechsel und hoffe, daß Sie schöne Ferien hatten.',\n  'score': -3.3361663818359375}]\n```\n\nThese are negative as they are log probabilities, the closest to zero being the top sequence\n\nAs a sanity check, the BLEU score should be a perfect `100/100` in all cases:\n\n```\nINFO:root:13a tokenization, cased\nINFO:root:BLEU = 100.00 100.0/100.0/100.0/100.0 (BP = 1.000 ratio = 1.000 hyp_len = 34 ref_len = 34)\nINFO:root:13a tokenization, caseless\nINFO:root:BLEU = 100.00 100.0/100.0/100.0/100.0 (BP = 1.000 ratio = 1.000 hyp_len = 34 ref_len = 34)\nINFO:root:International tokenization, cased\nINFO:root:BLEU = 100.00 100.0/100.0/100.0/100.0 (BP = 1.000 ratio = 1.000 hyp_len = 34 ref_len = 34)\nINFO:root:International tokenization, caseless\nINFO:root:BLEU = 100.00 100.0/100.0/100.0/100.0 (BP = 1.000 ratio = 1.000 hyp_len = 34 ref_len = 34)\n```\n\n## Embeddings\n\nAfter training for a while, some interesting patterns arise. This project integrates them into the [Embedding Projector](https://www.tensorflow.org/tensorboard/tensorboard_projector_plugin).\n\nIn the shared vocabulary between the encoder (english) and decoder (german) we can see some cosine similarities:\n\n#### British with *britischen* and *nationaler*\n![british](https://github.com/AndreiMoraru123/Neural-Machine-Translation/assets/81184255/0ca4381e-5435-4e06-b4c6-3c53ad844d8e)\n\n#### will and *wollte* (\u0026 *konnte*, *mochte*, *wurde*)\n![will](https://github.com/AndreiMoraru123/Neural-Machine-Translation/assets/81184255/05b2d4f3-9f9b-4c96-8a23-6f9c25051eff)\n\n#### *Bedenken* (pondering) is closest to *glaube* (believe)\n![image](https://github.com/AndreiMoraru123/Neural-Machine-Translation/assets/81184255/5de08f03-271c-498b-9ad8-5b49273e0a97)\n\n#### *Entschließung* (resolution) gets associated with *completed* \n\n![Resolution60k](https://github.com/AndreiMoraru123/machine-translation/assets/81184255/2839ae4e-1cfd-4ca0-a160-fd1fd5abf948)\n\n#### *gessammelt* (collected) maps to *decision* and *Bestimmung* (determination) as well as *verstärkt* (strenghtened)\n\n![strengthened_collected_consistent60k](https://github.com/AndreiMoraru123/machine-translation/assets/81184255/4c0743d5-3acd-4e95-a208-8f66e04d80ff)\n\n#### *Change* also gets associated with *neuer* (new)\n\n![change_neuer_60k](https://github.com/AndreiMoraru123/machine-translation/assets/81184255/7e4320a6-e543-4fc1-bccd-ad08683b38ae)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreimoraru123%2Fneural-machine-translation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandreimoraru123%2Fneural-machine-translation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreimoraru123%2Fneural-machine-translation/lists"}