{"id":20049682,"url":"https://github.com/Esmail-ibraheem/Transformer-pytorch","last_synced_at":"2025-05-05T11:31:03.103Z","repository":{"id":215283023,"uuid":"737771506","full_name":"Esmail-ibraheem/Transformer-pytorch","owner":"Esmail-ibraheem","description":"Language to Language Transformer model from scartch using pure Pytorch where I used my transformer model for translation task. from the paper \"Attention is all you Need\" 2017 using pytorch.","archived":false,"fork":false,"pushed_at":"2024-09-04T06:16:13.000Z","size":9032,"stargazers_count":8,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-08T21:51:07.448Z","etag":null,"topics":["llm","machine-translation","paper-implementations","pytorch","transformers-models"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Esmail-ibraheem.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-01T12:47:46.000Z","updated_at":"2025-02-06T14:43:47.000Z","dependencies_parsed_at":"2025-02-01T16:48:18.875Z","dependency_job_id":"c6fa5b6e-e54f-4a84-882d-b90f3473e7b2","html_url":"https://github.com/Esmail-ibraheem/Transformer-pytorch","commit_stats":null,"previous_names":["esmail-ibraheem/lltra-model.","esmail-ibraheem/transformer-pytorch"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Esmail-ibraheem%2FTransformer-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Esmail-ibraheem%2FTransformer-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Esmail-ibraheem%2FTransformer-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Esmail-ibraheem%2FTransformer-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Esmail-ibraheem","download_url":"https://codeload.github.com/Esmail-ibraheem/Transformer-pytorch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252488926,"owners_count":21756238,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llm","machine-translation","paper-implementations","pytorch","transformers-models"],"created_at":"2024-11-13T11:51:44.860Z","updated_at":"2025-05-05T11:30:59.155Z","avatar_url":"https://github.com/Esmail-ibraheem.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eLlTRA-Model.\u003c/h1\u003e\n\n![Transformer_Diagram_2 drawio](https://github.com/Esmail-ibraheem/LlTRA-Model/assets/113830751/cdc030e6-ed4c-4040-a00d-50ab780cafb7)\n\n\n\nLlTRA stands for: Language to Language Transformer model from the paper \"Attention is all you Need\", building transformer model:[Transformer model](https://github.com/Esmail-ibraheem/Transformer-model) from scratch and using it for translation using pytorch.\n\n### Problem Statement:\n\nIn the rapidly evolving landscape of natural language processing (NLP) and machine translation, there exists a persistent challenge in achieving accurate and contextually rich language-to-language transformations. Existing models often struggle with capturing nuanced semantic meanings, context preservation, and maintaining grammatical coherence across different languages. Additionally, the demand for efficient cross-lingual communication and content generation has underscored the need for a versatile language transformer model that can seamlessly navigate the intricacies of diverse linguistic structures.\n\n---\n\n### Goal:\n\nDevelop a specialized language-to-language transformer model that accurately translates from the Arabic language to the English language, ensuring semantic fidelity, contextual awareness, cross-lingual adaptability, and the retention of grammar and style. The model should provide efficient training and inference processes to make it practical and accessible for a wide range of applications, ultimately contributing to the advancement of Arabic-to-English language translation capabilities.\n\n---\n\n### Dataset used:\n\nfrom hugging Face \n[huggingface/opus_infopankki](https://huggingface.co/datasets/opus_infopankki/viewer/ar-en/train?p=3)\n\n---\n\n### Configuration:\n\nthis is the settings of the model, You can customize the source and target languages, sequence lengths for each, the number of epochs, batch size, and more.\n```python\ndef Get_configuration():\n    return {\n        \"batch_size\": 8,\n        \"num_epochs\": 30,\n        \"lr\": 10**-4,\n        \"sequence_length\": 100,\n        \"d_model\": 512,\n        \"datasource\": 'opus_infopankki',\n        \"source_language\": \"ar\",\n        \"target_language\": \"en\",\n        \"model_folder\": \"weights\",\n        \"model_basename\": \"tmodel_\",\n        \"preload\": \"latest\",\n        \"tokenizer_file\": \"tokenizer_{0}.json\",\n        \"experiment_name\": \"runs/tmodel\"\n    }\n```\n---\n\n### Search algorithm used:\n\nGreedy Algorithm for finding which token has the maximum probability.\n\n---\n\n### Training:\n\nI used my drive to upload the project and then connected it to the Google Collab to train it:\n\n- hours of training: 4 hours.\n- epochs: 20.\n- number of dataset rows: 2,934,399.\n- size of the dataset: 95MB.\n- size of the auto-converted parquet files: 153MB.\n- Arabic tokens: 29999.\n- English tokens: 15697.\n- pre-trained model in collab.\n- BLEU score from Arabic to English: 19.7 \n---\n\n### Some Results:\n\n    SOURCE: العائلات الناطقة بلغة أجنبية لديها الحق في خدمات الترجمة عند اللزوم.\n    TARGET: A foreign-language family is entitled to interpreting services as necessary.\n    PREDICTED: in a native language, it is provided by the services of the services for the elderly.\n    --------------------------------------------------------------------------------\n    SOURCE: قمت بارتكاب جرائم وتُعتبر بأنك خطير على النظام أو الأمن العام.\n    TARGET: you have committed crimes and are considered a danger to public order or safety\n    PREDICTED: you have committed crimes and are considered a danger to public order or safety\n    --------------------------------------------------------------------------------\n    SOURCE: عندما تلتحق بالدراسة، فستحصل على الحق في إنجاز كلتا الدرجتين العلميتين.\n    TARGET: When you are accepted into an institute of higher education, you receive the right to complete both degrees.\n    PREDICTED: When you have a of residence, you will receive a higher education degree.\n    --------------------------------------------------------------------------------\n    SOURCE: اللجنة لا تتداول حالات التهميش والتمييز المتعلقة بالعمل.\n    TARGET: The Tribunal does not handle cases of employment-related discrimination.\n    PREDICTED: The does not have to pay and the work.\n    --------------------------------------------------------------------------------\n    SOURCE: يجب عليك أيضاً أن تستطيع إثبات على سبيل المثال بالوصفة الطبية أو بالتقرير الطبي بأن الغرض من الدواء هو استخدامك أنت الشخصي.\n    TARGET: In addition, you must be able to prove with a prescription or medical certificate, for example, that the medicine is intended for your personal use.\n    PREDICTED: You must also have to prove your identity with a friend or friend, for example, that the medicine is intended for your personal use.\n    --------------------------------------------------------------------------------\n    SOURCE: إذا كان لديك ترخيص إقامة في فنلندا، ولكن لم تُمنح ترخيص إقامة استمراري، فسوف تصدر دائرة شؤون الهجرة قراراً بالترحيل.\n    TARGET: If you already have a residence permit in Finland but are not granted a residence permit extension, the Finnish Immigration Service makes a deportation decision.\n    PREDICTED: If you have a residence permit in but are not granted a residence permit, the Service makes a decision.\n--- \n\ncheck the theoretical part: [Theoretical part](https://github.com/Esmail-ibraheem/Transformer-model-theoretical-part)\n\ndeveloping process: \n\n1. https://youtu.be/oPaQGmBDxh4\n2. https://youtu.be/Nm5CUo7ol18?si=5U3m4IM7crJ23Zzu\n3. https://youtu.be/O-jfimyP6Tw?si=ucPweGo2b7gh2rrI\n\n---\n\n- [my pre trained model](https://colab.research.google.com/drive/1DX7NJoTMQ0py93nvzmEVRYxwSzYt0dRH?usp=drive_link)\n- [my model in drive](https://drive.google.com/drive/folders/1GR6U1rLWZo--3r0ZLWTmM2Ge8Zq_NkE-?usp=drive_link)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEsmail-ibraheem%2FTransformer-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FEsmail-ibraheem%2FTransformer-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEsmail-ibraheem%2FTransformer-pytorch/lists"}