{"id":13787135,"url":"https://github.com/jerinphilip/ilmulti","last_synced_at":"2025-04-14T07:07:33.965Z","repository":{"id":62570475,"uuid":"148030258","full_name":"jerinphilip/ilmulti","owner":"jerinphilip","description":"Tooling to play around with multilingual machine translation for Indian Languages.","archived":false,"fork":false,"pushed_at":"2022-03-05T14:46:38.000Z","size":5749,"stargazers_count":22,"open_issues_count":5,"forks_count":4,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-14T07:07:27.846Z","etag":null,"topics":["indian-languages","machine-translation","machine-translation-models","multilingual-translation","multilingual-translations","pytorch","tokenizer","wrappers"],"latest_commit_sha":null,"homepage":"http://preon.iiit.ac.in/~jerin/bhasha","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jerinphilip.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-09-09T13:52:36.000Z","updated_at":"2025-03-09T11:28:02.000Z","dependencies_parsed_at":"2022-11-03T17:15:47.111Z","dependency_job_id":null,"html_url":"https://github.com/jerinphilip/ilmulti","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jerinphilip%2Filmulti","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jerinphilip%2Filmulti/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jerinphilip%2Filmulti/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jerinphilip%2Filmulti/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jerinphilip","download_url":"https://codeload.github.com/jerinphilip/ilmulti/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248837280,"owners_count":21169374,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["indian-languages","machine-translation","machine-translation-models","multilingual-translation","multilingual-translations","pytorch","tokenizer","wrappers"],"created_at":"2024-08-03T20:00:29.589Z","updated_at":"2025-04-14T07:07:33.941Z","avatar_url":"https://github.com/jerinphilip.png","language":"Python","funding_links":[],"categories":["**Tools, Libraries, Models**","Urdu NLP Tools, Libraries and Models"],"sub_categories":["Translation","Translation Models"],"readme":"# ilmulti\n\nThis repository houses tooling used to create the models on the\nleaderboard of WAT-Tasks. We provide wrappers to models which are\ntrained via [pytorch/fairseq](http://github.com/pytorch/fairseq) to\ntranslate. Installation and usage intructions are provided below.\n\n* **Training**: We use a separate fork of\n  [pytorch/fairseq](http://github.com/pytorch/fairseq) at\n  [jerinphilip/fairseq-ilmt](http://github.com/jerinphilip/fairseq-ilmt) for\n  training to optimize for our cluster and to plug and play data\n  easily.\n\n* **Pretrained Models and Other Resources**: \n  [preon.iiit.ac.in/~jerin/bhasha](http://preon.iiit.ac.in/~jerin/bhasha)\n\n\n## Installation\n\nThe code is tested to work with the fairseq-fork which is branched from v0.8.0 and torch version 1.0.0.\n\n```bash\n# --user is optional\n\n# Check requirements.txt, packages for translation:\n# fairseq-ilmt@lrec-2020 and torch  are not enabled by default.\npython3 -m pip install -r requirements.txt --user  \n\n# Once requirements are installed, you can install ilmulti into library.\n\npython3 setup.py install --user \n\n```\n\n**Downloading Models**: The script\n[`scripts/download-and-setup-models.sh`](./scripts/download-and-setup-models.sh)\ndownloads the model and dictionary files required for running\n[`examples/mm_all.py`](./examples/mm_all.py). Which models to download can be\nconfigured in the script.\n\nA working example using the wrappers in this code can be found in [this](https://colab.research.google.com/drive/1KOvjawhzPXOQ6RLlFBFeInkuuR0QAWTK?usp=sharing) colab notebook. Thanks @Nimishasri.\n\n## Usage\n\n```python3\nfrom ilmulti.translator import from_pretrained\n\ntranslator = from_pretrained(tag='mm-all')\nsample = translator(\"The quick brown fox jumps over the lazy dog\", tgt_lang='hi')\n```\n\nThe code works with three main components:\n\n### 1. Segmenter\n\nAlso sentence-tokenizer. To handle segmenting a block of text into sentences,\naccounting for some Indian Language delimiters. \n\n1. PatternSegmenter: There  is a bit crude and rule based implementation\n   contributed by [Binu Jasim](https://github.com/bnjasim).\n2. PunktSegmenter: changed this to an unsupervised learnt PunktTokenizer\n\n### 2. Tokenization\n\nWe use [SentencePiece](https://github.com/google/sentencepiece) to\nas an unsupervised tokenizer for Indian languages, which works\nsurprisingly well in our experiments. There are trained models on\nwhatever corpora we could find for the specific languages in\n[sentencepiece/models](./sentencepiece/models) of 4000 vocabulary units\nand 8000 vocabulary units.\n\nTraining a joint SentencePiece over all languages lead to character\nlevel tokenization for under-represented languages and since there isn't\nmuch to gain due to the difference in scripts, we use individual\ntokenizers for each language. Combined however, this will have less than\n4000 x |#languages| as some common English code mixes come in. This\nhowever, makes the MT system robust in some sense to code-mixed inputs.\n\n### 3. Translator\n\nTranslator is a wrapper around a\n[fairseq](https://github.com/pytorch/fairseq) which we have reused for\nsome web-interfaces and demos.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjerinphilip%2Filmulti","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjerinphilip%2Filmulti","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjerinphilip%2Filmulti/lists"}