{"id":13789408,"url":"https://github.com/goru001/inltk","last_synced_at":"2025-05-12T06:31:38.814Z","repository":{"id":41086435,"uuid":"177263796","full_name":"goru001/inltk","owner":"goru001","description":"Natural Language Toolkit for Indic Languages aims to provide out of the box support for various NLP tasks that an application developer might need","archived":false,"fork":false,"pushed_at":"2024-01-20T12:40:33.000Z","size":831,"stargazers_count":830,"open_issues_count":37,"forks_count":161,"subscribers_count":37,"default_branch":"master","last_synced_at":"2025-04-20T22:46:58.303Z","etag":null,"topics":["data-augmentation","deep-learning","indic-languages","nlp","pytorch","sentence-embeddings","sentence-encoding","sentence-similarity","word-embeddings"],"latest_commit_sha":null,"homepage":"https://inltk.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/goru001.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-23T08:28:20.000Z","updated_at":"2025-04-15T05:46:54.000Z","dependencies_parsed_at":"2024-06-19T01:53:00.401Z","dependency_job_id":"f2982feb-7d4f-47f7-87f9-81761be670e4","html_url":"https://github.com/goru001/inltk","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goru001%2Finltk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goru001%2Finltk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goru001%2Finltk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goru001%2Finltk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/goru001","download_url":"https://codeload.github.com/goru001/inltk/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252940883,"owners_count":21828766,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-augmentation","deep-learning","indic-languages","nlp","pytorch","sentence-embeddings","sentence-encoding","sentence-similarity","word-embeddings"],"created_at":"2024-08-03T22:00:21.742Z","updated_at":"2025-05-12T06:31:38.792Z","avatar_url":"https://github.com/goru001.png","language":"Python","readme":"## Natural Language Toolkit for Indic Languages (iNLTK)\n\n[![Gitter](https://badges.gitter.im/inltk/community.svg)](https://gitter.im/inltk/community?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge) [![Downloads](https://pepy.tech/badge/inltk)](https://pepy.tech/project/inltk)\n\niNLTK aims to provide out of the box support for various NLP tasks \nthat an application developer might need for Indic languages. Paper for iNLTK library has been accepted at EMNLP-2020's NLP-OSS workshop. Here's the [link to the paper](https://www.aclweb.org/anthology/2020.nlposs-1.10/)\n\n\n### Documentation\n\nCheckout detailed docs along with Installation instructions\n at https://inltk.readthedocs.io\n\n\n### Supported languages\n\n#### Native languages\n\n| Language | Code \u003ccode-of-language\u003e |\n|:--------:|:----:|\n|   Hindi  |  hi  |\n|  Punjabi |  pa  |\n| Gujarati |  gu  |\n|  Kannada |  kn  |\n| Malayalam |  ml  |\n|   Oriya   |  or  |\n|  Marathi |  mr  |\n|  Bengali |  bn  |\n|   Tamil  |  ta  |\n|   Urdu  |  ur  |\n|  Nepali  |  ne  |\n| Sanskrit |  sa  |\n|   English  |  en  |\n|   Telugu  |  te  |\n\n#### Code Mixed languages\n\n| Language | Script |Code \u003ccode-of-language\u003e |\n|:--------:|:----:|:----:|\n| Hinglish (Hindi+English)  |  Latin  |  hi-en  |\n| Tanglish (Tamil+English) |  Latin  |  ta-en  |\n| Manglish (Malayalam+English) |  Latin  |  ml-en  |\n\n#### Repositories containing models used in iNLTK\n\n|  Language |                            Repository                            |                                                                                     Dataset used for Language modeling                                                                                     | Perplexity of ULMFiT LM\u003cbr\u003e(on validation set) | Perplexity of TransformerXL LM\u003cbr\u003e(on validation set) |                                                                                                                                                                                    Dataset used for Classification                                                                                                                                                                                    |  Classification:\u003cbr\u003e Test set Accuracy  |   Classification: \u003cbr\u003eTest set MCC   |                                                                                                                                                                                                  Classification: Notebook\u003cbr\u003efor Reproducibility                                                                                                                                                                                                 |                                                                                      ULMFiT Embeddings visualization                                                                                     |                                                                                  TransformerXL Embeddings visualization                                                                                  |\n|:---------:|:----------------------------------------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:----------------------------------------------:|:-----------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------:|:------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|\n|   Hindi   |     [NLP for Hindi](https://github.com/goru001/nlp-for-hindi)    | [Hindi Wikipedia Articles - 172k](https://www.kaggle.com/disisbig/hindi-wikipedia-articles-172k)\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e[Hindi Wikipedia Articles - 55k](https://www.kaggle.com/disisbig/hindi-wikipedia-articles-55k) |             34.06\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e35.87             |                 26.09\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e34.78                | [BBC News Articles](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#publicly-available-classification-datasets)\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e[IIT Patna Movie Reviews](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#publicly-available-classification-datasets)\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e[IIT Patna Product Reviews](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#publicly-available-classification-datasets) | 78.75\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e57.74\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e75.71 | 0.71\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e0.37\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e0.59 | [Notebook](https://github.com/goru001/nlp-for-hindi/blob/master/classification-benchmarks/Hindi_Classification_Model_BBC_Articles.ipynb)\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e[Notebook](https://github.com/goru001/nlp-for-hindi/blob/master/classification-benchmarks/Hindi_Classification_Model_IITP%2BMovie.ipynb)\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e[Notebook](https://github.com/goru001/nlp-for-hindi/blob/master/classification-benchmarks/Hindi_Classification_Model_IITP_Product.ipynb) |         [Hindi Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-hindi/master/language-model/embedding_projector_config_30k.json)        |    [Hindi Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-hindi/master/language-model/embedding_projector_config_transformerxl.json)   |\n|  Bengali  |   [NLP for Bengali](https://github.com/goru001/nlp-for-bengali)  |                                                          [Bengali Wikipedia Articles](https://www.kaggle.com/disisbig/bengali-wikipedia-articles)                                                          |                      41.2                      |                          39.3                         |                                                                                                                               [Bengali News Articles (Soham Articles)](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#publicly-available-classification-datasets)                                                                                                                              |                  90.71                  |                 0.87                 |                                                                                                                                                               [Notebook](https://github.com/goru001/nlp-for-bengali/blob/master/classification/Bengali_Classification_Model.ipynb)                                                                                                                                                               |         [Bengali Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-bengali/master/language-model/embedding_projector_config.json)        |   [Bengali Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-bengali/master/language-model/embedding_projector_transformer_config.json)  |\n|  Gujarati |  [NLP for Gujarati](https://github.com/goru001/nlp-for-gujarati) |                                                         [Gujarati Wikipedia Articles](https://www.kaggle.com/disisbig/gujarati-wikipedia-articles)                                                         |                      34.12                     |                         28.12                         |                                                                                                                                 [iNLTK Headlines Corpus - Gujarati](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#publicly-available-classification-datasets)                                                                                                                                 |                  91.05                  |                 0.86                 |                                                                                                                                                              [Notebook](https://github.com/goru001/nlp-for-gujarati/blob/master/classification/Gujarati_Classification_Model.ipynb)                                                                                                                                                              |        [Gujarati Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-gujarati/master/language-model/embedding_projector_config.json)       |  [Gujarati Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-gujarati/master/language-model/embedding_projector_transformer_config.json) |\n| Malayalam | [NLP for Malayalam](https://github.com/goru001/nlp-for-malyalam) |                                                        [Malayalam Wikipedia Articles](https://www.kaggle.com/disisbig/malayalam-wikipedia-articles)                                                        |                      26.39                     |                         25.79                         |                                                                                                                                 [iNLTK Headlines Corpus - Malayalam](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#publicly-available-classification-datasets)                                                                                                                                |                  95.56                  |                 0.93                 |                                                                                                                                                              [Notebook](https://github.com/goru001/nlp-for-malyalam/blob/master/classification/Malyalam_Classification_Model.ipynb)                                                                                                                                                              |       [Malayalam Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-malyalam/master/language-model/embedding_projector_config.json)       | [Malayalam Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-malyalam/master/language-model/embedding_projector_transformer_config.json) |\n|  Marathi  |   [NLP for Marathi](https://github.com/goru001/nlp-for-marathi)  |                                                          [Marathi Wikipedia Articles](https://www.kaggle.com/disisbig/marathi-wikipedia-articles)                                                          |                       18                       |                         17.42                         |                                                                                                                                  [iNLTK Headlines Corpus - Marathi](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#publicly-available-classification-datasets)                                                                                                                                 |                  92.40                  |                 0.85                 |                                                                                                                                                               [Notebook](https://github.com/goru001/nlp-for-marathi/blob/master/classification/Marathi_Classification_Model.ipynb)                                                                                                                                                               |         [Marathi Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-marathi/master/language-model/embedding_projector_config.json)        |   [Marathi Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-marathi/master/language-model/embedding_projector_transformer_config.json)  |\n|   Tamil   |     [NLP for Tamil](https://github.com/goru001/nlp-for-tamil)    |                                                            [Tamil Wikipedia Articles](https://www.kaggle.com/disisbig/tamil-wikipedia-articles)                                                            |                      19.80                     |                         17.22                         |                                                                                                                                   [iNLTK Headlines Corpus - Tamil](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#publicly-available-classification-datasets)                                                                                                                                  |                  95.22                  |                 0.92                 |                                                                                                                                                                      [Notebook](https://github.com/goru001/nlp-for-tamil/blob/master/classification/Tamil_Classifier.ipynb)                                                                                                                                                                      |           [Tamil Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-tamil/master/language-model/embedding_projector_config.json)          |     [Tamil Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-tamil/master/language-model/embedding_projector_transformer_config.json)    |\n|  Punjabi  |   [NLP for Punjabi](https://github.com/goru001/nlp-for-punjabi)  |                                                          [Punjabi Wikipedia Articles](https://www.kaggle.com/disisbig/punjabi-wikipedia-articles)                                                          |                      24.40                     |                         14.03                         |                                                                                                                      [IndicNLP News Article Classification Dataset - Punjabi](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#indicnlp-news-article-classification-dataset)                                                                                                                     |                  97.12                  |                 0.96                 |                                                                                                                                                               [Notebook](https://github.com/goru001/nlp-for-punjabi/blob/master/classification/Panjabi_Classification_Model.ipynb)                                                                                                                                                               |         [Punjabi Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-punjabi/master/language-model/embedding_projector_config.json)        |   [Punjabi Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-punjabi/master/language-model/embedding_projector_transformer_config.json)  |\n|  Kannada  |   [NLP for Kannada](https://github.com/goru001/nlp-for-kannada)  |                                                          [Kannada Wikipedia Articles](https://www.kaggle.com/disisbig/kannada-wikipedia-articles)                                                          |                      70.10                     |                         61.97                         |                                                                                                                      [IndicNLP News Article Classification Dataset - Kannada](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#indicnlp-news-article-classification-dataset)                                                                                                                     |                  98.87                  |                 0.98                 |                                                                                                                                                               [Notebook](https://github.com/goru001/nlp-for-kannada/blob/master/classification/Kannada_Classification_Model.ipynb)                                                                                                                                                               |         [Kannada Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-kannada/master/language-model/embedding_projector_config.json)        |   [Kannada Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-kannada/master/language-model/embedding_projector_transformer_config.json)  |\n|   Oriya   |     [NLP for Oriya](https://github.com/goru001/nlp-for-odia)     |                                                             [Oriya Wikipedia Articles](https://www.kaggle.com/disisbig/odia-wikipedia-articles)                                                            |                      26.57                     |                         26.81                         |                                                                                                                       [IndicNLP News Article Classification Dataset - Oriya](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#indicnlp-news-article-classification-dataset)                                                                                                                      |                  98.83                  |                 0.98                 |                                                                                                                                                                  [Notebook](https://github.com/goru001/nlp-for-odia/blob/master/classification/Oriya_Classification_Model.ipynb)                                                                                                                                                                 |           [Oriya Embeddings Projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-odia/master/language-model/embedding_projector_config.json)           |     [Oriya Embeddings Projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-odia/master/language-model/embedding_projector_transformer_config.json)     |\n|  Sanskrit |  [NLP for Sanskrit](https://github.com/goru001/nlp-for-sanskrit) |                                                         [Sanskrit Wikipedia Articles](https://www.kaggle.com/disisbig/sanskrit-wikipedia-articles)                                                         |                       ~6                       |                           ~3                          |                                                                                                                                                          [Sanskrit Shlokas Dataset](https://www.kaggle.com/disisbig/sanskrit-shlokas-dataset)                                                                                                                                                         |             84.3 (valid set)            |                                      |                                                                                                                                                                                                                                                                                                                                                                                                                                                  |        [Sanskrit Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-sanskrit/master/language-model/embedding_projector_config.json)       |  [Sanskrit Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-sanskrit/master/language-model/embedding_projector_transformer_config.json) |\n|   Nepali  |    [NLP for Nepali](https://github.com/goru001/nlp-for-nepali)   |                                                           [Nepali Wikipedia Articles](https://www.kaggle.com/disisbig/nepali-wikipedia-articles)                                                           |                      31.5                      |                          29.3                         |                                                                                                                                                               [Nepali News Dataset](https://www.kaggle.com/disisbig/nepali-news-dataset)                                                                                                                                                              |             98.5 (valid set)            |                                      |                                                                                                                                                                                                                                                                                                                                                                                                                                                  |          [Nepali Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-nepali/master/language-model/embedding_projector_config.json)         |    [Nepali Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-nepali/master/language-model/embedding_projector_transformer_config.json)   |\n|    Urdu   |    [NLP for Urdu](https://github.com/anuragshas/nlp-for-urdu)    |                                                             [Urdu Wikipedia Articles](https://www.kaggle.com/disisbig/urdu-wikipedia-articles)                                                             |                      13.19                     |                         12.55                         |                                                                                                                                                                 [Urdu News Dataset](https://www.kaggle.com/disisbig/urdu-news-dataset)                                                                                                                                                                |            95.28 (valid set)            |                                      |                                                                                                                                                                                                                                                                                                                                                                                                                                                  |          [Urdu Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/anuragshas/nlp-for-urdu/master/language-model/embedding_projector_config.json)          |    [Urdu Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/anuragshas/nlp-for-urdu/master/language-model/embedding_projector_transformer_config.json)    |\n|   Telugu  | [NLP for Telugu](https://github.com/Shubhamjain27/nlp-for-telugu)    |                                                             [Telugu Wikipedia Articles](https://www.kaggle.com/shubhamjain27/telugu-wikipedia-articles)                                                |                       27.47                    |                         29.44                         |                                                                                                                                              [Telugu News Dataset](https://www.kaggle.com/shubhamjain27/telugu-news-articles)\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e[Telugu News Andhra Jyoti](https://www.kaggle.com/shubhamjain27/telugu-newspaperdata)                                                                                                                                                                         |              95.4\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e92.09                       |                                     | [Notebook](https://github.com/Shubhamjain27/nlp-for-telugu/tree/master/classification/Telugu_Classification_Model.ipynb) \u003cbr\u003e\u003cbr\u003e\u003cbr\u003e[Notebook](https://github.com/Shubhamjain27/nlp-for-telugu/tree/master/classification/Telugu_news_classification_Andhra_Jyoti.ipynb)                                                                                                                                                                                                                                                                                                    |                        [Telugu Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/Shubhamjain27/nlp-for-telugu/master/language-model/embedding_projector_config.json)    |    [Telugu Embeddings projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/Shubhamjain27/nlp-for-telugu/master/language-model/embedding_projector_transformer_config.json)    |\n|  Tanglish |  [NLP for Tanglish](https://github.com/goru001/nlp-for-tanglish) |                                             [Synthetic Tanglish Dataset](https://drive.google.com/drive/folders/1M4Sx_clF0iP1y-JG3OhfacFKTDoHXCR1?usp=sharing)                                             |                      37.50                     |                           -                           |                                                                                                                                                      Dravidian Codemix HASOC @ FIRE 2020\u003cbr\u003e\u003cbr\u003eDravidian Codemix Sentiment Analysis @ FIRE 2020                                                                                                                                                      |   F1 Score: 0.88\u003cbr\u003e\u003cbr\u003eF1 Score: 0.62  |                   -                  |                                                                                                 [Notebook](https://github.com/goru001/nlp-for-tanglish/blob/master/classification/classification_model_hasoc.ipynb)\u003cbr\u003e\u003cbr\u003e[Notebook](https://github.com/goru001/nlp-for-tanglish/blob/master/classification/classification_model_dc_fire.ipynb)                                                                                                 |        [Tanglish Embeddings Projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-tanglish/master/language-model/embedding_projector_config.json)       |                                                                                                     -                                                                                                    |\n|  Manglish |  [NLP for Manglish](https://github.com/goru001/nlp-for-manglish) |                                             [Synthetic Manglish Dataset](https://drive.google.com/drive/folders/1M4Sx_clF0iP1y-JG3OhfacFKTDoHXCR1?usp=sharing)                                             |                      45.84                     |                           -                           |                                                                                                                                                      Dravidian Codemix HASOC @ FIRE 2020\u003cbr\u003e\u003cbr\u003eDravidian Codemix Sentiment Analysis @ FIRE 2020                                                                                                                                                      |   F1 Score: 0.74\u003cbr\u003e\u003cbr\u003eF1 Score: 0.69  |                   -                  |                                                                                                 [Notebook](https://github.com/goru001/nlp-for-manglish/blob/master/classification/classification_model_hasoc.ipynb)\u003cbr\u003e\u003cbr\u003e[Notebook](https://github.com/goru001/nlp-for-manglish/blob/master/classification/classification_model_dc_fire.ipynb)                                                                                                 | [Manglish Embeddings Projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-manglish/master/language-model/embedding_projector_config_latin_script.json) |                                                                                                     -                                                                                                    |\n| Hinglish  | [NLP for Hinglish](https://github.com/goru001/nlp-for-hinglish)  | [Synthetic Hinglish Dataset](https://www.dropbox.com/sh/as5fg8jsrljt6k7/AADnSLlSNJPeAndFycJGurOUa?dl=0)                                                                                                    | 86.48                                          | -                                                     | -                                                                                                                                                                                                                                                                                                                                                                                                     | -                                       | -                                    | -                                                                                                                                                                                                                                                                                                                                                                                                                                                | [Hinglish Embeddings Projection](https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/goru001/nlp-for-hinglish/main/language_model/embedding_projector_config.json)                | -                                                                                                                                                                                                        |\n\n\nNote: English model has been directly taken from [fast.ai](https://github.com/fastai/fastai)\n\n#### Effect of using Transfer Learning + Paraphrases from iNLTK\n\n|  Language |                            Repository                            |                                                       Dataset used for Classification                                                      | Results on using\u003cbr\u003ecomplete training set | Percentage Decrease \u003cbr\u003ein Training set size | Results on using\u003cbr\u003ereduced training set\u003cbr\u003ewithout Paraphrases | Results on using\u003cbr\u003ereduced training set\u003cbr\u003ewith Paraphrases |\n|:---------:|:----------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------------------------:|:--------------------------------------------:|:------------------------------------------------------------:|:---------------------------------------------------------:|\n|   Hindi   |     [NLP for Hindi](https://github.com/goru001/nlp-for-hindi)    |         [IIT Patna Movie Reviews](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#publicly-available-classification-datasets)        |     Accuracy: 57.74\u003cbr\u003e\u003cbr\u003eMCC: 37.23     |               80% (2480 -\u003e 496)              |               Accuracy: 47.74\u003cbr\u003e\u003cbr\u003eMCC: 20.50              |             Accuracy: 56.13\u003cbr\u003e\u003cbr\u003eMCC: 34.39             |\n|  Bengali  |   [NLP for Bengali](https://github.com/goru001/nlp-for-bengali)  | [Bengali News Articles (Soham Articles)](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#publicly-available-classification-datasets) |     Accuracy: 90.71\u003cbr\u003e\u003cbr\u003eMCC: 87.92     |              99% (11284 -\u003e 112)              |               Accuracy: 69.88\u003cbr\u003e\u003cbr\u003eMCC: 61.56              |             Accuracy: 74.06\u003cbr\u003e\u003cbr\u003eMCC: 65.08             |\n|  Gujarati |  [NLP for Gujarati](https://github.com/goru001/nlp-for-gujarati) |    [iNLTK Headlines Corpus - Gujarati](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#publicly-available-classification-datasets)   |     Accuracy: 91.05\u003cbr\u003e\u003cbr\u003eMCC: 86.09     |               90% (5269 -\u003e 526)              |               Accuracy: 80.88\u003cbr\u003e\u003cbr\u003eMCC: 70.18              |             Accuracy: 81.03\u003cbr\u003e\u003cbr\u003eMCC: 70.44             |\n| Malayalam | [NLP for Malayalam](https://github.com/goru001/nlp-for-malyalam) |   [iNLTK Headlines Corpus - Malayalam](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#publicly-available-classification-datasets)   |     Accuracy: 95.56\u003cbr\u003e\u003cbr\u003eMCC: 93.29     |               90% (5036 -\u003e 503)              |               Accuracy: 82.38\u003cbr\u003e\u003cbr\u003eMCC: 73.47              |             Accuracy: 84.29\u003cbr\u003e\u003cbr\u003eMCC: 76.36             |\n|  Marathi  |   [NLP for Marathi](https://github.com/goru001/nlp-for-marathi)  |    [iNLTK Headlines Corpus - Marathi](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#publicly-available-classification-datasets)    |     Accuracy: 92.40\u003cbr\u003e\u003cbr\u003eMCC: 85.23     |               95% (9672 -\u003e 483)              |               Accuracy: 84.13\u003cbr\u003e\u003cbr\u003eMCC: 68.59              |             Accuracy: 84.55\u003cbr\u003e\u003cbr\u003eMCC: 69.11             |\n|   Tamil   |     [NLP for Tamil](https://github.com/goru001/nlp-for-tamil)    |     [iNLTK Headlines Corpus - Tamil](https://github.com/ai4bharat-indicnlp/indicnlp_corpus#publicly-available-classification-datasets)     |     Accuracy: 95.22\u003cbr\u003e\u003cbr\u003eMCC: 92.70     |               95% (5346 -\u003e 267)              |               Accuracy: 86.25\u003cbr\u003e\u003cbr\u003eMCC: 79.42              |             Accuracy: 89.84\u003cbr\u003e\u003cbr\u003eMCC: 84.63             |\n\nFor more details around implementation or to reproduce results, checkout respective repositories. \n\n### Contributing\n\n##### Add a new language support\n\nIf you would like to add support for language of your own choice to iNLTK,\n please start with checking/raising a issue [here](https://github.com/goru001/inltk/issues)\n \nPlease checkout the steps I'd [mentioned here for Telugu](https://github.com/goru001/inltk/issues/1)\nto begin with. They should be almost similar for other languages as well.\n\n##### Improving models/using models for your own research\n\nIf you would like to take iNLTK's models and refine them with your own \ndataset or build your own custom models on top of it, please check out the \nrepositories in the above table for the language of your choice. The repositories above \ncontain links to datasets, pretrained models, classifiers and all of the code for that.\n\n##### Add new functionality\n\nIf you wish for a particular functionality in iNLTK - Start by checking/raising a issue [here](https://github.com/goru001/inltk/issues)\n\n\n### What's next\n\n\n#### ..and being worked upon\n`Shout out if you want to help :)`\n\n* Add [Maithili](https://github.com/goru001/inltk/issues/10) support\n\n\n#### ..and NOT being worked upon\n\n`Shout out if you want to lead :)`\n\n* Add NER support for all languages\n* Add Textual Entailment support for all languages\n* Work on a [unified model for all the languages](https://github.com/goru001/inltk/issues/14)\n* [POS support](https://github.com/goru001/inltk/issues/13) in iNLTK\n* Add translations - to and from languages in iNLTK + English\n\n\n\n### iNLTK's Appreciation\n\n* [By Jeremy Howard on Twitter](https://twitter.com/jeremyphoward/status/1111318198891110402)\n* [By Sebastian Ruder on Twitter](https://twitter.com/seb_ruder/status/1207074241830674438)\n* [By Vincent Boucher](https://www.linkedin.com/feed/update/urn:li:activity:6517137647310241792/), [By Philip Vollet](https://www.linkedin.com/posts/philipvollet_machinelearning-datascience-nlp-activity-6698220942910468096-phA-), [By Steve Nouri](https://www.linkedin.com/posts/stevenouri_india-artificialintelligence-technology-activity-6698815315498868736-vYmZ) on [LinkedIn](https://www.linkedin.com/search/results/content/?keywords=inltk)\n* [By Kanimozhi](https://www.linkedin.com/feed/update/urn:li:activity:6517277916030701568), [By Soham](https://www.linkedin.com/feed/update/urn:li:activity:6513084638955696128), [By Imaad](https://www.linkedin.com/feed/update/urn:li:activity:6536258026687557632/) on [LinkedIn](https://www.linkedin.com/search/results/content/?keywords=inltk)\n* iNLTK was [trending on GitHub](https://github.motakasoft.com/trending/ranking/monthly/?d=2019-05-01\u0026l=python\u0026page=2) in May 2019\n\n\n### Citation\n\nIf you use this library in your research, please consider citing:\n\n```latex\n@inproceedings{arora-2020-inltk,\n    title = \"i{NLTK}: Natural Language Toolkit for Indic Languages\",\n    author = \"Arora, Gaurav\",\n    booktitle = \"Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)\",\n    month = nov,\n    year = \"2020\",\n    address = \"Online\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://www.aclweb.org/anthology/2020.nlposs-1.10\",\n    doi = \"10.18653/v1/2020.nlposs-1.10\",\n    pages = \"66--71\",\n    abstract = \"We present iNLTK, an open-source NLP library consisting of pre-trained language models and out-of-the-box support for Data Augmentation, Textual Similarity, Sentence Embeddings, Word Embeddings, Tokenization and Text Generation in 13 Indic Languages. By using pre-trained models from iNLTK for text classification on publicly available datasets, we significantly outperform previously reported results. On these datasets, we also show that by using pre-trained models and data augmentation from iNLTK, we can achieve more than 95{\\%} of the previous best performance by using less than 10{\\%} of the training data. iNLTK is already being widely used by the community and has 40,000+ downloads, 600+ stars and 100+ forks on GitHub. The library is available at https://github.com/goru001/inltk.\",\n}\n```\n","funding_links":[],"categories":["Libraries","Python","\u003ca name='Libraries'\u003e\u003c/a\u003eLibraries and Tools"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoru001%2Finltk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoru001%2Finltk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoru001%2Finltk/lists"}