{"id":13772969,"url":"https://github.com/anoopkunchukuttan/indic_nlp_library","last_synced_at":"2025-05-11T05:33:58.646Z","repository":{"id":21910477,"uuid":"25234558","full_name":"anoopkunchukuttan/indic_nlp_library","owner":"anoopkunchukuttan","description":"Resources and tools for Indian language Natural Language Processing","archived":false,"fork":false,"pushed_at":"2024-06-07T07:14:41.000Z","size":9783,"stargazers_count":581,"open_issues_count":30,"forks_count":161,"subscribers_count":32,"default_branch":"master","last_synced_at":"2025-04-12T23:03:20.116Z","etag":null,"topics":["indian-languages","natural-language-processing","python"],"latest_commit_sha":null,"homepage":"http://anoopkunchukuttan.github.io/indic_nlp_library/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anoopkunchukuttan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-10-15T01:56:20.000Z","updated_at":"2025-04-10T09:47:23.000Z","dependencies_parsed_at":"2024-04-18T07:32:17.821Z","dependency_job_id":"13d43f65-f9c6-46e9-a5dd-91def62c340e","html_url":"https://github.com/anoopkunchukuttan/indic_nlp_library","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anoopkunchukuttan%2Findic_nlp_library","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anoopkunchukuttan%2Findic_nlp_library/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anoopkunchukuttan%2Findic_nlp_library/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anoopkunchukuttan%2Findic_nlp_library/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anoopkunchukuttan","download_url":"https://codeload.github.com/anoopkunchukuttan/indic_nlp_library/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253523689,"owners_count":21921815,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["indian-languages","natural-language-processing","python"],"created_at":"2024-08-03T17:01:09.741Z","updated_at":"2025-05-11T05:33:58.372Z","avatar_url":"https://github.com/anoopkunchukuttan.png","language":"Python","funding_links":[],"categories":["Open Source Projects","Generic Repositories","Generic","\u003ca name='Libraries'\u003e\u003c/a\u003eLibraries and Tools","NLP per Language"],"sub_categories":["\u003e 1k 🌟","Libraries and Tooling"],"readme":"# Indic NLP Library\n\nThe goal of the Indic NLP Library is to build Python based libraries for common text processing and Natural Language Processing in Indian languages. Indian languages share a lot of similarity in terms of script, phonology, language syntax, etc. and this library is an attempt to provide a general solution to very commonly required toolsets for Indian language text.\n\nThe library provides the following functionalities:\n\n- Text Normalization\n- Script Information\n- Word Tokenization and Detokenization\n- Sentence Splitting \n- Word Segmentation\n- Syllabification\n- Script Conversion\n- Romanization\n- Indicization\n\n**Note**: _Shatanuvadak_ translation and _BrahmiNet_ transliteration APIs are no longer supported. You can use newer [IndicTrans](https://github.com/AI4Bharat/indicTrans) translation and [IndicXlit](https://github.com/AI4Bharat/IndicXlit) transliteration models we developed at [AI4Bharat](https://ai4bharat.iitm.ac.in). In fact, you can find many state-of-the-art datasets and models on the AI4Bharat homepage.   \n\nThe data resources required by the Indic NLP Library are hosted in a different repository. These resources are required for some modules. You can download from the [Indic NLP Resources](https://github.com/anoopkunchukuttan/indic_nlp_resources) project. \n\n**If you are interested in Indian language NLP resources, you should check the [Indic NLP Catalog](https://github.com/indicnlpweb/indicnlp_catalog) for pointers.**\n\n## Pre-requisites\n\n- Python 3.x \n   - (For Python 2.x version check the tag `PYTHON_2.7_FINAL_JAN_2019`. Not actively supporting Python 2.x anymore, but will try to maintain as much compatibility as possible)\n- [Indic NLP Resources](https://github.com/anoopkunchukuttan/indic_nlp_resources)\n- [Urduhack](https://github.com/urduhack/urduhack): Needed only if Urdu normalization is required. It has other dependencies like Tensorflow.\n- Other dependencies are listed in setup.py\n\n\n## Configuration\n\n- Installation from pip:\n\n    `pip install indic-nlp-library`\n\n- If you want to use the project from the github repo, add the project to the Python Path: \n\n    - Clone this repository\n    - Install dependencies: `pip install -r requirements.txt`\n    - Run: `export PYTHONPATH=$PYTHONPATH:\u003cproject base directory\u003e`\n\n- In either case, export the path to the _Indic NLP Resources_ directory\n\n    Run: `export INDIC_RESOURCES_PATH=\u003cpath to Indic NLP resources\u003e` \n\n## Usage \n\nYou can use the Python API to access all the features of the library. Many of the most common operations are also accessible via a unified commandline API. \n\n### Getting Started\n\nCheck [this IPython Notebook](http://nbviewer.jupyter.org/url/anoopkunchukuttan.github.io/indic_nlp_library/doc/indic_nlp_examples.ipynb) for examples to use the Python API.\n  - You can find the Python 2.x Notebook [here](http://nbviewer.jupyter.org/url/anoopkunchukuttan.github.io/indic_nlp_library/doc/indic_nlp_examples_2_7.ipynb) \n\n### Documentation\n\nYou can find detailed documentation  [HERE](https://indic-nlp-library.readthedocs.io/en/latest)\n\nThis documents the Python API as well as the commandline reference. \n\n## Citing\n\nIf you use this library, please include the following citation: \n\n```\n@misc{kunchukuttan2020indicnlp,\nauthor = \"Anoop Kunchukuttan\",\ntitle = \"{The IndicNLP Library}\",\nyear = \"2020\",\nhowpublished={\\url{https://github.com/anoopkunchukuttan/indic_nlp_library/blob/master/docs/indicnlp.pdf}}\n}\n```\nYou can find the document [HERE](docs/indicnlp.pdf)\n\n## Website\n\n`http://anoopkunchukuttan.github.io/indic_nlp_library`\n\n## Author\nAnoop Kunchukuttan ([anoop.kunchukuttan@gmail.com](anoop.kunchukuttan@gmail.com))\n\n## Companies, Organizations, Projects using IndicNLP Library\n\n- [AI4Bharat-IndicNLPSuite](https://indicnlp.ai4bharat.org)\n- [The Classical Language Toolkit](http://cltk.org)\n- [Microsoft NLP Recipes](https://github.com/microsoft/nlp-recipes)\n- [Facebook M2M-100](https://github.com/pytorch/fairseq/tree/master/examples/m2m_100)\n\n## Revision Log\n\n\n0.81 : 26 May 2021 \n    \n    - Bug fix in version number extraction\n\n0.80 : 24 May 2021 \n\n    - Improved sentence splitting\n    - Bug fixes\n    - Support for Urdu Normalizer\n\n0.71 : 03 Sep 2020 \n\n    - Improved documentation\n    - Bug fixes\n\n0.7 : 02 Apr 2020:\n\n    - Unified commandline \n    - Improved documentation\n    - Added setup.py\n\n0.6 : 16 Dec 2019:\n\n    - New romanizer and indicizer\n    - Script Unifiers\n    - Improved script normalizers\n    - Added contrib directory for sample uses\n    - changed to MIT license \n\n0.5 : 03 Jun 2019: \n\n    - Improved word tokenizer to handle dates and numbers. \n    - Added sentence splitter that can handle common prefixes/honorofics and uses some heuristics.\n    - Added detokenizer\n    - Added acronym transliterator that can convert English acronyms to Brahmi-derived scripts\n\n0.4 : 28 Jan 2019: Ported to Python 3, and lots of feature additions since last release; primarily around script information, script similarity and syllabification.\n\n0.3 : 21 Oct 2014: Supports morph-analysis between Indian languages\n\n0.2 : 13 Jun 2014: Supports transliteration between Indian languages and tokenization of Indian languages \n\n0.1 : 12 Mar 2014: Initial version. Supports text normalization.\n\n## LICENSE\n\nIndic NLP Library is released under the MIT license\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanoopkunchukuttan%2Findic_nlp_library","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanoopkunchukuttan%2Findic_nlp_library","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanoopkunchukuttan%2Findic_nlp_library/lists"}