{"id":15014196,"url":"https://github.com/ljvmiranda921/calamancy","last_synced_at":"2025-04-06T12:08:29.960Z","repository":{"id":65865557,"uuid":"552637690","full_name":"ljvmiranda921/calamanCy","owner":"ljvmiranda921","description":"NLP pipelines for Tagalog using spaCy ","archived":false,"fork":false,"pushed_at":"2025-02-06T18:13:35.000Z","size":1001,"stargazers_count":52,"open_issues_count":1,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-06T12:08:24.554Z","etag":null,"topics":["computational-linguistics","low-resource-languages","low-resource-nlp","machine-learning","natural-language-processing","ner","nlp","spacy"],"latest_commit_sha":null,"homepage":"https://ljvmiranda921.github.io/calamanCy/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ljvmiranda921.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-17T01:40:37.000Z","updated_at":"2025-03-26T11:25:49.000Z","dependencies_parsed_at":"2023-10-17T06:20:59.851Z","dependency_job_id":"6a09fcd0-5819-41ed-b0f4-78fec2f12e78","html_url":"https://github.com/ljvmiranda921/calamanCy","commit_stats":{"total_commits":229,"total_committers":3,"mean_commits":76.33333333333333,"dds":0.008733624454148492,"last_synced_commit":"8cc7d86775e687192f5a893225885bffb22529ae"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ljvmiranda921%2FcalamanCy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ljvmiranda921%2FcalamanCy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ljvmiranda921%2FcalamanCy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ljvmiranda921%2FcalamanCy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ljvmiranda921","download_url":"https://codeload.github.com/ljvmiranda921/calamanCy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247478323,"owners_count":20945266,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computational-linguistics","low-resource-languages","low-resource-nlp","machine-learning","natural-language-processing","ner","nlp","spacy"],"created_at":"2024-09-24T19:45:18.731Z","updated_at":"2025-04-06T12:08:29.941Z","avatar_url":"https://github.com/ljvmiranda921.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"https://raw.githubusercontent.com/ljvmiranda921/calamanCy/master/logo.png\" width=\"125\" height=\"125\" align=\"right\" /\u003e\n\n# calamanCy: NLP pipelines for Tagalog\n\n[![GitHub workflow](https://github.com/ljvmiranda921/calamancy/actions/workflows/test.yml/badge.svg)](https://github.com/ljvmiranda921/calamanCy/actions/workflows/test.yml)\n[![PyPI](https://img.shields.io/pypi/v/calamancy?labelColor=%23272c32\u0026color=%2333cc56\u0026logo=pypi\u0026logoColor=white)](https://pypi.org/project/calamanCy/)\n[![Paper](https://img.shields.io/badge/read-EMNLP%20paper-blue?logo=semanticscholar)](https://www.semanticscholar.org/reader/598a4f17da8f4cddfab3c7c10bc96a05078e7a91)\n\n**calamanCy** is a Tagalog natural language preprocessing framework made with\n[spaCy](https://spacy.io). Its goal is to provide pipelines and datasets for\ndownstream NLP tasks. This repository contains material for using calamanCy,\nreproduction of results, and guides on usage.\n\n\u003e calamanCy takes inspiration from other language-specific [spaCy Universe frameworks](https://spacy.io/universe) such as\n\u003e [DaCy](https://github.com/centre-for-humanities-computing/DaCy), [huSpaCy](https://github.com/huspacy/huspacy),\n\u003e and [graCy](https://github.com/jmyerston/graCy). The name is based from [_calamansi_](https://en.wikipedia.org/wiki/Calamansi),\n\u003e a citrus fruit native to the Philippines and used in traditional Filipino cuisine.\n\n🌐 **Website**: [https://ljvmiranda921.github.io/calamanCy](https://ljvmiranda921.github.io/calamanCy)\n\n## 📰 News\n\n- [2025-01-19] Released v0.2.0 models with significantly improved performance on syntactic parsing and NER! All thanks to the newly-released [UD-NewsCrawl](https://huggingface.co/datasets/UD-Filipino/UD_Tagalog-NewsCrawl) treebank! See full changes in this [blogpost](https://ljvmiranda921.github.io/calamanCy/news/release-v020/).\n- [2024-08-01] Released new NER-only models based on [GLiNER](https://github.com/urchade/GLiNER)! You can find the models in [this HuggingFace collection](https://huggingface.co/collections/ljvmiranda921/calamancy-models-for-tagalog-nlp-65629cc46ef2a1d0f9605c87). Span-Marker and calamanCy models are still superior, but GLiNER offers a lot of extensibility on unseen entity labels. You can find the training pipeline [here](https://github.com/ljvmiranda921/calamanCy/tree/master/models/v0.1.0-gliner).\n- [2024-07-02] I talked about calamanCy during my guest lecture, \"Artisanal Filipino NLP Resources in the time of Large Language Models,\" @ DLSU Manila. You can find the slides (and an accompanying blog post) [here](https://ljvmiranda921.github.io/notebook/2024/07/02/talk-dlsu/).\n- [2023-12-05] We released the paper [**calamanCy: A Tagalog Natural Language Processing Toolkit**](https://aclanthology.org/2023.nlposs-1.1/) and will be presented in the NLP-OSS workshop at EMNLP 2023! Feel free to check out the [Tagalog NLP collection in HuggingFace](https://huggingface.co/collections/ljvmiranda921/calamancy-models-for-tagalog-nlp-65629cc46ef2a1d0f9605c87).\n- [2023-11-01] The named entity recognition (NER) dataset used to train the NER component of calamanCy has now a corresponding paper: [**Developing a Named Entity Recognition Dataset for Tagalog**](https://aclanthology.org/2023.nlposs-1.1/)! It will be presented in the SEALP workshop at IJCNLP-AACL 2023! The dataset is also available [in HuggingFace](https://huggingface.co/datasets/ljvmiranda921/tlunified-ner). I've also talked about my thoughts on the annotation process [in my blog](https://ljvmiranda921.github.io/notebook/2023/07/03/devlog-calamancy/).\n- [2023-08-01] First release of calamanCy! Please check out [this blog post](https://ljvmiranda921.github.io/projects/2023/08/01/calamancy/) to learn more and read some of my preliminary work back in February [here](https://ljvmiranda921.github.io/notebook/2023/02/04/tagalog-pipeline/).\n\n## 🔧 Installation\n\nTo get started with calamanCy, simply install it using `pip` by running the\nfollowing line in your terminal:\n\n```sh\npip install calamanCy\n```\n\n### Development\n\nIf you are developing calamanCy, first clone the repository:\n\n```sh\ngit clone git@github.com:ljvmiranda921/calamanCy.git\n```\n\nThen, create a virtual environment and install the dependencies:\n\n```sh\npython -m venv venv\nvenv/bin/pip install -e .  # requires pip\u003e=23.0\nvenv/bin/pip install .[dev]\n\n# Activate the virtual environment\nsource venv/bin/activate\n```\n\nor alternatively, use `make dev`.\n\n### Running the tests\n\nWe use [pytest](https://docs.pytest.org/en/7.4.x/) as our test runner:\n\n```sh\npython -m pytest --pyargs calamancy\n```\n\n## 👩‍💻 Usage\n\nTo use calamanCy you first have to download either the medium, large, or\ntransformer model. To see a list of all available models, run:\n\n```python\nimport calamancy\nfrom model in calamancy.models():\n    print(model)\n\n# ..\n# tl_calamancy_md-0.1.0\n# tl_calamancy_lg-0.1.0\n# tl_calamancy_trf-0.1.0\n```\n\nTo download and load a model, run:\n\n```python\nnlp = calamancy.load(\"tl_calamancy_md-0.1.0\")\ndoc = nlp(\"Ako si Juan de la Cruz\")\n```\n\nThe `nlp` object is an instance of spaCy's [`Language`\nclass](https://spacy.io/api/language) and you can use it as any other spaCy\npipeline. You can also [access these models on Hugging Face](https://huggingface.co/ljvmiranda921) 🤗.\n\n## 📦 Models and Datasets\n\ncalamanCy provides Tagalog models and datasets that you can use in your spaCy\npipelines. You can download them directly or use the `calamancy` Python library\nto access them. The training procedure for each pipeline can be found in the\n`models/` directory. They are further subdivided into versions. Each folder is\nan instance of a [spaCy project](https://spacy.io/usage/projects).\n\nHere are the models for the latest release:\n\n| Model                       | Pipelines                                   | Description                                                                                                  |\n| --------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |\n| tl_calamancy_md (73.7 MB)   | tok2vec, tagger, morphologizer, parser, ner | CPU-optimized Tagalog NLP model. Pretrained using the TLUnified dataset. Using floret vectors (50k keys)     |\n| tl_calamancy_lg (431.9 MB)  | tok2vec, tagger, morphologizer, parser, ner | CPU-optimized large Tagalog NLP model. Pretrained using the TLUnified dataset. Using fastText vectors (714k) |\n| tl_calamancy_trf (775.6 MB) | transformer, tagger, parser, ner            | GPU-optimized transformer Tagalog NLP model. Uses roberta-tagalog-base as context vectors.                   |\n\n## 📓 API\n\nThe calamanCy library contains utility functions that help you load its models\nand infer on your text. You can think of these functions as \"syntactic sugar\"\nto the spaCy API. We highly recommend checking out the [spaCy Doc\nobject](https://spacy.io/api/doc), as it provides the most flexibility.\n\n### Loaders\n\nThe loader functions provide an easier interface to download calamanCy models.\nThese models are hosted on [HuggingFace](https://huggingface.co/ljvmiranda921)\nso you can try them out first before downloading.\n\n#### \u003ckbd\u003efunction\u003c/kbd\u003e `get_latest_version`\n\nReturn the latest version of a calamanCy model.\n\n| Argument    | Type  | Description                      |\n| ----------- | ----- | -------------------------------- |\n| `model`     | `str` | The string indicating the model. |\n| **RETURNS** | `str` | The latest version of the model. |\n\n#### \u003ckbd\u003efunction\u003c/kbd\u003e `models`\n\nGet a list of valid calamanCy models.\n\n| Argument    | Type        | Description                    |\n| ----------- | ----------- | ------------------------------ |\n| **RETURNS** | `List[str]` | List of valid calamanCy models |\n\n#### \u003ckbd\u003efunction\u003c/kbd\u003e `load`\n\nLoad a calamanCy model as a [spaCy language pipeline](https://spacy.io/usage/processing-pipelines).\n\n| Argument    | Type                                        | Description                                                                                  |\n| ----------- | ------------------------------------------- | -------------------------------------------------------------------------------------------- |\n| `model`     | `str`                                       | The model to download. See the available models at [`calamancy.models()`](#function-models). |\n| `force`     | `bool`                                      | Force download the model. Defaults to `False`.                                               |\n| `**kwargs`  | `dict`                                      | Additional arguments to `spacy.load()`.                                                      |\n| **RETURNS** | [`Language`](https://spacy.io/api/language) | A spaCy language pipeline.                                                                   |\n\n### Inference\n\nBelow are lightweight utility classes for users who are not familiar with spaCy's\nprimitives. They are only useful for inference and not for training. If you wish\nto train on top of these calamanCy models (e.g., text categorization,\ntask-specific NER, etc.), we advise you to follow the standard [spaCy training\nworkflow](https://spacy.io/usage/training).\n\nGeneral usage: first, you need to instantiate a class with the name of a model.\nThen, you can use the `__call__` method to perform the prediction. The output\nis of the type `Iterable[Tuple[str, Any]]` where the first part of the tuple\nis the token and the second part is its label.\n\n#### \u003ckbd\u003emethod\u003c/kbd\u003e `EntityRecognizer.__call__`\n\nPerform named entity recognition (NER). By default, it uses the v0.1.0 of\n[TLUnified-NER](https://huggingface.co/datasets/ljvmiranda921/tlunified-ner)\nwith the following entity labels: _PER (Person), ORG (Organization), LOC\n(Location)._\n\n| Argument   | Type                        | Description                             |\n| ---------- | --------------------------- | --------------------------------------- |\n| `text`     | `str`                       | The text to get the entities from.      |\n| **YIELDS** | `Iterable[Tuple[str, str]]` | the token and its entity in IOB format. |\n\n#### \u003ckbd\u003emethod\u003c/kbd\u003e `Tagger.__call__`\n\nPerform parts-of-speech tagging. It uses the annotations from the\n[TRG](https://universaldependencies.org/treebanks/tl_trg/index.html) and\n[Ugnayan](https://universaldependencies.org/treebanks/tl_ugnayan/index.html)\ntreebanks with the following tags: _ADJ, ADP, ADV, AUX, DET, INTJ, NOUN, PART,\nPRON, PROPN, PUNCT, SCONJ, VERB._\n\n| Argument   | Type                                    | Description                                         |\n| ---------- | --------------------------------------- | --------------------------------------------------- |\n| `text`     | `str`                                   | The text to get the POS tags from.                  |\n| **YIELDS** | `Iterable[Tuple[str, Tuple[str, str]]]` | the token and its coarse- and fine-grained POS tag. |\n\n#### \u003ckbd\u003emethod\u003c/kbd\u003e `Parser.__call__`\n\nPerform syntactic dependency parsing. It uses the annotations from the\n[TRG](https://universaldependencies.org/treebanks/tl_trg/index.html) and\n[Ugnayan](https://universaldependencies.org/treebanks/tl_ugnayan/index.html) treebanks.\n\n| Argument   | Type                        | Description                                    |\n| ---------- | --------------------------- | ---------------------------------------------- |\n| `text`     | `str`                       | The text to get the dependency relations from. |\n| **YIELDS** | `Iterable[Tuple[str, str]]` | the token and its dependency relation.         |\n\n## 📝 Reporting Issues\n\nIf you have questions regarding the usage of `calamanCy`, bug reports, or just\nwant to give us feedback after giving it a spin, please use the [Issue\ntracker](https://github.com/ljvmiranda921/calamancy/issues). Thank you!\n\n## 📜 Citation\n\nIf you are citing the open-source software, please use:\n\n```bib\n@inproceedings{miranda-2023-calamancy,\n    title = \"calaman{C}y: A {T}agalog Natural Language Processing Toolkit\",\n    author = \"Miranda, Lester James\",\n    editor = \"Tan, Liling  and\n      Milajevs, Dmitrijs  and\n      Chauhan, Geeticka  and\n      Gwinnup, Jeremy  and\n      Rippeth, Elijah\",\n    booktitle = \"Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)\",\n    month = dec,\n    year = \"2023\",\n    address = \"Singapore, Singapore\",\n    publisher = \"Empirical Methods in Natural Language Processing\",\n    url = \"https://aclanthology.org/2023.nlposs-1.1\",\n    pages = \"1--7\",\n    abstract = \"We introduce calamanCy, an open-source toolkit for constructing natural language processing (NLP) pipelines for Tagalog. It is built on top of spaCy, enabling easy experimentation and integration with other frameworks. calamanCy addresses the development gap by providing a consistent API for building NLP applications and offering general-purpose multitask models with out-of-the-box support for dependency parsing, parts-of-speech (POS) tagging, and named entity recognition (NER). calamanCy aims to accelerate the progress of Tagalog NLP by consolidating disjointed resources in a unified framework.The calamanCy toolkit is available on GitHub: https://github.com/ljvmiranda921/calamanCy.\",\n}\n```\n\nIf you are citing the [NER dataset](https://huggingface.co/ljvmiranda921), please use:\n\n```bib\n@inproceedings{miranda-2023-developing,\n    title = \"Developing a Named Entity Recognition Dataset for {T}agalog\",\n    author = \"Miranda, Lester James\",\n    editor = \"Wijaya, Derry  and\n      Aji, Alham Fikri  and\n      Vania, Clara  and\n      Winata, Genta Indra  and\n      Purwarianti, Ayu\",\n    booktitle = \"Proceedings of the First Workshop in South East Asian Language Processing\",\n    month = nov,\n    year = \"2023\",\n    address = \"Nusa Dua, Bali, Indonesia\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2023.sealp-1.2\",\n    doi = \"10.18653/v1/2023.sealp-1.2\",\n    pages = \"13--20\",\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fljvmiranda921%2Fcalamancy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fljvmiranda921%2Fcalamancy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fljvmiranda921%2Fcalamancy/lists"}