https://github.com/LGUG2Z/tashkil

A lightweight Rust library for removing Arabic diacritics
https://github.com/LGUG2Z/tashkil

arabic dari diacritics diacritics-removal pashto persian rust text-processing urdu

Last synced: 3 months ago
JSON representation

A lightweight Rust library for removing Arabic diacritics

• Host: GitHub
• URL: https://github.com/LGUG2Z/tashkil
• Owner: LGUG2Z
• License: mit
• Created: 2022-10-16T21:27:38.000Z (over 1 year ago)
• Default Branch: master
• Last Pushed: 2022-10-16T21:31:28.000Z (over 1 year ago)
• Last Synced: 2024-03-18T15:43:56.602Z (4 months ago)
• Topics: arabic, dari, diacritics, diacritics-removal, pashto, persian, rust, text-processing, urdu
• Language: Rust
• Homepage:
• Size: 2.93 KB
• Stars: 16
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Lists

• awesome-stars - tashkil

README

        
# Tashkil

A lightweight Rust library for removing Arabic diacritics (تَشْكِيل)

This library exposes a single function, `tashkil::remove()`, which removes from a `&str` all diacritics in the [unicode specification for the Arabic alphabet and its variants](https://www.unicode.org/charts/PDF/U0600.pdf).

It is my hope that this library can be used to improve search results in [Meilisearch](https://github.com/meilisearch/MeiliSearch/) for languages using the Arabic alphabet and its variants, similarly to how [`niqqud`](https://github.com/benny-n/niqqud) has been used to [improve search results for Hebrew](https://docs.meilisearch.com/learn/advanced/tokenization.html#deep-dive-the-meilisearch-tokenizer).