{"id":13781474,"url":"https://github.com/roshan-research/hazm","last_synced_at":"2026-02-20T00:32:40.777Z","repository":{"id":39620642,"uuid":"13956112","full_name":"roshan-research/hazm","owner":"roshan-research","description":"Persian NLP Toolkit","archived":false,"fork":false,"pushed_at":"2025-12-21T16:00:08.000Z","size":26375,"stargazers_count":1370,"open_issues_count":3,"forks_count":204,"subscribers_count":22,"default_branch":"master","last_synced_at":"2026-02-11T18:46:03.021Z","etag":null,"topics":["dependency-parser","embeddings","farsi","lemmatization","natural-language-processing","nlp","normalization","persian","persian-nlp","pos-tagging","python","text-processing","tokenizer"],"latest_commit_sha":null,"homepage":"https://www.roshan-ai.ir/hazm/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/roshan-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2013-10-29T13:10:08.000Z","updated_at":"2026-02-08T00:39:10.000Z","dependencies_parsed_at":"2023-09-29T22:30:16.299Z","dependency_job_id":"14f2a4a9-0586-4db7-b96d-e8d9c26c158f","html_url":"https://github.com/roshan-research/hazm","commit_stats":{"total_commits":1268,"total_committers":42,"mean_commits":30.19047619047619,"dds":0.6175078864353312,"last_synced_commit":"270e8977a092e94605821f4742996d7b3e173dc0"},"previous_names":["sobhe/hazm"],"tags_count":17,"template":false,"template_full_name":null,"purl":"pkg:github/roshan-research/hazm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roshan-research%2Fhazm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roshan-research%2Fhazm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roshan-research%2Fhazm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roshan-research%2Fhazm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/roshan-research","download_url":"https://codeload.github.com/roshan-research/hazm/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roshan-research%2Fhazm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29637412,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-19T22:32:43.237Z","status":"ssl_error","status_checked_at":"2026-02-19T22:32:38.330Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dependency-parser","embeddings","farsi","lemmatization","natural-language-processing","nlp","normalization","persian","persian-nlp","pos-tagging","python","text-processing","tokenizer"],"created_at":"2024-08-03T18:01:26.338Z","updated_at":"2026-02-20T00:32:40.770Z","avatar_url":"https://github.com/roshan-research.png","language":"Python","funding_links":[],"categories":["Python","NLP per Language"],"sub_categories":["Libraries"],"readme":"# Hazm - Persian NLP Toolkit\n\n![Tests](https://img.shields.io/github/actions/workflow/status/roshan-research/hazm/test.yml?branch=master)\n![PyPI - Downloads](https://img.shields.io/github/downloads/roshan-research/hazm/total)\n![PyPI - Python Version](https://img.shields.io/pypi/pyversions/hazm)\n![GitHub](https://img.shields.io/github/license/roshan-research/hazm)\n\n[**Hazm**](https://www.roshan-ai.ir/hazm/) is a python library to perform natural language processing tasks on Persian text. It offers various features for analyzing, processing, and understanding Persian text. You can use Hazm to normalize text, tokenize sentences and words, lemmatize words, assign part-of-speech tags, identify dependency relations, create word and sentence embeddings, or read popular Persian corpora.\n\n[![sample](documentation/assets/sample.png)](documentation/assets/sample.png)\n\n## Features\n\n- **Normalization:** Converts text to a standard form (diacritics removal, ZWNJ correction, etc).\n- **Tokenization:** Splits text into sentences and words.\n- **Lemmatization:** Reduces words to their base forms.\n- **POS tagging:** Assigns a part of speech to each word.\n- **Dependency parsing:** Identifies the syntactic relations between words.\n- **Embedding:** Creates vector representations of words and sentences.\n- **Hugging Face Integration:** Automatically download and cache pretrained models from the Hub.\n- **Persian corpora reading:** Easily read popular Persian corpora with ready-made scripts.\n\n## Installation\n\nTo install the latest version of Hazm (requires Python 3.12+), run:\n\n    pip install hazm\n\nTo use the pretrained models from Hugging Face, ensure you have the `huggingface-hub` package:\n\n    pip install huggingface-hub\n\n## Pretrained-Models\n\nHazm supports automatic downloading of pretrained models. You can find all available models (POS Tagger, Chunker, Embeddings, etc.) on our official Hugging Face page:\n\n👉 [**Roshan Research on Hugging Face**](https://huggingface.co/roshan-research/models)\n\nWhen using Hazm, simply provide the `repo_id` and `model_filename` as shown in the examples below, and the library will handle the rest.\n\n## Usage\n\n```python\nfrom hazm import *\n\n# ===============================\n# Stemming\n# ===============================\nstemmer = Stemmer()\nstem = stemmer.stem('کتاب‌ها')\nprint(stem) # کتاب\n\n# ===============================\n# Normalizing\n# ===============================\nnormalizer = Normalizer()\nnormalized_text = normalizer.normalize('من کتاب های زیــــادی دارم .')\nprint(normalized_text) # من کتاب‌های زیادی دارم.\n\n# ===============================\n# Lemmatizing\n# ===============================\nlemmatizer = Lemmatizer()\nlem = lemmatizer.lemmatize('می‌نویسیم')\nprint(lem) # نوشت#نویس\n\n# ===============================\n# Sentence tokenizing\n# ===============================\nsentence_tokenizer = SentenceTokenizer()\nsent_tokens = sentence_tokenizer.tokenize('ما کتاب می‌خوانیم. یادگیری خوب است.')\nprint(sent_tokens) # ['ما کتاب می\\u200cخوانیم.', 'یادگیری خوب است.']\n\n# ===============================\n# Word tokenizing\n# ===============================\nword_tokenizer = WordTokenizer()\nword_tokens = word_tokenizer.tokenize('ما کتاب می‌خوانیم')\nprint(word_tokens) # ['ما', 'کتاب', 'می\\u200cخوانیم']\n\n# ===============================\n# Part of speech tagging\n# ===============================\ntagger = POSTagger(repo_id=\"roshan-research/hazm-postagger\", model_filename=\"pos_tagger.model\")\ntagged_words = tagger.tag(word_tokens)\nprint(tagged_words) # [('ما', 'PRON'), ('کتاب', 'NOUN'), ('می\\u200cخوانیم', 'VERB')]\n\n# ===============================\n# Chunking\n# ===============================\nchunker = Chunker(repo_id=\"roshan-research/hazm-chunker\", model_filename=\"chunker.model\")\nchunked_tree = tree2brackets(chunker.parse(tagged_words))\nprint(chunked_tree) # [ما NP] [کتاب NP] [می‌خوانیم VP]\n\n# ===============================\n# Word embedding\n# ===============================\nword_embedding = WordEmbedding.load(repo_id='roshan-research/hazm-word-embedding', model_filename='fasttext_skipgram_300.bin', model_type='fasttext')\nodd_word = word_embedding.doesnt_match(['کتاب', 'دفتر', 'قلم', 'پنجره'])\nprint(odd_word) # پنجره\n\n# ===============================\n# Sentence embedding\n# ===============================\nsent_embedding = SentEmbedding.load(repo_id='roshan-research/hazm-sent-embedding', model_filename='sent2vec-naab.model')\nsentence_similarity = sent_embedding.similarity('او شیر میخورد','شیر غذا می‌خورد')\nprint(sentence_similarity) # 0.4643607437610626\n\n# ===============================\n# Dependency parsing\n# ===============================\nparser = DependencyParser(tagger=tagger, lemmatizer=lemmatizer, repo_id=\"roshan-research/hazm-dependency-parser\", model_filename=\"langModel.mco\")\ndependency_graph = parser.parse(word_tokens)\nprint(dependency_graph)\n\"\"\"\n{0:  {'address': 0,\n      'ctag': 'TOP',\n      'deps': defaultdict(\u003cclass 'list'\u003e, {'root': [3]}),\n      'feats': None,\n      'head': None,\n      'lemma': None,\n      'rel': None,\n      'tag': 'TOP',\n      'word': None},\n  1: {'address': 1,\n      'ctag': 'PRON',\n      'deps': defaultdict(\u003cclass 'list'\u003e, {}),\n      'feats': '_',\n      'head': 3,\n      'lemma': 'ما',\n      'rel': 'SBJ',\n      'tag': 'PRON',\n      'word': 'ما'},\n  2: {'address': 2,\n      'ctag': 'NOUN',\n      'deps': defaultdict(\u003cclass 'list'\u003e, {}),\n      'feats': '_',\n      'head': 3,\n      'lemma': 'کتاب',\n      'rel': 'OBJ',\n      'tag': 'NOUN',\n      'word': 'کتاب'},\n  3: {'address': 3,\n      'ctag': 'VERB',\n      'deps': defaultdict(\u003cclass 'list'\u003e, {'SBJ': [1], 'OBJ': [2]}),\n      'feats': '_',\n      'head': 0,\n      'lemma': 'خواند#خوان',\n      'rel': 'root',\n      'tag': 'VERB',\n      'word': 'می\\u200cخوانیم'}})\n\n\"\"\"\n```\n\n## Documentation\n\nVisit https://roshan-ai.ir/hazm to view the full documentation.\n\n## Evaluation\n\n| Module name      |           |\n| :--------------- | --------- |\n| DependencyParser | **85.6%** |\n| POSTagger        | **98.8%** |\n| Chunker          | **93.4%** |\n| Lemmatizer       | **89.9%** |\n\n|                                | Metric          | Value   |\n| ------------------------------ | --------------- | ------- |\n| **SpacyPOSTagger**             | Precision       | 0.99250 |\n|                                | Recall          | 0.99249 |\n|                                | F1-Score        | 0.99249 |\n| **EZ Detection in SpacyPOSTagger** | Precision   | 0.99301 |\n|                                | Recall          | 0.99297 |\n|                                | F1-Score        | 0.99298 |\n| **SpacyChunker**                | Accuracy        | 96.53%  |\n|                                | F-Measure       | 95.00%  |\n|                                | Recall          | 95.17%  |\n|                                | Precision       | 94.83%  |\n| **SpacyDependencyParser**       | TOK Accuracy    | 99.06   |\n|                                | UAS             | 92.30   |\n|                                | LAS             | 89.15   |\n|                                | SENT Precision  | 98.84   |\n|                                | SENT Recall     | 99.38   |\n|                                | SENT F-Measure  | 99.11   |\n\n[![Star History Chart](https://api.star-history.com/svg?repos=roshan-research/hazm\u0026type=Date)](https://star-history.com/#roshan-research/hazm\u0026Date)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froshan-research%2Fhazm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Froshan-research%2Fhazm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froshan-research%2Fhazm/lists"}