{"id":21996646,"url":"https://github.com/silvermete0r/qaznltk","last_synced_at":"2025-04-30T20:48:57.462Z","repository":{"id":207328379,"uuid":"718978142","full_name":"silvermete0r/QazNLTK","owner":"silvermete0r","description":"A package for working with Kazakh language text processing.","archived":false,"fork":false,"pushed_at":"2025-02-15T10:46:40.000Z","size":60,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-30T19:51:12.407Z","etag":null,"topics":["kazakh-language","kazakh-latin-alphabet","kazakh-translation","natural-language-processing","nltk-python"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/qaznltk/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/silvermete0r.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-15T07:26:36.000Z","updated_at":"2025-02-15T10:46:44.000Z","dependencies_parsed_at":"2024-10-18T16:17:50.328Z","dependency_job_id":"2af10715-4405-42a9-bd13-b1aae389dec6","html_url":"https://github.com/silvermete0r/QazNLTK","commit_stats":null,"previous_names":["silvermete0r/qaznltk"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/silvermete0r%2FQazNLTK","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/silvermete0r%2FQazNLTK/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/silvermete0r%2FQazNLTK/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/silvermete0r%2FQazNLTK/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/silvermete0r","download_url":"https://codeload.github.com/silvermete0r/QazNLTK/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251781772,"owners_count":21642972,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kazakh-language","kazakh-latin-alphabet","kazakh-translation","natural-language-processing","nltk-python"],"created_at":"2024-11-29T22:12:11.430Z","updated_at":"2025-04-30T20:48:57.455Z","avatar_url":"https://github.com/silvermete0r.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://sun9-11.userapi.com/impg/L_xiLxzenYgyzCVPJtXRCb-A1PTpHvIOSLmzcQ/XWD4xSHW5Vw.jpg?size=1280x440\u0026quality=95\u0026sign=efe38d48a22e0ca8ca8729a89ea0d401\u0026type=album\"\u003e\u003cbr\u003e\n\u003c/div\u003e\n\n-----------------\n\n# QazNLTK: a package for working with Kazakh language text processing.\n\n[![PyPI Latest Release](https://img.shields.io/pypi/v/qaznltk.svg)](https://pypi.org/project/qaznltk/) [![PyPI Downloads](https://img.shields.io/pypi/dm/qaznltk.svg?label=PyPI%20downloads)](https://pypi.org/project/qaznltk/) [![PyPI Downloads](https://static.pepy.tech/badge/qaznltk)](https://pepy.tech/projects/qaznltk)\n\n\n## What is it?\n\n**QazNLTK** provides developers with a fast and convenient tool for processing text in the Kazakh language. Tailored for the unique linguistic characteristics of Kazakh, this library offers a comprehensive set of tools for natural language processing, like: tokenization, sentence segmentation, evaluation similarity score and tranliteration of kazakh language cyrillic-latin.\n\n## Table of Contents\n\n- [Main Features](#main-features)\n- [Where to get it](#where-to-get-it)\n- [Dependencies](#dependencies)\n- [License](#license)\n- [Getting Help](#getting-help)\n- [Contributing to QazNLTK](#contributing-to-qaznltk)\n\n## Main Features\nHere are just a few of the things that qaznltk does well:\n\n1) Kazakh language Text Tokenizing by keyword frequencies:\n``` Python\nfrom qaznltk import qaznltk as qnltk\nqn = qnltk.QazNLTK()\n\ntext = input(\"Enter text: \")\ntokens = qn.tokenize(text)\nprint(tokens)\n\n# Input: Біздің өміріміз үлкен өзен іспетті. Сіздің қайығыңыздың қиындықтардан жеңіл өтіп, махаббат иірімінде басқаруын жоғалтпай, бақыт сарқырамасына жетуін тілеймін!\n# Output: [('өміріміз', 1), ('үлкен', 1), ('өзен', 1), ('іспетті', 1), ('сіздің', 1), ('қайығыңыздың', 1), ('қиындықтардан', 1), ('жеңіл', 1), ('өтіп', 1), ('махаббат', 1), ('иірімінде', 1), ('басқаруын', 1), ('жоғалтпай', 1), ('бақыт', 1), ('сарқырамасына', 1), ('жетуін', 1), ('тілеймін', 1)]\n```\n\n2) Kazakh language Text Segmentation into sentences:\n``` Python\nfrom qaznltk import qaznltk as qnltk\nqn = qnltk.QazNLTK()\n\ntext = input(\"Enter text: \")\nsent_tokens = qn.sent_tokenize(text)\nprint(sent_tokens)\n\n# Input: Біздің өміріміз үлкен өзен іспетті. Сіздің қайығыңыздың қиындықтардан жеңіл өтіп, махаббат иірімінде басқаруын жоғалтпай, бақыт сарқырамасына жетуін тілеймін!\n# Output: ['Біздің өміріміз үлкен өзен іспетті.', 'Сіздің қайығыңыздың қиындықтардан жеңіл өтіп, махаббат иірімінде басқаруын жоғалтпай, бақыт сарқырамасына жетуін тілеймін!']\n```\n\n3) Evaluate Difference score between 2 text:\n``` Python\nfrom qaznltk import qaznltk as qnltk\nqn = qnltk.QazNLTK()\n\ntextA = input(\"Enter text A: \")\ntextB = input(\"Enter text B: \")\nsimilarity_score = qn.calc_similarity(textA, textB)\nprint(similarity_score)\n\n# Input: textA = \"Еңбегіне қарай — құрмет, Жасына қарай — ізет.\", textB = \"Еңбегіне қарай табысы, Ерлігіне қарай дабысы.\"\n# Output: 0.368421052631579\n```\n\n4) Convert Kazakh language Text from Cyrillic to Latin using ISO-9 Standard:\n``` Python\nfrom qaznltk import qaznltk as qnltk\nqn = qnltk.QazNLTK()\n\ntext = input(\"Enter text: \")\nlatin_text = qn.convert2latin_iso9(text)\nprint(latin_text)\n\n# Input: Бүгін қандай керемет күн! \n# Output: Bùgìn k̦andaj keremet kùn!\n```\n\n5) Convert Kazakh language Text from Latin to Cyrillic using ISO-9 Standard:\n``` Python\nfrom qaznltk import qaznltk as qnltk\nqn = qnltk.QazNLTK()\n\ntext = input(\"Enter text: \")\ncyrillic_text = qn.convert2cyrillic_iso9(text)\nprint(cyrillic_text)\n\n# Input: Bùgìn k̦andaj keremet kùn!\n# Output: Бүгін қандай керемет күн!\n```\n\n6) Sentiment Analysis of Kazakh language text [`negative: -1`, `neutral: 0`, `positive: 1`]:\n``` Python\nfrom qaznltk import qaznltk as qnltk\nqn = qnltk.QazNLTK()\n\ntext = input(\"Enter text: \")\nsentimize_score = qn.sentimize(text)\nprint(sentimize_score)\n\n# Input: Бұл мақала өте нашар жазылған.\n# Output: -1.0 (negative)\n```\n\n7) Converting any number `N` into kazakh language number words [`N \u003c= 10^31`]:\n``` Python\nfrom qaznltk import qaznltk as qnltk\nqn = qnltk.QazNLTK()\n\nn = int(input())\nprint(qn.num2word(n))\n\n# Input: N = 1465\n# Output: бір мың төрт жүз алпыс бес\n```\n\n8) Extracting information from IIN (Individual Identification Number) [`IIN: 12 digits`]:\n``` Python\nfrom qaznltk import qaznltk as qnltk\nqn = qnltk.QazNLTK()\n\niin = input(\"Enter IIN: \")\nprint(qn.get_info_from_iin(iin))\n\n# Input: 990408482390\n# Output: {'status': 'success', 'date_of_birth': '08.04.1999', 'century_of_birth': '20', 'gender': 'female', 'sequence_number': 8239, 'control_discharge': 0}\n```\n\n9) KNN Search on TF-IDF matrix embeddings of Kazakh language text:\n``` Python\nfrom qaznltk import vectorizer\n\nqn_vectorizer = vectorizer.QazNLTKVectorizer()\ntf_idf_matrix = qn_vectorizer.fit_transform(documents)\n\nknn = vectorizer.KNN(tf_idf_matrix)\n\nquery = \"Еліміздің алтын күні жарық күн.\"\n\nquery_vector = qn_vectorizer.transform([query])[0]\n\nresults = knn.search(query_vector, k=3)\n\nfor idx, distance in results:\n    print(f\"Document: {documents[idx]}, Distance: {distance}\")\n\n# Input:\n# documents = [\n#     \"Ер — елінде, гүл — жерінде.\",\n#     \"Өз елінде көртышқан да батыр.\",\n#     \"Өз елінің иті де қадірлі.\",\n#     \"Отан үшін күрес — ерге тиген үлес.\",\n#     \"Орағың өткір болса, қарың талмайды, Отаның берік болса, жауың алмайды.\",\n#     \"Елінен безген ер болмас, Көлінен безген қаз болмас.\",\n#     \"Сағынған елін аңсайды, Сары ала қаз көлін аңсайды.\",\n#     \"Жат жердің қаршығасынан, Өз еліңнің қарғасы артық.\",\n#     \"Егілмеген жер жетім, Елінен айырылған ер жетім.\",\n#     \"Ерінен айырылған көмгенше жылайды, Елінен айырылған өлгенше жылайды.\",\n#     \"Отан — отбасынан басталады.\",\n#     \"Опасызда oтан жоқ.\",\n#     \"Отан оттан да ыстық.\",\n#     \"Отансыз адам — ормансыз бұлбұл.\"\n# ]\n\n# Output:\n# Document: Орағың өткір болса, қарың талмайды, Отаның берік болса, жауың алмайды., Distance: 0.6740830490255459\n# Document: Жат жердің қаршығасынан, Өз еліңнің қарғасы артық., Distance: 0.7040525969511919\n# Document: Өз елінде көртышқан да батыр., Distance: 0.7453452762306501\n```\n\n10) QazPerry\n\nGemma 2: 2B QazPerry is a fine-tuned version of the Gemma 2B model, optimized for the Kazakh language. This model is part of the QazPerry initiative, which aims to develop Small Large Language Models (SLLMs) to enhance Kazakh NLP capabilities. It has been fine-tuned using the saillab/alpaca_kazakh_taco dataset with a LoRA adaptation approach to improve efficiency.\n\n```bash\npip install keras-nlp huggingface_hub\n```\n\n```python\nfrom huggingface_hub import hf_hub_download\nimport keras\nimport keras_nlp\n\nrepo_id = \"silvermete0r/Gemma2_2B_QazPerry\"\nfilename = \"Gemma2_2B_QazPerry.keras\"\n\nmodel_path = hf_hub_download(repo_id=repo_id, filename=filename)\n\ngemma_lm = keras.models.load_model(model_path, custom_objects={\"GemmaCausalLM\": keras_nlp.models.GemmaCausalLM})\n\nprompt = \"Instruction:\\nҚазақша бірдеңе айтшы?\\n\\nResponse:\\n\"\n\nprint(gemma_lm.generate(prompt))\n```\n\n* Kaggle Model: https://www.kaggle.com/models/armanzhalgasbayev/gemma2_2b_qazperry/\n* HuggingFace Model: https://huggingface.co/silvermete0r/Gemma2_2B_QazPerry\n* First Inference QazPerry:  https://www.kaggle.com/code/armanzhalgasbayev/gemma2-2b-qazperry-first-inference\n* QazPerry Fine-Tuning Notebook: https://www.kaggle.com/code/armanzhalgasbayev/kazakh-llm-based-on-gemma-2b-kz\n* QazPerry Demo Inference: https://huggingface.co/spaces/silvermete0r/Gemma_2_2B_QazPerry_Inference_Demo\n* Training Dataset: https://huggingface.co/datasets/saillab/alpaca_kazakh_taco\n\n\n* **Test Samples:** https://vk.com/club121755042\n\n## Where to get it\nThe source code is currently hosted on GitHub at: https://github.com/silvermete0r/QazNLTK.git\n\nBinary installers for the latest released version are available at the [Python\nPackage Index (PyPI)](https://pypi.org/project/qaznltk).\n\n```sh\npip install qaznltk\n```\n\n![image](https://github.com/user-attachments/assets/057554b7-629b-4f11-bf0f-05b6116b3716)\n\n\n## Dependencies\n- Package was developed on built-in python functions (pure python); \n\n\n## License\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n\n## Getting Help\n\n📧 [supwithproject@gmail.com](https://gmail.com/)\n\n## Contributing to qaznltk\n\nAll contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.\n\n\u003chr\u003e\n\n[Go to Top](#table-of-contents)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsilvermete0r%2Fqaznltk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsilvermete0r%2Fqaznltk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsilvermete0r%2Fqaznltk/lists"}