{"id":15454751,"url":"https://github.com/bikatr7/kairyou","last_synced_at":"2025-04-21T09:20:27.369Z","repository":{"id":221142292,"uuid":"753577156","full_name":"Bikatr7/Kairyou","owner":"Bikatr7","description":"Quickly preprocesses Japanese text using NLP/NER from SpaCy for Japanese translation or other NLP tasks. ","archived":false,"fork":false,"pushed_at":"2025-01-19T04:29:39.000Z","size":380,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-20T00:54:23.612Z","etag":null,"topics":["japanese","ner","nlp","preprocess","spacy"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-2.1","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Bikatr7.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-06T11:57:06.000Z","updated_at":"2025-04-03T13:40:31.000Z","dependencies_parsed_at":"2024-03-08T10:40:39.773Z","dependency_job_id":"54a038e3-1c16-49d2-a639-dbee27b6b08e","html_url":"https://github.com/Bikatr7/Kairyou","commit_stats":{"total_commits":114,"total_committers":2,"mean_commits":57.0,"dds":0.07017543859649122,"last_synced_commit":"923590b4f9d8396d39448c5d7e34c064a3646f6b"},"previous_names":["bikatr7/kairyou"],"tags_count":35,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bikatr7%2FKairyou","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bikatr7%2FKairyou/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bikatr7%2FKairyou/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bikatr7%2FKairyou/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Bikatr7","download_url":"https://codeload.github.com/Bikatr7/Kairyou/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249834786,"owners_count":21331988,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["japanese","ner","nlp","preprocess","spacy"],"created_at":"2024-10-01T22:05:19.414Z","updated_at":"2025-04-20T00:54:32.762Z","avatar_url":"https://github.com/Bikatr7.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"---------------------------------------------------------------------------------------------------------------------------------------------------\n**Table of Contents**\n\n- [Quick Start](#quick-start)\n- [Installation](#installation)\n- [Usage](#usage)\n  - [Kairyou](#kairyou)\n  - [KatakanaUtil](#katakanautil)\n  - [Indexer](#indexer)\n- [License](#license)\n- [Contact](#contact)\n- [Contribution](#contribution)\n- [Notes](#notes)\n- [Inspirations](#inspirations)\n\n---------------------------------------------------------------------------------------------------------------------------------------------------\n\n## Kairyou\n\nQuickly preprocesses Japanese text using NLP/NER from SpaCy for Japanese translation or other NLP tasks. \n\n---------------------------------------------------------------------------------------------------------------------------------------------------\n**Quick Start**\u003ca name=\"quick-start\"\u003e\u003c/a\u003e\n\nTo get started with Kairyou, install the package via pip:\n\n```bash\npip install kairyou\n```\n\nYou must also install the jp_core_news_lg model from spaCy. This can be done by running the following command:\n\nfor windows:\n\n```bash\npython -m spacy download ja_core_news_lg\n```\n\nfor linux:\n\n```bash\npython3 -m spacy download ja_core_news_lg\n```\n\nThen, you can preprocess Japanese text by importing Kairyou and/or KatakanaUtil/Indexer:\n```python\nfrom kairyou import Kairyou, KatakanaUtil, Indexer\n```\n\nFollow the usage examples provided in the [Usage](#usage) section for detailed instructions on preprocessing text and handling katakana.\n\n---------------------------------------------------------------------------------------------------------------------------------------------------\n\n**Installation**\u003ca name=\"installation\"\u003e\u003c/a\u003e\n\nPython 3.10+\n\nKairyou can be installed using pip:\n\n\n```bash\npip install kairyou\n```\n\nThis will install Kairyou along with its dependencies, including spaCy and a few other packages.\n\nThese are the dependencies/requirements that will be installed:\n```\nsetuptools\u003e=61.0\n\nwheel\n\nsetuptools_scm\u003e=6.0\n\ntomli\n\nspacy==3.7.5\n\nregex\u003e=2023.3.23\n\n```\n---------------------------------------------------------------------------------------------------------------------------------------------------\n\n**Usage**\u003ca name=\"usage\"\u003e\u003c/a\u003e\n\n---------------------------------------------------------------------------------------------------------------------------------------------------\n\n**Kairyou**\u003ca name=\"kairyou\"\u003e\u003c/a\u003e\n\nKairyou is the global preprocessor client. Here's an example of how to use it:\n\n```python\nfrom kairyou import Kairyou\n\ntext = \"Your Japanese text here.\"\nreplacement_json = \"path/to/your/replacement_rules.json\"  ## or a dict of rules\npreprocessed_text, preprocessing_log, error_log = Kairyou.preprocess(text, replacement_json)\n\nprint(preprocessed_text)\n```\n\nCurrently, Kairyou supports two json types, \"Kudasai\" and \"Fukuin\". \"Kudasai\" is the native type and originated from that program, Fukuin is what the original onegai program used, as well as what the kroatoan's Fukuin program uses. No major differences in replacement are present between the two.\n\nKairyou performs some post-processing on the text to correct any issues that may have arisen during the preprocessing.\n\n[Blank Kudasai Json](examples/blank_kudasai.json)\n\n[Example Kudasai Json](examples/cote_kudasai.json)\n\n[Blank Fukuin Json](examples/blank_fukuin.json)\n\n[Example Fukuin Json](examples/cote_fukuin.json)\n\n---------------------------------------------------------------------------------------------------------------------------------------------------\n\n**KatakanaUtil**\u003ca name=\"katakanautil\"\u003e\u003c/a\u003e\n\nKatakanaUtil provides utility functions for handling katakana characters in Japanese text. Example usage:\n\n```python\nfrom kairyou import KatakanaUtil\n\nkatakana_word = \"カタカナ\"\nif KatakanaUtil.is_katakana_only(katakana_word):\n    print(f\"{katakana_word} is composed only of Katakana characters.\")\n```\n\nThe following functions are available in KatakanaUtil:\n\nis_katakana_only: Returns True if the input string is composed only of katakana characters.\n\nis_actual_word: Returns True if the input string is a actual Japanese Katakana word (not just something made up or a name). List of words can be found [here](src/kairyou/words.py).\n\nis_punctuation: Returns True if the input string is punctuation (Both Japanese and English punctuation are supported). List of punctuation can be found [here](src/kairyou/katakana_util.py).\n\nis_repeating_sequence: Returns True if the input string is just a repeating sequence of characters. (e.g. \"ジロジロ\")\n\nis_more_punctuation_than_japanese: Returns True if the input string has more punctuation than Japanese characters.\n\nis_partially_english: Returns True if the input string has any English characters in it.\n\n---------------------------------------------------------------------------------------------------------------------------------------------------\n\n**Indexer**\u003ca name=\"indexer\"\u003e\u003c/a\u003e\n\nIndexer is for \"indexing\" Japanese text. What this means is that, given input_text, a knowledge_base, and a replacements_json. It will return a list of new \"names\", and the occurrence which was flagged.\n\nWhat is considered a name is a bit complicated. But:\n1. Must have the \"person\" label when using spaCy's NER.\n2. Cannot have more punctuation than Japanese characters.\n3. Cannot be a repeating sequence of characters.\n4. Cannot be an actual Japanese Katakana word.\n5. Cannot have any english characters in it. (Names that are pre-replaced would have already been in the other texts, plus the JP NER model seems to have trouble with ENG in general)\n\nSo, it'll return names that don't in the other texts.\n\nThis can be done via index()\n```py\nfrom kairyou import Indexer\n\ninput_text = \"Your Japanese text here.\" ## or a path to a text file\nknowledge_base = [\"more Japanese text here.\", \"even_more_japanese_text_here\"] ## or a path to a text file or directory full of text files\nreplacements_json = \"path/to/your/replacement_rules.json\"  ## or a dict of rules\n\n## You can optionally send a list of strings to ignore, but this is not required.\n\nNamesAndOccurrences, indexing_log = Indexer.index(input_text, knowledge_base, replacements_json) ## also takes in a list of strings to ignore, defaults to []\n```\n\nNamesAndOccurrences is a list of named tuples, with the following fields:\n1. name: The name that was found.\n2. occurrence: The occurrence of the name in the input_text.\n\nIndexing_log is a string of the log of the indexing process. What was indexed, it's occurrence, what was ignored, and time elapsed.\n\nIndex works with both Fukuin and Kudasai jsons.\n\n---------------------------------------------------------------------------------------------------------------------------------------------------\n\n**License**\u003ca name=\"license\"\u003e\u003c/a\u003e\n\nThis project, Kairyou, is licensed under the GNU Lesser General Public License v2.1 (LGPLv2.1) - see the LICENSE file for complete details.\n\nThe LGPL is a permissive copyleft license that enables this software to be freely used, modified, and distributed. It is particularly designed for libraries, allowing them to be included in both open source and proprietary software. When using or modifying Kairyou, you can choose to release your work under the LGPLv2.1 to contribute back to the community or incorporate it into proprietary software as per the license's permissions.\n\nUnder the LGPLv2.1, any modifications made to Kairyou's libraries must be shared under the same license, ensuring the open source nature of the library itself is maintained. However, it allows the wider work that includes the library to remain under different licensing terms, provided the terms of the LGPLv2.1 are met for the library.\n\nI encourage the use of Kairyou within all projects. This approach ensures a balance between open collaboration and the flexibility required for proprietary development.\n\nFor a thorough explanation of the conditions and how they may apply to your use of the project, please review the full license text. This comprehensive documentation will offer guidance on your rights and obligations when utilizing LGPLv2.1 licensed software.\n\n---------------------------------------------------------------------------------------------------------------------------------------------------\n\n**Contact**\u003ca name=\"contact\"\u003e\u003c/a\u003e\n\nIf you have any questions or suggestions, feel free to reach out to me at [Bikatr7@proton.me](mailto:Bikatr7@proton.me)\n\nAlso feel free to check out the [GitHub repository](https://github.com/Bikatr7/Kairyou) for this project.\n\nOr the issue tracker [here](https://github.com/Bikatr7/Kairyou/issues).\n\n---------------------------------------------------------------------------------------------------------------------------------------------------\n\n**Contribution**\u003ca name=\"contribution\"\u003e\u003c/a\u003e\n\nContributions are welcome! I don't have a specific format for contributions, but please feel free to submit a pull request or open an issue if you have any suggestions or improvements.\n\n---------------------------------------------------------------------------------------------------------------------------------------------------\n\n**Notes**\u003ca name=\"notes\"\u003e\u003c/a\u003e\n\nKairyou was originally developed as a part of [Kudasai](https://github.com/Bikatr7/Kudasai), a Japanese preprocessor later turned Machine Translator. It was later split off into its own package to be used independently of Kudasai for multiple reasons.\n\nKairyou gets its name from the Japanese word \"Reform\" (改良) which is pronounced \"Kairyou\". Which was chosen for two reasons, the first being that it was chosen during a large Kudasai rework, and the second being that it is a Japanese preprocessor, and the name seemed fitting.\n\nThis package is also my first serious attempt at creating a Python package, so I'm sure there are some things that could be improved. Feedback is welcomed.\n\n---------------------------------------------------------------------------------------------------------------------------------------------------\n\n**Inspirations**\u003ca name=\"inspirations\"\u003e\u003c/a\u003e\n\nKudasai and by extension Kairyou was originally derived from [Void's Script](https://github.com/Atreyagaurav/mtl-related-scripts) later [Onegai](https://github.com/Atreyagaurav/onegai)\n\nKairyou also took some inspiration from [Fukuin](https://github.com/kroatoanjp/nlp-mtl-preprocessing-script) and it's approach with Katakana.\n\nThanks to all of the above for the inspiration and the work they put into their projects.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbikatr7%2Fkairyou","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbikatr7%2Fkairyou","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbikatr7%2Fkairyou/lists"}