{"id":20630391,"url":"https://github.com/koichiyasuoka/unidic2ud","last_synced_at":"2025-06-25T09:05:13.177Z","repository":{"id":62586373,"uuid":"204508086","full_name":"KoichiYasuoka/UniDic2UD","owner":"KoichiYasuoka","description":"Tokenizer POS-tagger Lemmatizer and Dependency-parser for modern and contemporary Japanese","archived":false,"fork":false,"pushed_at":"2024-11-20T10:26:58.000Z","size":167645,"stargazers_count":37,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-23T07:03:05.225Z","etag":null,"topics":["dependency-parser","japanese-language","nlp"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KoichiYasuoka.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-26T15:42:04.000Z","updated_at":"2025-04-08T01:24:46.000Z","dependencies_parsed_at":"2024-12-26T00:06:23.241Z","dependency_job_id":"87c67016-1b80-499a-bbbd-aa855feab621","html_url":"https://github.com/KoichiYasuoka/UniDic2UD","commit_stats":{"total_commits":386,"total_committers":1,"mean_commits":386.0,"dds":0.0,"last_synced_commit":"8482d0695327f261b764bfa4bc2375a1fa6f1138"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/KoichiYasuoka/UniDic2UD","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoichiYasuoka%2FUniDic2UD","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoichiYasuoka%2FUniDic2UD/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoichiYasuoka%2FUniDic2UD/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoichiYasuoka%2FUniDic2UD/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KoichiYasuoka","download_url":"https://codeload.github.com/KoichiYasuoka/UniDic2UD/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KoichiYasuoka%2FUniDic2UD/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261841946,"owners_count":23217912,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dependency-parser","japanese-language","nlp"],"created_at":"2024-11-16T14:07:44.760Z","updated_at":"2025-06-25T09:05:13.144Z","avatar_url":"https://github.com/KoichiYasuoka.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Current PyPI packages](https://badge.fury.io/py/unidic2ud.svg)](https://pypi.org/project/unidic2ud/)\n\n# UniDic2UD\n\nTokenizer, POS-tagger, lemmatizer, and dependency-parser for modern and contemporary Japanese, working on [Universal Dependencies](https://universaldependencies.org/format.html).\n\n## Basic usage\n\n```py\n\u003e\u003e\u003e import unidic2ud\n\u003e\u003e\u003e nlp=unidic2ud.load(\"kindai\")\n\u003e\u003e\u003e s=nlp(\"其國を治めんと欲する者は先づ其家を齊ふ\")\n\u003e\u003e\u003e print(s)\n# text = 其國を治めんと欲する者は先づ其家を齊ふ\n1\t其\t其の\tDET\t連体詞\t_\t2\tdet\t_\tSpaceAfter=No|Translit=ソノ\n2\t國\t国\tNOUN\t名詞-普通名詞-一般\t_\t4\tobj\t_\tSpaceAfter=No|Translit=クニ\n3\tを\tを\tADP\t助詞-格助詞\t_\t2\tcase\t_\tSpaceAfter=No|Translit=ヲ\n4\t治め\t収める\tVERB\t動詞-一般\t_\t7\tadvcl\t_\tSpaceAfter=No|Translit=オサメ\n5\tん\tむ\tAUX\t助動詞\t_\t4\taux\t_\tSpaceAfter=No|Translit=ン\n6\tと\tと\tADP\t助詞-格助詞\t_\t4\tcase\t_\tSpaceAfter=No|Translit=ト\n7\t欲する\t欲する\tVERB\t動詞-一般\t_\t8\tacl\t_\tSpaceAfter=No|Translit=ホッスル\n8\t者\t者\tNOUN\t名詞-普通名詞-一般\t_\t14\tnsubj\t_\tSpaceAfter=No|Translit=モノ\n9\tは\tは\tADP\t助詞-係助詞\t_\t8\tcase\t_\tSpaceAfter=No|Translit=ハ\n10\t先づ\t先ず\tADV\t副詞\t_\t14\tadvmod\t_\tSpaceAfter=No|Translit=マヅ\n11\t其\t其の\tDET\t連体詞\t_\t12\tdet\t_\tSpaceAfter=No|Translit=ソノ\n12\t家\t家\tNOUN\t名詞-普通名詞-一般\t_\t14\tobj\t_\tSpaceAfter=No|Translit=ウチ\n13\tを\tを\tADP\t助詞-格助詞\t_\t12\tcase\t_\tSpaceAfter=No|Translit=ヲ\n14\t齊ふ\t整える\tVERB\t動詞-一般\t_\t0\troot\t_\tSpaceAfter=No|Translit=トトノフ\n\n\u003e\u003e\u003e t=s[7]\n\u003e\u003e\u003e print(t.id,t.form,t.lemma,t.upos,t.xpos,t.feats,t.head.id,t.deprel,t.deps,t.misc)\n7 欲する 欲する VERB 動詞-一般 _ 8 acl _ SpaceAfter=No|Translit=ホッスル\n\n\u003e\u003e\u003e print(s.to_tree())\n    其 \u003c══╗         det(決定詞)\n    國 ═╗═╝\u003c╗       obj(目的語)\n    を \u003c╝   ║       case(格表示)\n  治め ═╗═╗═╝\u003c╗     advcl(連用修飾節)\n    ん \u003c╝ ║   ║     aux(動詞補助成分)\n    と \u003c══╝   ║     case(格表示)\n欲する ═══════╝\u003c╗   acl(連体修飾節)\n    者 ═╗═══════╝\u003c╗ nsubj(主語)\n    は \u003c╝         ║ case(格表示)\n  先づ \u003c══════╗   ║ advmod(連用修飾語)\n    其 \u003c══╗   ║   ║ det(決定詞)\n    家 ═╗═╝\u003c╗ ║   ║ obj(目的語)\n    を \u003c╝   ║ ║   ║ case(格表示)\n  齊ふ ═════╝═╝═══╝ root(親)\n\n\u003e\u003e\u003e f=open(\"trial.svg\",\"w\")\n\u003e\u003e\u003e f.write(s.to_svg())\n\u003e\u003e\u003e f.close()\n```\n![trial.svg](https://raw.githubusercontent.com/KoichiYasuoka/UniDic2UD/master/trial.png)\n\n`unidic2ud.load(UniDic,UDPipe)` loads a natural language processor pipeline, which uses `UniDic` for tokenizer POS-tagger and lemmatizer, then uses `UDPipe` for dependency-parser. The default `UDPipe` is `UDPipe=\"japanese-modern\"`. Available `UniDic` options are:\n\n* `UniDic=\"gendai\"`: Use [現代書き言葉UniDic](https://clrd.ninjal.ac.jp/unidic/download_all.html#unidic_bccwj).\n* `UniDic=\"spoken\"`: Use [現代話し言葉UniDic](https://clrd.ninjal.ac.jp/unidic/download_all.html#unidic_csj).\n* `UniDic=\"novel\"`: Use [近現代口語小説UniDic](https://clrd.ninjal.ac.jp/unidic/download_all.html#unidic_novel).\n* `UniDic=\"qkana\"`: Use [旧仮名口語UniDic](https://clrd.ninjal.ac.jp/unidic/download_all.html#unidic_qkana).\n* `UniDic=\"kindai\"`: Use [近代文語UniDic](https://clrd.ninjal.ac.jp/unidic/download_all.html#unidic_kindai).\n* `UniDic=\"kinsei\"`: Use [近世江戸口語UniDic](https://clrd.ninjal.ac.jp/unidic/download_all.html#unidic_kinsei-edo).\n* `UniDic=\"kyogen\"`: Use [中世口語UniDic](https://clrd.ninjal.ac.jp/unidic/download_all.html#unidic_chusei-kougo).\n* `UniDic=\"wakan\"`: Use [中世文語UniDic](https://clrd.ninjal.ac.jp/unidic/download_all.html#unidic_chusei-bungo).\n* `UniDic=\"wabun\"`: Use [中古和文UniDic](https://clrd.ninjal.ac.jp/unidic/download_all.html#unidic_wabun).\n* `UniDic=\"manyo\"`: Use [上代語UniDic](https://clrd.ninjal.ac.jp/unidic/download_all.html#unidic_jodai).\n* `UniDic=None`: Use `UDPipe` for tokenizer, POS-tagger, lemmatizer, and dependency-parser.\n\n`unidic2ud.UniDic2UDEntry.to_tree()` has an option `to_tree(BoxDrawingWidth=2)` for old terminals, whose Box Drawing characters are \"fullwidth\".\n\nYou can simply use `unidic2ud` on the command line:\n```sh\necho 其國を治めんと欲する者は先づ其家を齊ふ | unidic2ud -U kindai\n```\n\n## CaboCha emulator usage\n\n```py\n\u003e\u003e\u003e import unidic2ud.cabocha as CaboCha\n\u003e\u003e\u003e c=CaboCha.Parser(\"kindai\")\n\u003e\u003e\u003e s=c.parse(\"其國を治めんと欲する者は先づ其家を齊ふ\")\n\u003e\u003e\u003e print(s.toString(CaboCha.FORMAT_TREE_LATTICE))\n  其-D\n  國を-D\n治めんと-D\n    欲する-D\n        者は-------D\n          先づ-----D\n              其-D |\n              家を-D\n                齊ふ\nEOS\n* 0 1D 0/0 0.000000\n其\t連体詞,*,*,*,*,*,其の,ソノ,*,DET\tO\t1\u003c-det-2\n* 1 2D 0/1 0.000000\n國\t名詞,普通名詞,一般,*,*,*,国,クニ,*,NOUN\tO\t2\u003c-obj-4\nを\t助詞,格助詞,*,*,*,*,を,ヲ,*,ADP\tO\t3\u003c-case-2\n* 2 3D 0/1 0.000000\n治め\t動詞,一般,*,*,*,*,収める,オサメ,*,VERB\tO\t4\u003c-advcl-7\nん\t助動詞,*,*,*,*,*,む,ン,*,AUX\tO\t5\u003c-aux-4\nと\t助詞,格助詞,*,*,*,*,と,ト,*,ADP\tO\t6\u003c-case-4\n* 3 4D 0/0 0.000000\n欲する\t動詞,一般,*,*,*,*,欲する,ホッスル,*,VERB\tO\t7\u003c-acl-8\n* 4 8D 0/1 0.000000\n者\t名詞,普通名詞,一般,*,*,*,者,モノ,*,NOUN\tO\t8\u003c-nsubj-14\nは\t助詞,係助詞,*,*,*,*,は,ハ,*,ADP\tO\t9\u003c-case-8\n* 5 8D 0/0 0.000000\n先づ\t副詞,*,*,*,*,*,先ず,マヅ,*,ADV\tO\t10\u003c-advmod-14\n* 6 7D 0/0 0.000000\n其\t連体詞,*,*,*,*,*,其の,ソノ,*,DET\tO\t11\u003c-det-12\n* 7 8D 0/1 0.000000\n家\t名詞,普通名詞,一般,*,*,*,家,ウチ,*,NOUN\tO\t12\u003c-obj-14\nを\t助詞,格助詞,*,*,*,*,を,ヲ,*,ADP\tO\t13\u003c-case-12\n* 8 -1D 0/0 0.000000\n齊ふ\t動詞,一般,*,*,*,*,整える,トトノフ,*,VERB\tO\t14\u003c-root\nEOS\n\u003e\u003e\u003e for c in [s.chunk(i) for i in range(s.chunk_size())]:\n...   if c.link\u003e=0:\n...     print(c,\"-\u003e\",s.chunk(c.link))\n...\n其 -\u003e 國を\n國を -\u003e 治めんと\n治めんと -\u003e 欲する\n欲する -\u003e 者は\n者は -\u003e 齊ふ\n先づ -\u003e 齊ふ\n其 -\u003e 家を\n家を -\u003e 齊ふ\n```\n`CaboCha.Parser(UniDic)` is an alias for `unidic2ud.load(UniDic,UDPipe=\"japanese-modern\")`, and its default is `UniDic=None`. `CaboCha.Tree.toString(format)` has five available formats:\n* `CaboCha.FORMAT_TREE`: tree (numbered as 0)\n* `CaboCha.FORMAT_LATTICE`: lattice (numbered as 1)\n* `CaboCha.FORMAT_TREE_LATTICE`: tree + lattice (numbered as 2)\n* `CaboCha.FORMAT_XML`: XML (numbered as 3)\n* `CaboCha.FORMAT_CONLL`: Universal Dependencies CoNLL-U (numbered as 4)\n\nYou can simply use `udcabocha` on the command line:\n```sh\necho 其國を治めんと欲する者は先づ其家を齊ふ | udcabocha -U kindai -f 2\n```\n`-U UniDic` specifies `UniDic`. `-f format` specifies the output format in 0 to 4 above (default is `-f 0`) and in 5 to 8 below:\n* `-f 5`: `to_tree()`\n* `-f 6`: `to_tree(BoxDrawingWidth=2)`\n* `-f 7`: `to_svg()`\n* `-f 8`: [raw DOT](https://graphviz.readthedocs.io/en/stable/manual.html#using-raw-dot) graph through [Immediate Catena Analysis](https://koichiyasuoka.github.io/deplacy/#deplacydot)\n\n![dot.png](https://raw.githubusercontent.com/KoichiYasuoka/UniDic2UD/master/dot.png)\n\nTry [notebook](https://colab.research.google.com/github/KoichiYasuoka/UniDic2UD/blob/master/udcabocha.ipynb) for Google Colaboratory.\n\n## Usage via spaCy\n\nIf you have already installed [spaCy](https://pypi.org/project/spacy/) 2.1.0 or later, you can use `UniDic` via spaCy Language pipeline.\n\n```py\n\u003e\u003e\u003e import unidic2ud.spacy\n\u003e\u003e\u003e nlp=unidic2ud.spacy.load(\"kindai\")\n\u003e\u003e\u003e d=nlp(\"其國を治めんと欲する者は先づ其家を齊ふ\")\n\u003e\u003e\u003e print(unidic2ud.spacy.to_conllu(d))\n# text = 其國を治めんと欲する者は先づ其家を齊ふ\n1\t其\t其の\tDET\t連体詞\t_\t2\tdet\t_\tSpaceAfter=No|Translit=ソノ\n2\t國\t国\tNOUN\t名詞-普通名詞-一般\t_\t4\tobj\t_\tSpaceAfter=No|Translit=クニ\n3\tを\tを\tADP\t助詞-格助詞\t_\t2\tcase\t_\tSpaceAfter=No|Translit=ヲ\n4\t治め\t収める\tVERB\t動詞-一般\t_\t7\tadvcl\t_\tSpaceAfter=No|Translit=オサメ\n5\tん\tむ\tAUX\t助動詞\t_\t4\taux\t_\tSpaceAfter=No|Translit=ン\n6\tと\tと\tADP\t助詞-格助詞\t_\t4\tcase\t_\tSpaceAfter=No|Translit=ト\n7\t欲する\t欲する\tVERB\t動詞-一般\t_\t8\tacl\t_\tSpaceAfter=No|Translit=ホッスル\n8\t者\t者\tNOUN\t名詞-普通名詞-一般\t_\t14\tnsubj\t_\tSpaceAfter=No|Translit=モノ\n9\tは\tは\tADP\t助詞-係助詞\t_\t8\tcase\t_\tSpaceAfter=No|Translit=ハ\n10\t先づ\t先ず\tADV\t副詞\t_\t14\tadvmod\t_\tSpaceAfter=No|Translit=マヅ\n11\t其\t其の\tDET\t連体詞\t_\t12\tdet\t_\tSpaceAfter=No|Translit=ソノ\n12\t家\t家\tNOUN\t名詞-普通名詞-一般\t_\t14\tobj\t_\tSpaceAfter=No|Translit=ウチ\n13\tを\tを\tADP\t助詞-格助詞\t_\t12\tcase\t_\tSpaceAfter=No|Translit=ヲ\n14\t齊ふ\t整える\tVERB\t動詞-一般\t_\t0\troot\t_\tSpaceAfter=No|Translit=トトノフ\n\n\u003e\u003e\u003e t=d[6]\n\u003e\u003e\u003e print(t.i+1,t.orth_,t.lemma_,t.pos_,t.tag_,t.head.i+1,t.dep_,t.whitespace_,t.norm_)\n7 欲する 欲する VERB 動詞-一般 8 acl  ホッスル\n\n\u003e\u003e\u003e from deplacy.deprelja import deprelja\n\u003e\u003e\u003e for b in unidic2ud.spacy.bunsetu_spans(d):\n...   for t in b.lefts:\n...     print(unidic2ud.spacy.bunsetu_span(t),\"-\u003e\",b,\"(\"+deprelja[t.dep_]+\")\")\n...\n其 -\u003e 國を (決定詞)\n國を -\u003e 治めんと (目的語)\n治めんと -\u003e 欲する (連用修飾節)\n欲する -\u003e 者は (連体修飾節)\n其 -\u003e 家を (決定詞)\n者は -\u003e 齊ふ (主語)\n先づ -\u003e 齊ふ (連用修飾語)\n家を -\u003e 齊ふ (目的語)\n```\n\n`unidic2ud.spacy.load(UniDic,parser)` loads a spaCy pipeline, which uses `UniDic` for tokenizer POS-tagger and lemmatizer (as shown above), then uses `parser` for dependency-parser. The default `parser` is `parser=\"japanese-modern\"` and available options are:\n\n* `parser=\"ja_core_news_sm\"`: Use [spaCy Japanese model](https://spacy.io/models/ja) (small).\n* `parser=\"ja_core_news_md\"`: Use spaCy Japanese model (middle).\n* `parser=\"ja_core_news_lg\"`: Use spaCy Japanese model (large).\n* `parser=\"ja_ginza\"`: Use [GiNZA](https://github.com/megagonlabs/ginza).\n* `parser=\"japanese-gsd\"`: Use [UDPipe Japanese model](http://hdl.handle.net/11234/1-3131).\n* `parser=\"stanza_ja\"`: Use [Stanza Japanese model](https://stanfordnlp.github.io/stanza/available_models.html).\n\n## Installation for Linux\n\nTar-ball is available for Linux, and is installed by default when you use `pip`:\n```sh\npip install unidic2ud\n```\n\nBy default installation, `UniDic` is invoked through Web APIs. If you want to invoke them locally and faster, you can download `UniDic` which you use just as follows:\n```sh\npython -m unidic2ud download kindai\npython -m unidic2ud dictlist\n```\nLicenses of dictionaries and models are: GPL/LGPL/BSD for `gendai` and `spoken`; CC BY-NC-SA 4.0 for others.\n\n## Installation for Cygwin\n\nMake sure to get `gcc-g++` `python37-pip` `python37-devel` packages, and then:\n```sh\npip3.7 install unidic2ud\n```\nUse `python3.7` command in [Cygwin](https://www.cygwin.com/install.html) instead of `python`.\n\n## Installation for Jupyter Notebook (Google Colaboratory)\n\n```py\n!pip install unidic2ud\n```\n## Benchmarks\n\nResults of [舞姬/雪國/荒野より-Benchmarks](https://colab.research.google.com/github/KoichiYasuoka/UniDic2UD/blob/master/benchmark/benchmark.ipynb)\n\n|[舞姬](https://github.com/KoichiYasuoka/UniDic2UD/blob/master/benchmark/maihime-benchmark.tar.gz)|LAS|MLAS|BLEX|\n|---------------|-----|-----|-----|\n|UniDic=\"kindai\"|81.13|70.37|77.78|\n|UniDic=\"qkana\" |79.25|70.37|77.78|\n|UniDic=\"kinsei\"|72.22|60.71|64.29|\n\n|[雪國](https://github.com/KoichiYasuoka/UniDic2UD/blob/master/benchmark/yukiguni-benchmark.tar.gz)|LAS|MLAS|BLEX|\n|---------------|-----|-----|-----|\n|UniDic=\"qkana\" |89.29|85.71|81.63|\n|UniDic=\"kinsei\"|89.29|85.71|77.55|\n|UniDic=\"kindai\"|84.96|81.63|77.55|\n\n|[荒野より](https://github.com/KoichiYasuoka/UniDic2UD/blob/master/benchmark/koyayori-benchmark.tar.gz)|LAS|MLAS|BLEX|\n|---------------|-----|-----|-----|\n|UniDic=\"kindai\"|76.44|61.54|53.85|\n|UniDic=\"qkana\" |75.39|61.54|53.85|\n|UniDic=\"kinsei\"|71.88|58.97|51.28|\n\n## Author\n\nKoichi Yasuoka (安岡孝一)\n\n## References\n\n* 安岡孝一: [形態素解析部の付け替えによる近代日本語(旧字旧仮名)の係り受け解析](http://hdl.handle.net/2433/254677), 情報処理学会研究報告, Vol.2020-CH-124「人文科学とコンピュータ」, No.3 (2020年9月5日), pp.1-8.\n* 安岡孝一: [漢日英Universal Dependencies平行コーパスとその差異](http://hdl.handle.net/2433/245218), 人文科学とコンピュータシンポジウム「じんもんこん2019」論文集 (2019年12月), pp.43-50.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoichiyasuoka%2Funidic2ud","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkoichiyasuoka%2Funidic2ud","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoichiyasuoka%2Funidic2ud/lists"}