{"id":13484322,"url":"https://github.com/p-geon/ja-tokenizer-docker-py","last_synced_at":"2025-03-27T16:30:41.544Z","repository":{"id":40413854,"uuid":"489845958","full_name":"p-geon/ja-tokenizer-docker-py","owner":"p-geon","description":"Mecab + NEologd + Docker + Python3","archived":false,"fork":false,"pushed_at":"2022-05-10T07:55:48.000Z","size":26,"stargazers_count":35,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-30T18:42:31.772Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/p-geon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-08T04:34:41.000Z","updated_at":"2023-08-02T18:43:05.000Z","dependencies_parsed_at":"2022-08-09T19:40:49.144Z","dependency_job_id":null,"html_url":"https://github.com/p-geon/ja-tokenizer-docker-py","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p-geon%2Fja-tokenizer-docker-py","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p-geon%2Fja-tokenizer-docker-py/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p-geon%2Fja-tokenizer-docker-py/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p-geon%2Fja-tokenizer-docker-py/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/p-geon","download_url":"https://codeload.github.com/p-geon/ja-tokenizer-docker-py/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245882223,"owners_count":20687853,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T17:01:22.512Z","updated_at":"2025-03-27T16:30:40.456Z","avatar_url":"https://github.com/p-geon.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook"],"sub_categories":[],"readme":"# 日本語言語処理つめあわせDockerコンテナ\n\n\u003c!-- - Docker HUB (TODO)--\u003e\n- Build Docker\n- **MeCab + NEologd**\n- **HuggingFace Tokenizer with Ja-Bert**\n- **Word2vec**\n- Show help\n\n## 解説\n\n時間があるときに作ります…ごめんなさい。\n\n\u003c!--\n## Docker Hub (TODO)\n\nhttps://hub.docker.com/repository/docker/hyperpigeon/ja-tokernizer-py\n\n（DockerHub 経由で `git clone` せずに分かち書きできるようにしたい。まだやってないですごめんなさい）\n\n```\ndocker pull hyperpigeon/ja-tokernizer-py\n...\n```\n--\u003e\n\n\n## 環境\n\n- Docker: `Docker version 20.10.9, build c2ea9bc` (おそらく19.以上なら大丈夫だと思います)\n\n\n## Build docker (local)\n\n`git clone` した後に `make run` すれば、すべてのコンテナが生成・実行されます。\n\n```\ngit clone git@github.com:p-geon/ja-tokenizer-docker-py.git\ncd ja-tokenizer-docker-py\nmake run\n```\n\n## Notebooks\n\n以下の三種類とも jupyter notebook を用意し、手を加えやすくしました。\nMakefile コマンドで `docker build` + `docker run` できます。\n起動後は URL をブラウザに貼り付けてブラウザで操作してください。\n\n- tokenizer_mecab_neologd: `make notebook_tokenizer_mecab_neologd`\n- tokenizer_huggingface: `make notebook_tokenizer_huggingface`\n- notebook_word2vec: `make notebook_word2vec`\n\n\n注意点として、環境とノートブックが一対一対応なので、\n\n`notebook_tokenizer_mecab_neologd` では `notebooks/tokenizer_mecab_neologd.ipynb` の環境しか動きません。\nほかも同様です。名前で判断してください。\n\n（なんかいい解決方法あったら教えて下さい。）\n\n\n## MeCab + NEologd\n\n[NEologd](https://github.com/neologd/mecab-ipadic-neologd) ベースの MeCab を使って分かち書きする簡易スクリプトです。\n`Makefile` にある変数 `TEST_SENTENCE` の文字列を分かち書きするスクリプトをコンテナ内で走らせます。\n\n**usage**\n\n```\nmake tokenizer_mecab_neologd\n```\n\n**result**\n\n```\ndocker run -it --rm \\\n\t-v `pwd`:/work \\\n\tja-tokenizer-mecab-neologd \\\n\tpython ./scripts/tokenizer_mecab_neologd.py \\\n\t\t--sentence \"ピジョンとジョン・レノンが融合してピジョンレノンと成った。\" \\\n\t\t--dir_dict /usr/lib/x86_64-linux-gnu/mecab/dic/mecab-ipadic-neologd\n\tBOS/EOS\nピジョン\t名詞\nと\t助詞\nジョン・レノン\t名詞\nが\t助詞\n融合\t名詞\nする\t動詞\nて\t助詞\nピジョンレノン\t名詞\nと\t助詞\n成る\t動詞\nた\t助動詞\n。\t記号\n\tBOS/EOS\n```\n\n\n## HuggingFace Tokenizer with Ja-Bert\n\n[東北大のBERT-Tokenizer](https://huggingface.co/cl-tohoku/bert-base-japanese) による分かち書きの簡易スクリプトをコンテナ内で走らせます。\n`Makefile` にある変数 `TEST_SENTENCE` の文字列を分かち書きするスクリプトをコンテナ内で走らせます。\n\nネットワーク負荷軽減のため、キャッシュを `src/huggingface/` 内に確保します。\n\n**usage**\n\n```\nmake tokenizer_huggingface\n```\n\n**result**\n\n```\ndocker run -it --rm \\\n\t-v `pwd`:/work \\\n\tja-tokenizer-touhoku-bert \\\n\tpython ./scripts/tokenizer_huggingface.py \\\n\t\t--sentence \"ピジョンとジョン・レノンが融合してピジョンレノンと成った。\"\nDownloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 104/104 [00:00\u003c00:00, 108kB/s]\nDownloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 479/479 [00:00\u003c00:00, 483kB/s]\nDownloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 252k/252k [00:00\u003c00:00, 316kB/s]\ntokenized:  ['ピ', '##ジョン', 'と', 'ジョン', '・', 'レノ', '##ン', 'が', '融合', 'し', 'て', 'ピ', '##ジョン', '##レノ', '##ン', 'と', '成っ', 'た', '。']\n```\n\n\n## Word2vec\n\n[gensim](https://radimrehurek.com/gensim/) + [日本語Wikipediaエンティティベクトル](http://www.cl.ecei.tohoku.ac.jp/~m-suzuki/jawiki_vector/) で単語をベクトル化します。\n`Makefile` にある変数 `TEST_WORD` の文字列を分かち書きするスクリプトをコンテナ内で走らせます。\n\n**usage**\n\n```\nmake word2vec\n```\n\n**result**\n\n```\ndocker run -it --rm \\\n\t-v `pwd`:/work \\\n\tja-word2vec \\\n\tpython ./scripts/word2vec.py \\\n\t\t--word \"ピジョン\" \\\n\t\t--bin_entity_filename /entity_vector.model.bin\n\u003cclass 'numpy.ndarray'\u003e (200,) [ 0.18742366  0.07586656  0.03849505  0.38004303 -0.00524301 -0.20294596\n  0.20340575  0.49917793 -0.2951132  -0.08509478]\n```\n\n(表示しているベクトルは都合上、先頭から10個分)\n\n\n## Show help\n\n`Makefile` のコマンドと、その概要を表示します。\n\n**usage**\n\n```\nmake help\n```\n\n**result**\n\n```\ntokenizer_mecab_neologd tokenizing with MeCab + NEologd\ntokenizer_huggingface tokenizing with huggingface tokenizer\nword2vec             convert word to vector (numpy)\nrun                  test all\nhelp                 show help (this)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fp-geon%2Fja-tokenizer-docker-py","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fp-geon%2Fja-tokenizer-docker-py","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fp-geon%2Fja-tokenizer-docker-py/lists"}