{"id":20751018,"url":"https://github.com/vaaaaanquish/jananese_giza_mecab","last_synced_at":"2026-04-21T12:02:44.376Z","repository":{"id":113485697,"uuid":"250277379","full_name":"vaaaaanquish/jananese_giza_mecab","owner":"vaaaaanquish","description":"統計的機械翻訳におけるアライメントツールGIZA++とmecab、neologdを動かすdockerとexample","archived":false,"fork":false,"pushed_at":"2020-03-27T04:44:51.000Z","size":5,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-18T03:40:39.147Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vaaaaanquish.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-26T14:14:14.000Z","updated_at":"2020-03-29T06:22:47.000Z","dependencies_parsed_at":"2023-06-28T17:46:41.209Z","dependency_job_id":null,"html_url":"https://github.com/vaaaaanquish/jananese_giza_mecab","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vaaaaanquish%2Fjananese_giza_mecab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vaaaaanquish%2Fjananese_giza_mecab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vaaaaanquish%2Fjananese_giza_mecab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vaaaaanquish%2Fjananese_giza_mecab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vaaaaanquish","download_url":"https://codeload.github.com/vaaaaanquish/jananese_giza_mecab/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243049595,"owners_count":20227861,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T08:29:42.677Z","updated_at":"2026-04-21T12:02:44.362Z","avatar_url":"https://github.com/vaaaaanquish.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# jananese_giza_mecab\n統計的機械翻訳におけるアライメントツールGIZA++を利用するため、日本語に対してmecab、neologdを動かすdockerとexample\n\n# Usage\n\ndocker buildでGIZA++、Python (pyenv)、MeCab、ipadic-neologdが入ったUbuntu環境ができる。\n```\n# build\ndocker build giza .\n\n# run\ndocker run -it giza\n```\n\n## example\n\n以下のサンプルを含む。\n\n - mecab_tokenize_example.py : mecabの辞書が扱えている事を確認する\n - make_example_data.py : GIZA++サンプル用の小さなデータ(en, ja)をmecabを使って生成する\n - train.sh : GIZA++を学習させる\n\n### mecab、neologdの確認\n\n```\npython mecab_tokenize_example.py\n\n# neologd true: neologd は 中居正広の金曜日のスマイルたちへ を 1つ の 単語 と し ます\n# neologd false: neologd は 中居 正広 の 金曜日 の スマイル たち へ を 1つ の 単語 と し ます\n```\n\n### GIZA++を動かす\n\n最小の学習データを生成する。英語(小文字化処理済み)と日本語(parse済み)のファイルがenとjaという名前で出力される。\n```\npython make_example_data.py\n```\n\nGIZA++を動かす。上記コマンドで生成したen、jaを入力としている。フォーマットなどはen、jaを参照。\n```\n./train.sh\n```\n\n以下のようにファイルが出力される。\n```\nroot@5608d61af3d6:~# ls -l\ntotal 120\n2020-03-27.043534.Anything.A3.final\n2020-03-27.043534.Anything.D4.final\n2020-03-27.043534.Anything.Decoder.config\n2020-03-27.043534.Anything.a3.final\n2020-03-27.043534.Anything.d3.final\n2020-03-27.043534.Anything.d4.final\n2020-03-27.043534.Anything.gizacfg\n2020-03-27.043534.Anything.n3.final\n2020-03-27.043534.Anything.p0_3.final\n2020-03-27.043534.Anything.perp\n2020-03-27.043534.Anything.t3.final\n2020-03-27.043534.Anything.trn.src.vcb\n2020-03-27.043534.Anything.trn.trg.vcb\n2020-03-27.043534.Anything.tst.src.vcb\n2020-03-27.043534.Anything.tst.trg.vcb\n```\n\n以下の3つのファイルが最終的な単語ID、頻度、尤度になっているので翻訳等に利用できる。\n- *.t3.final\n- *.trn.src.vcb\n- *.trn.trg.vcb \n\n\n# Reference\n参考文献。Mosesも含む。\n\n - GIZA++の実行、解説\n    - GIZA++ の使い方 how-to-use-giza-pp.md - mosmeh https://gist.github.com/mosmeh/80f239f7372e4caf87bf\n    - Mosesの後輩への引き継ぎのためのまとめ - kanjirz50 https://qiita.com/kanjirz50/items/c8ca50466b57a619e2a2\n    - Phrase-base Statistical Machine Translation - 中澤 敏明 https://slidesplayer.net/slide/14610301/\n    - Using GIZA++ http://wiki.apertium.org/wiki/Using_GIZA%2B%2B\n\t- Using GIZA++ to Obtain Word Alignment Between Bilingual Sentences - Masato Hagiwara https://masatohagiwara.net/using-giza-to-obtain-word-alignment-between-bilingual-sentences.html\n\t- 統計的機械翻訳ツールキットcicada - 自然言語処理 on Mac https://hjym-u.hatenadiary.org/entry/20131102/1383383771\n\t- 「音声言語処理と自然言語処理」演習 - 土屋 雅稔，山本 一公 httpsoronasha.co.jp/np/data/docs1/978-4-339-02888-1_2.pdf\n    - ニューラル機械翻訳の動向@IBIS2017 - Toshiaki Nakazawa https://www.slideshare.net/ToshiakiNakazawa/ibis2017\n - 拡張\n    - 単語境界が明示されていない言語を対象とした対訳辞書の自動構築 - 北陸先端科学技術大学院大学情報科学研究科 王 馨竹 https://dspace.jaist.ac.jp/dspace/bitstream/10119/14172/5/paper.pdf\n\t- HMM Word and Phrase Alignment for Statistical Machine Translation - Yonggang Deng http://www.mt-archive.info/HLT-EMNLP-2005-Deng.pdf\n    - Bayesian Subtree Alignment Model based on Dependency Trees - Toshiaki Nakazawa https://www.aclweb.org/anthology/I11-1089.pdf\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvaaaaanquish%2Fjananese_giza_mecab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvaaaaanquish%2Fjananese_giza_mecab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvaaaaanquish%2Fjananese_giza_mecab/lists"}