{"id":13599829,"url":"https://github.com/daac-tools/vibrato","last_synced_at":"2025-05-15T03:06:19.439Z","repository":{"id":57819943,"uuid":"511060177","full_name":"daac-tools/vibrato","owner":"daac-tools","description":"🎤 vibrato: Viterbi-based accelerated tokenizer","archived":false,"fork":false,"pushed_at":"2025-04-11T16:51:11.000Z","size":1136,"stargazers_count":357,"open_issues_count":8,"forks_count":15,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-05-13T03:39:29.097Z","etag":null,"topics":["japanese","morphological-analysis","nlp","rust","segmentation","tokenization","tokenizer"],"latest_commit_sha":null,"homepage":"https://docs.rs/vibrato","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/daac-tools.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-07-06T08:50:52.000Z","updated_at":"2025-05-08T07:53:39.000Z","dependencies_parsed_at":"2023-11-28T02:31:20.001Z","dependency_job_id":"7fe0923a-08d5-4e1b-b81c-ebadcef74e87","html_url":"https://github.com/daac-tools/vibrato","commit_stats":{"total_commits":156,"total_committers":3,"mean_commits":52.0,"dds":"0.46153846153846156","last_synced_commit":"6b61e2327c1bd76c7a9bc4c44c8bc3cede6070d7"},"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daac-tools%2Fvibrato","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daac-tools%2Fvibrato/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daac-tools%2Fvibrato/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daac-tools%2Fvibrato/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/daac-tools","download_url":"https://codeload.github.com/daac-tools/vibrato/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254264765,"owners_count":22041793,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["japanese","morphological-analysis","nlp","rust","segmentation","tokenization","tokenizer"],"created_at":"2024-08-01T17:01:12.813Z","updated_at":"2025-05-15T03:06:19.416Z","avatar_url":"https://github.com/daac-tools.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# 🎤 vibrato: VIterbi-Based acceleRAted TOkenizer\n\n[![Crates.io](https://img.shields.io/crates/v/vibrato)](https://crates.io/crates/vibrato)\n[![Documentation](https://docs.rs/vibrato/badge.svg)](https://docs.rs/vibrato)\n[![Build Status](https://github.com/daac-tools/vibrato/actions/workflows/rust.yml/badge.svg)](https://github.com/daac-tools/vibrato/actions)\n[![Slack](https://img.shields.io/badge/join-chat-brightgreen?logo=slack)](https://join.slack.com/t/daac-tools/shared_invite/zt-1pwwqbcz4-KxL95Nam9VinpPlzUpEGyA)\n\nVibrato is a fast implementation of tokenization (or morphological analysis) based on the Viterbi algorithm.\n\nA Python wrapper is also available [here](https://github.com/daac-tools/python-vibrato).\n\n[Wasm Demo](https://vibrato-demo.pages.dev/) (takes a little time to load the model.)\n\n## Features\n\n### Fast tokenization\n\nVibrato is a Rust reimplementation of the fast tokenizer [MeCab](https://taku910.github.io/mecab/),\nalthough its implementation has been simplified and optimized for even faster tokenization.\nEspecially for language resources with a large matrix\n(e.g., [`unidic-cwj-3.1.1`](https://clrd.ninjal.ac.jp/unidic/back_number.html#unidic_cwj) with a matrix of 459 MiB),\nVibrato will run faster thanks to cache-efficient id mappings.\n\nFor example, the following figure shows an experimental result of\ntokenization time with MeCab and its reimplementations.\nThe detailed experimental settings and other results are available on [Wiki](https://github.com/daac-tools/vibrato/wiki/Speed-Comparison).\n\n![](./figures/comparison.svg)\n\n### MeCab compatibility\n\nVibrato supports options for outputting tokenized results identical to MeCab, such as ignoring whitespace.\n\n### Training parameters\n\nVibrato also supports training parameters (or costs) in dictionaries from your corpora.\nThe detailed description can be found [here](./docs/train.md).\n\n## Basic usage\n\nThis software is implemented in Rust.\nFirst of all, install `rustc` and `cargo` following the [official instructions](https://www.rust-lang.org/tools/install).\n\n### 1. Dictionary preparation\n\nYou can easily get started with Vibrato by downloading a precompiled dictionary.\n[The Releases page](https://github.com/daac-tools/vibrato/releases) distributes\nseveral precompiled dictionaries from different resources.\n\nHere, consider to use [mecab-ipadic v2.7.0](https://taku910.github.io/mecab/).\n(Specify an appropriate Vibrato release tag to `VERSION` such as `v0.5.0`.)\n\n```\n$ wget https://github.com/daac-tools/vibrato/releases/download/VERSION/ipadic-mecab-2_7_0.tar.xz\n$ tar xf ipadic-mecab-2_7_0.tar.xz\n```\n\nYou can also compile or train system dictionaries from your own resources.\nSee the [docs](./docs/) for more advanced usage.\n\n### 2. Tokenization\n\nTo tokenize sentences using the system dictionary, run the following command.\n\n```\n$ echo '本とカレーの街神保町へようこそ。' | cargo run --release -p tokenize -- -i ipadic-mecab-2_7_0/system.dic.zst\n```\n\nThe resultant tokens will be output in the Mecab format.\n\n```\n本\t名詞,一般,*,*,*,*,本,ホン,ホン\nと\t助詞,並立助詞,*,*,*,*,と,ト,ト\nカレー\t名詞,固有名詞,地域,一般,*,*,カレー,カレー,カレー\nの\t助詞,連体化,*,*,*,*,の,ノ,ノ\n街\t名詞,一般,*,*,*,*,街,マチ,マチ\n神保\t名詞,固有名詞,地域,一般,*,*,神保,ジンボウ,ジンボー\n町\t名詞,接尾,地域,*,*,*,町,マチ,マチ\nへ\t助詞,格助詞,一般,*,*,*,へ,ヘ,エ\nようこそ\t感動詞,*,*,*,*,*,ようこそ,ヨウコソ,ヨーコソ\n。\t記号,句点,*,*,*,*,。,。,。\nEOS\n```\n\nIf you want to output tokens separated by spaces, specify `-O wakati`.\n\n```\n$ echo '本とカレーの街神保町へようこそ。' | cargo run --release -p tokenize -- -i ipadic-mecab-2_7_0/system.dic.zst -O wakati\n本 と カレー の 街 神保 町 へ ようこそ 。\n```\n\n### Notes for Vibrato APIs\n\nThe distributed models are compressed in zstd format.\nIf you want to load these compressed models with the `vibrato` API,\nyou must decompress them outside of the API.\n\n```rust\n// Requires zstd crate or ruzstd crate\nlet reader = zstd::Decoder::new(File::open(\"path/to/system.dic.zst\")?)?;\nlet dict = Dictionary::read(reader)?;\n```\n\n## Tokenization options\n\n### MeCab-compatible options\n\nVibrato is a reimplementation of the MeCab algorithm,\nbut with the default settings it can produce different tokens from MeCab.\n\nFor example, MeCab ignores spaces (more precisely, `SPACE` defined in `char.def`) in tokenization.\n\n```\n$ echo \"mens second bag\" | mecab\nmens\t名詞,固有名詞,組織,*,*,*,*\nsecond\t名詞,一般,*,*,*,*,*\nbag\t名詞,固有名詞,組織,*,*,*,*\nEOS\n```\n\nHowever, Vibrato handles such spaces as tokens with the default settings.\n\n```\n$ echo 'mens second bag' | cargo run --release -p tokenize -- -i ipadic-mecab-2_7_0/system.dic.zst\nmens\t名詞,固有名詞,組織,*,*,*,*\n \t記号,空白,*,*,*,*,*\nsecond\t名詞,固有名詞,組織,*,*,*,*\n \t記号,空白,*,*,*,*,*\nbag\t名詞,固有名詞,組織,*,*,*,*\nEOS\n```\n\nIf you want to obtain the same results as MeCab, specify the arguments `-S` and `-M 24`.\n\n```\n$ echo 'mens second bag' | cargo run --release -p tokenize -- -i ipadic-mecab-2_7_0/system.dic.zst -S -M 24\nmens\t名詞,固有名詞,組織,*,*,*,*\nsecond\t名詞,一般,*,*,*,*,*\nbag\t名詞,固有名詞,組織,*,*,*,*\nEOS\n```\n\n`-S` indicates if spaces are ignored.\n`-M` indicates the maximum grouping length for unknown words.\n\n#### Notes\n\nThere are corner cases where tokenization results in different outcomes due to cost tiebreakers.\nHowever, this would be not an essential problem.\n\n### User dictionary\n\nYou can use your user dictionary along with the system dictionary.\nThe user dictionary must be in the CSV format.\n\n```\n\u003csurface\u003e,\u003cleft-id\u003e,\u003cright-id\u003e,\u003ccost\u003e,\u003cfeatures...\u003e\n```\n\nThe first four columns are always required.\nThe others (i.e., `\u003cfeatures...\u003e`) are optional.\n\nFor example,\n\n```\n$ cat user.csv\n神保町,1293,1293,334,カスタム名詞,ジンボチョウ\n本とカレーの街,1293,1293,0,カスタム名詞,ホントカレーノマチ\nようこそ,3,3,-1000,感動詞,ヨーコソ,Welcome,欢迎欢迎,Benvenuto,Willkommen\n```\n\nTo use the user dictionary, specify the file with the `-u` argument.\n\n```\n$ echo '本とカレーの街神保町へようこそ。' | cargo run --release -p tokenize -- -i ipadic-mecab-2_7_0/system.dic.zst -u user.csv\n本とカレーの街\tカスタム名詞,ホントカレーノマチ\n神保町\tカスタム名詞,ジンボチョウ\nへ\t助詞,格助詞,一般,*,*,*,へ,ヘ,エ\nようこそ\t感動詞,ヨーコソ,Welcome,欢迎欢迎,Benvenuto,Willkommen\n。\t記号,句点,*,*,*,*,。,。,。\nEOS\n```\n\n## More advanced usages\n\nThe directory [docs](./docs/) provides descriptions of more advanced usages such as training or benchmarking.\n\n## Slack\n\nWe have a Slack workspace for developers and users to ask questions and discuss a variety of topics.\n\n * https://daac-tools.slack.com/\n * Please get an invitation from [here](https://join.slack.com/t/daac-tools/shared_invite/zt-1pwwqbcz4-KxL95Nam9VinpPlzUpEGyA).\n\n## License\n\nLicensed under either of\n\n * Apache License, Version 2.0\n   ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)\n * MIT license\n   ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)\n\nat your option.\n\n## Contribution\n\nSee [the guidelines](./CONTRIBUTING.md).\n\n## References\n\nTechnical details of Vibrato are available in the following resources:\n\n- 神田峻介, 赤部晃一, 後藤啓介, 小田悠介.\n  [最小コスト法に基づく形態素解析におけるCPUキャッシュの効率化](https://www.anlp.jp/proceedings/annual_meeting/2023/pdf_dir/C2-4.pdf),\n  言語処理学会第29回年次大会 (NLP2023).\n- 赤部晃一, 神田峻介, 小田悠介.\n  [CRFに基づく形態素解析器のスコア計算の分割によるモデルサイズと解析速度の調整](https://www.anlp.jp/proceedings/annual_meeting/2023/pdf_dir/C2-1.pdf),\n  言語処理学会第29回年次大会 (NLP2023).\n- [MeCab互換な形態素解析器Vibratoの高速化技法](https://tech.legalforce.co.jp/entry/2022/09/20/133132),\n  LegalOn Technologies Engineering Blog (2022-09-20).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaac-tools%2Fvibrato","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdaac-tools%2Fvibrato","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaac-tools%2Fvibrato/lists"}