https://github.com/luozijun/rust-jieba

Rust jieba
https://github.com/luozijun/rust-jieba

hamming-distance hidden-markov-model hmm jaccard jieba minhash mmseg simhash similarity

Last synced: 2 months ago
JSON representation

Rust jieba

• Host: GitHub
• URL: https://github.com/luozijun/rust-jieba
• Owner: LuoZijun
• License: mit
• Created: 2018-08-28T02:49:44.000Z (almost 8 years ago)
• Default Branch: master
• Last Pushed: 2018-09-19T06:47:49.000Z (almost 8 years ago)
• Last Synced: 2024-04-15T07:23:23.908Z (over 2 years ago)
• Topics: hamming-distance, hidden-markov-model, hmm, jaccard, jieba, minhash, mmseg, simhash, similarity
• Language: Rust
• Size: 1.97 MB
• Stars: 2
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.rst
  • License: LICENSE

Awesome Lists containing this project

README

          
Rust Jieba

================

:Date: 08/28 2018

.. contents::

分词方法

----------

*   HMM

*   MMSEG

Build

-----------

.. code:: bash

    

    git clone https://github.com/LuoZijun/rust-jieba.git

    cd rust-jieba

    cargo run --bin mk_hmm_dict > src/hmm_dict.rs

    cargo run --bin mk_mmseg_dict > src/mk_mmseg_dict.rs

    cargo build

    cargo run --example hmm_cut

    cargo run --example mmseg_cut

    cargo test

    cargo bench

分词速度

----------

:样本: 钱钟书 - 《围城》

:处理器: Intel(R) Core(TM) i7-4770HQ @2.2 GHz

:单核吞吐量: ~= 4MB/s

参考资料

----------

*   `国家语委现代汉语语料库 `_

*   `互联网上开放的中文语料库有哪些 `_

*   `搜狗实验室_语料数据 `_