{"id":18336292,"url":"https://github.com/byconity/cppjieba","last_synced_at":"2025-04-09T19:55:21.673Z","repository":{"id":235470683,"uuid":"730138623","full_name":"ByConity/cppjieba","owner":"ByConity","description":null,"archived":false,"fork":false,"pushed_at":"2023-12-13T06:03:25.000Z","size":9583,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":6,"default_branch":"clickhouse-tokenizer-version","last_synced_at":"2025-04-09T19:55:12.486Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ByConity.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-12-11T09:39:01.000Z","updated_at":"2024-09-30T00:40:51.000Z","dependencies_parsed_at":"2024-04-23T14:19:17.782Z","dependency_job_id":null,"html_url":"https://github.com/ByConity/cppjieba","commit_stats":null,"previous_names":["byconity/cppjieba"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ByConity%2Fcppjieba","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ByConity%2Fcppjieba/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ByConity%2Fcppjieba/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ByConity%2Fcppjieba/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ByConity","download_url":"https://codeload.github.com/ByConity/cppjieba/tar.gz/refs/heads/clickhouse-tokenizer-version","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248103911,"owners_count":21048245,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T20:07:13.339Z","updated_at":"2025-04-09T19:55:21.648Z","avatar_url":"https://github.com/ByConity.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CppJieba [English](README_EN.md)\n\n[![CMake](https://github.com/yanyiwu/cppjieba/actions/workflows/cmake.yml/badge.svg)](https://github.com/yanyiwu/cppjieba/actions/workflows/cmake.yml)\n[![Author](https://img.shields.io/badge/author-@yanyiwu-blue.svg?style=flat)](http://yanyiwu.com/) \n[![Platform](https://img.shields.io/badge/platform-Linux,%20OS%20X,%20Windows-green.svg?style=flat)](https://github.com/yanyiwu/cppjieba)\n[![Performance](https://img.shields.io/badge/performance-excellent-brightgreen.svg?style=flat)](http://yanyiwu.com/work/2015/06/14/jieba-series-performance-test.html) \n[![Tag](https://img.shields.io/github/v/tag/yanyiwu/cppjieba.svg)](https://github.com/yanyiwu/cppjieba/releases)\n[![License](https://img.shields.io/badge/license-MIT-yellow.svg?style=flat)](http://yanyiwu.mit-license.org)\n[![Build status](https://ci.appveyor.com/api/projects/status/wl30fjnm2rhft6ta/branch/master?svg=true)](https://ci.appveyor.com/project/yanyiwu/cppjieba/branch/master)\n\n## 简介\n\nCppJieba是\"结巴(Jieba)\"中文分词的C++版本\n\n## 特性\n\n+ 源代码都写进头文件`include/cppjieba/*.hpp`里，`include`即可使用。\n+ 支持`utf8`编码。\n+ 项目自带较为完善的单元测试，核心功能中文分词(utf8)的稳定性接受过线上环境检验。\n+ 支持载自定义用户词典，多路径时支持分隔符'|'或者';'分隔。\n+ 支持 `Linux` , `Mac OSX`, `Windows` 操作系统。\n\n## 用法\n\n### 依赖软件\n\n* `g++ (version \u003e= 4.1 is recommended) or clang++`;\n* `cmake (version \u003e= 2.6 is recommended)`;\n\n### 下载和编译\n\n```sh\ngit clone --depth=10 --branch=master git://github.com/yanyiwu/cppjieba.git\ncd cppjieba\nmkdir build\ncd build\ncmake ..\nmake\n```\n\n有兴趣的可以跑跑测试(可选):\n\n```\nmake test\n```\n\n## Demo\n\n```\n./demo\n```\n\n结果示例：\n\n```\n[demo] Cut With HMM\n他/来到/了/网易/杭研/大厦\n[demo] Cut Without HMM\n他/来到/了/网易/杭/研/大厦\n我来到北京清华大学\n[demo] CutAll\n我/来到/北京/清华/清华大学/华大/大学\n小明硕士毕业于中国科学院计算所，后在日本京都大学深造\n[demo] CutForSearch\n小明/硕士/毕业/于/中国/科学/学院/科学院/中国科学院/计算/计算所/，/后/在/日本/京都/大学/日本京都大学/深造\n[demo] Insert User Word\n男默/女泪\n男默女泪\n[demo] CutForSearch Word With Offset\n[{\"word\": \"小明\", \"offset\": 0}, {\"word\": \"硕士\", \"offset\": 6}, {\"word\": \"毕业\", \"offset\": 12}, {\"word\": \"于\", \"offset\": 18}, {\"word\": \"中国\", \"offset\": 21}, {\"word\": \"科学\", \"offset\": 27}, {\"word\": \"学院\", \"offset\": 30}, {\"word\": \"科学院\", \"offset\": 27}, {\"word\": \"中国科学院\", \"offset\": 21}, {\"word\": \"计算\", \"offset\": 36}, {\"word\": \"计算所\", \"offset\": 36}, {\"word\": \"，\", \"offset\": 45}, {\"word\": \"后\", \"offset\": 48}, {\"word\": \"在\", \"offset\": 51}, {\"word\": \"日本\", \"offset\": 54}, {\"word\": \"京都\", \"offset\": 60}, {\"word\": \"大学\", \"offset\": 66}, {\"word\": \"日本京都大学\", \"offset\": 54}, {\"word\": \"深造\", \"offset\": 72}]\n[demo] Tagging\n我是拖拉机学院手扶拖拉机专业的。不用多久，我就会升职加薪，当上CEO，走上人生巅峰。\n[我:r, 是:v, 拖拉机:n, 学院:n, 手扶拖拉机:n, 专业:n, 的:uj, 。:x, 不用:v, 多久:m, ，:x, 我:r, 就:d, 会:v, 升职:v, 加薪:nr, ，:x, 当上:t, CEO:eng, ，:x, 走上:v, 人生:n, 巅峰:n, 。:x]\n[demo] Keyword Extraction\n我是拖拉机学院手扶拖拉机专业的。不用多久，我就会升职加薪，当上CEO，走上人生巅峰。\n[{\"word\": \"CEO\", \"offset\": [93], \"weight\": 11.7392}, {\"word\": \"升职\", \"offset\": [72], \"weight\": 10.8562}, {\"word\": \"加薪\", \"offset\": [78], \"weight\": 10.6426}, {\"word\": \"手扶拖拉机\", \"offset\": [21], \"weight\": 10.0089}, {\"word\": \"巅峰\", \"offset\": [111], \"weight\": 9.49396}]\n```\n\n详细请看 `test/demo.cpp`.\n\n### 分词结果示例\n\n**MPSegment**\n\nOutput:\n```\n我来到北京清华大学\n我/来到/北京/清华大学\n\n他来到了网易杭研大厦\n他/来到/了/网易/杭/研/大厦\n\n小明硕士毕业于中国科学院计算所，后在日本京都大学深造\n小/明/硕士/毕业/于/中国科学院/计算所/，/后/在/日本京都大学/深造\n\n```\n\n**HMMSegment**\n\n```\n我来到北京清华大学\n我来/到/北京/清华大学\n\n他来到了网易杭研大厦\n他来/到/了/网易/杭/研大厦\n\n小明硕士毕业于中国科学院计算所，后在日本京都大学深造\n小明/硕士/毕业于/中国/科学院/计算所/，/后/在/日/本/京/都/大/学/深/造\n\n```\n\n**MixSegment**\n\n```\n我来到北京清华大学\n我/来到/北京/清华大学\n\n他来到了网易杭研大厦\n他/来到/了/网易/杭研/大厦\n\n小明硕士毕业于中国科学院计算所，后在日本京都大学深造\n小明/硕士/毕业/于/中国科学院/计算所/，/后/在/日本京都大学/深造\n\n```\n\n**FullSegment**\n\n```\n我来到北京清华大学\n我/来到/北京/清华/清华大学/华大/大学\n\n他来到了网易杭研大厦\n他/来到/了/网易/杭/研/大厦\n\n小明硕士毕业于中国科学院计算所，后在日本京都大学深造\n小/明/硕士/毕业/于/中国/中国科学院/科学/科学院/学院/计算/计算所/，/后/在/日本/日本京都大学/京都/京都大学/大学/深造\n\n```\n\n**QuerySegment**\n\n```\n我来到北京清华大学\n我/来到/北京/清华/清华大学/华大/大学\n\n他来到了网易杭研大厦\n他/来到/了/网易/杭研/大厦\n\n小明硕士毕业于中国科学院计算所，后在日本京都大学深造\n小明/硕士/毕业/于/中国/中国科学院/科学/科学院/学院/计算所/，/后/在/中国/中国科学院/科学/科学院/学院/日本/日本京都大学/京都/京都大学/大学/深造\n\n```\n\n以上依次是MP,HMM,Mix三种方法的效果。  \n\n可以看出效果最好的是Mix，也就是融合MP和HMM的切词算法。即可以准确切出词典已有的词，又可以切出像\"杭研\"这样的未登录词。\n\nFull方法切出所有字典里的词语。\n\nQuery方法先使用Mix方法切词，对于切出来的较长的词再使用Full方法。\n\n### 自定义用户词典\n\n自定义词典示例请看`dict/user.dict.utf8`。\n\n没有使用自定义用户词典时的结果:\n\n```\n令狐冲/是/云/计算/行业/的/专家\n```\n\n使用自定义用户词典时的结果:\n\n```\n令狐冲/是/云计算/行业/的/专家\n```\n\n### 关键词抽取\n\n```\n我是拖拉机学院手扶拖拉机专业的。不用多久，我就会升职加薪，当上CEO，走上人生巅峰。\n[\"CEO:11.7392\", \"升职:10.8562\", \"加薪:10.6426\", \"手扶拖拉机:10.0089\", \"巅峰:9.49396\"]\n```\n\n详细请见 `test/demo.cpp`.\n\n### 词性标注\n\n```\n我是蓝翔技工拖拉机学院手扶拖拉机专业的。不用多久，我就会升职加薪，当上总经理，出任CEO，迎娶白富美，走上人生巅峰。\n[\"我:r\", \"是:v\", \"拖拉机:n\", \"学院:n\", \"手扶拖拉机:n\", \"专业:n\", \"的:uj\", \"。:x\", \"不用:v\", \"多久:m\", \"，:x\", \"我:r\", \"就:d\", \"会:v\", \"升职:v\", \"加薪:nr\", \"，:x\", \"当上:t\", \"CEO:eng\", \"，:x\", \"走上:v\", \"人生:n\", \"巅峰:n\", \"。:x\"]\n```\n\n详细请看 `test/demo.cpp`.\n\n支持自定义词性。\n比如在(`dict/user.dict.utf8`)增加一行\n\n```\n蓝翔 nz\n```\n\n结果如下：\n\n```\n[\"我:r\", \"是:v\", \"蓝翔:nz\", \"技工:n\", \"拖拉机:n\", \"学院:n\", \"手扶拖拉机:n\", \"专业:n\", \"的:uj\", \"。:x\", \"不用:v\", \"多久:m\", \"，:x\", \"我:r\", \"就:d\", \"会:v\", \"升职:v\", \"加薪:nr\", \"，:x\", \"当:t\", \"上:f\", \"总经理:n\", \"，:x\", \"出任:v\", \"CEO:eng\", \"，:x\", \"迎娶:v\", \"白富美:x\", \"，:x\", \"走上:v\", \"人生:n\", \"巅峰:n\", \"。:x\"]\n```\n\n## 其它词典资料分享\n\n+ [dict.367W.utf8] iLife(562193561 at qq.com)\n\n## 应用\n\n+ [GoJieba] go语言版本的结巴中文分词。\n+ [NodeJieba] Node.js 版本的结巴中文分词。\n+ [simhash] 中文文档的的相似度计算\n+ [exjieba] Erlang 版本的结巴中文分词。\n+ [jiebaR] R语言版本的结巴中文分词。\n+ [cjieba] C语言版本的结巴分词。\n+ [jieba_rb] Ruby 版本的结巴分词。\n+ [iosjieba] iOS 版本的结巴分词。\n+ [SqlJieba] MySQL 全文索引的结巴中文分词插件。\n+ [pg_jieba] PostgreSQL 数据库的分词插件。\n+ [simple] SQLite3 FTS5 数据库的分词插件。\n+ [gitbook-plugin-search-pro] 支持中文搜索的 gitbook 插件。\n+ [ngx_http_cppjieba_module] Nginx 分词插件。\n+ [cppjiebapy] 由 [jannson] 开发的供 python 模块调用的项目 [cppjiebapy], 相关讨论 [cppjiebapy_discussion] .\n+ [cppjieba-py] 由 [bung87] 基于 pybind11 封装的 python 模块,使用体验上接近于原jieba。\n+ [KeywordServer] 50行搭建一个中文关键词抽取服务。\n+ [cppjieba-server] CppJieba HTTP 服务器。\n+ [phpjieba] php版本的结巴分词扩展。\n+ [perl5-jieba] Perl版本的结巴分词扩展。\n+ [jieba-dlang] D 语言的结巴分词 Deimos Bindings。\n\n## 线上演示\n\n[Web-Demo](http://cppjieba-webdemo.herokuapp.com/)\n(建议使用chrome打开)\n\n## 性能评测\n\n[Jieba中文分词系列性能评测]\n\n## Sponsorship\n\n[![sponsorship](http://images.gitads.io/cppjieba)](https://tracking.gitads.io/?campaign=gitads\u0026repo=cppjieba\u0026redirect=gitads.io)\n\n## Contributors\n\n### Code Contributors\n\nThis project exists thanks to all the people who contribute.\n\u003ca href=\"https://github.com/yanyiwu/cppjieba/graphs/contributors\"\u003e\u003cimg src=\"https://opencollective.com/cppjieba/contributors.svg?width=890\u0026button=false\" /\u003e\u003c/a\u003e\n\n[GoJieba]:https://github.com/yanyiwu/gojieba\n[CppJieba]:https://github.com/yanyiwu/cppjieba\n[jannson]:https://github.com/jannson\n[cppjiebapy]:https://github.com/jannson/cppjiebapy\n[bung87]:https://github.com/bung87\n[cppjieba-py]:https://github.com/bung87/cppjieba-py\n[cppjiebapy_discussion]:https://github.com/yanyiwu/cppjieba/issues/1\n[NodeJieba]:https://github.com/yanyiwu/nodejieba\n[jiebaR]:https://github.com/qinwf/jiebaR\n[simhash]:https://github.com/yanyiwu/simhash\n[代码详解]:https://github.com/yanyiwu/cppjieba/wiki/CppJieba%E4%BB%A3%E7%A0%81%E8%AF%A6%E8%A7%A3\n[issue25]:https://github.com/yanyiwu/cppjieba/issues/25\n[exjieba]:https://github.com/falood/exjieba\n[KeywordServer]:https://github.com/yanyiwu/keyword_server\n[ngx_http_cppjieba_module]:https://github.com/yanyiwu/ngx_http_cppjieba_module\n[dict.367W.utf8]:https://github.com/qinwf/BigDict\n[cjieba]:http://github.com/yanyiwu/cjieba\n[jieba_rb]:https://github.com/altkatz/jieba_rb\n[iosjieba]:https://github.com/yanyiwu/iosjieba\n[SqlJieba]:https://github.com/yanyiwu/sqljieba\n[Jieba中文分词系列性能评测]:http://yanyiwu.com/work/2015/06/14/jieba-series-performance-test.html\n[pg_jieba]:https://github.com/jaiminpan/pg_jieba\n[gitbook-plugin-search-pro]:https://plugins.gitbook.com/plugin/search-pro\n[cppjieba-server]:https://github.com/yanyiwu/cppjieba-server\n[phpjieba]:https://github.com/jonnywang/phpjieba\n[perl5-jieba]:https://metacpan.org/pod/distribution/Lingua-ZH-Jieba/lib/Lingua/ZH/Jieba.pod\n[jieba-dlang]:https://github.com/shove70/jieba\n[simple]:https://github.com/wangfenjin/simple\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbyconity%2Fcppjieba","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbyconity%2Fcppjieba","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbyconity%2Fcppjieba/lists"}