{"id":22551246,"url":"https://github.com/sing1ee/dict_build","last_synced_at":"2026-01-11T05:03:36.767Z","repository":{"id":21176812,"uuid":"24481309","full_name":"sing1ee/dict_build","owner":"sing1ee","description":"自动构建中文词库：http://www.matrix67.com/blog/archives/5044","archived":true,"fork":false,"pushed_at":"2023-12-05T22:08:13.000Z","size":7367,"stargazers_count":649,"open_issues_count":2,"forks_count":221,"subscribers_count":37,"default_branch":"master","last_synced_at":"2025-03-09T05:12:01.786Z","etag":null,"topics":["dict","java","sort"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sing1ee.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-09-26T01:21:06.000Z","updated_at":"2025-03-04T02:15:13.000Z","dependencies_parsed_at":"2025-01-18T23:28:06.127Z","dependency_job_id":"b16d304a-55a0-4885-b852-1e6f9230f733","html_url":"https://github.com/sing1ee/dict_build","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/sing1ee/dict_build","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sing1ee%2Fdict_build","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sing1ee%2Fdict_build/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sing1ee%2Fdict_build/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sing1ee%2Fdict_build/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sing1ee","download_url":"https://codeload.github.com/sing1ee/dict_build/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sing1ee%2Fdict_build/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28287096,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-11T04:44:51.577Z","status":"ssl_error","status_checked_at":"2026-01-11T04:44:44.232Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dict","java","sort"],"created_at":"2024-12-07T17:11:40.251Z","updated_at":"2026-01-11T05:03:36.750Z","avatar_url":"https://github.com/sing1ee.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"构建词库\n==========\n\n从原始文本中，自动构建词库，目前只适用于中文。参考：\n\nhttp://www.matrix67.com/blog/archives/5044\n\n### new in 0.0.3\n\n1. 使用radix tree代替ternary search tree，提升性能。\n2. 加入LOG信息，展示抽取的进度。\n\n\n### new in 0.0.2\n\n1. 直接导入[java-merge-sort](https://github.com/cowtowncoder/java-merge-sort)源码, thx[@cowtowncoder](https://github.com/cowtowncoder)\n2. 将之前的maven项目，转变为一个gradle项目，方便打包使用。\n\n### 成词条件\n\n1. 互信息\n2. 左右熵\n3. 位置成词概率\n4. ngram 频率\n\n### 运行方法\n\n1. [下载](https://github.com/sing1ee/dict_build/blob/master/dict_build-0.0.3.tar?raw=true)或者gradle distTar打包程序\n2. 解压dict_build-x.x.x.tar\n3. 解压之后,进入bin. 运行:./dict_build 你的数据文件的绝对路径\n4. 结束之后,在数据文件同目录有文件:words_sort.data\n5. 四列分别为:词,词频,互信息,左右熵,位置成词概率.\n\n### 注意\n\n- 数据文件一定要是UTF8编码的\n- 如果数据文件较大, 出现out of memory问题，可以尝试如下方式，限mac和linux，其中2G可以根据实际情况调整\n\n```shell\nexport JAVA_OPTS=-Xmx2G\n./dict_build 你的数据文件的绝对路径\n```\n\n### 示例\n\n#### 《金瓶梅》抽取结果\n```shell\n西门庆  4754    6.727920454563199   2.0315193024276885  0.17472535684926388\n月娘    1829    6.491853096329675   2.3714166640957095  0.22135096835144072\n敬济    906 9.084808387804362   2.554594603718855   0.14485683987274656\n春梅    799 8.134426320220927   2.7880175589451714  0.16484505593416485\n玳安    796 8.228818690495881   2.865686193737731   0.11791820110723605\n后边    617 6.6293566200796095  4.008365154080131   0.2160373686259245\n玉楼    594 7.977279923499917   2.27346284978306    0.27518689925240297\n明日    580 6.189824558880018   2.705423396095033   0.1774535638537181\n两银子  458 6.129283016944967   2.351100547282295   0.3809078896437581\n小厮    454 7.257387842692652   3.945653525477103   0.16666666666666666\n打发    444 6.870364719583405   3.694604352707633   0.18409496065046307\n如今    410 6.643856189774725   2.1460777430093394  0.1780766096169519\n淫妇    382 7.768184324776926   3.277903508489837   0.2555205047318612\n桂姐    371 7.584962500721156   2.5922046565140424  0.36255305256284687\n老婆    331 6.266786540694902   3.5783015008688523  0.3758007117437722\n衣服    309 8.90388184573618    2.786139685416002   0.13284518828451883\n丫头    297 7.383704292474053   4.291010086795063   0.21875\n潘金莲  288 8.276124405274238   2.4955186567189194  0.35333669524289796\n昨日    285 6.857980995127572   2.6387249970833997  0.1774535638537181\n王婆    284 7.1799090900149345  2.3129267619188907  0.3758007117437722\n```\n\n#### 《西游记》抽取结果\n```shell\n八戒    1807    7.88874324889826    2.00952580557629    0.36441586280814575\n师父    1632    7.507794640198696   3.745294449785798   0.1371395690812608\n大圣    1270    6.599912842187128   2.7790919785432147  0.13128460061010055\n唐僧    1003    7.076815597050832   4.350465172292435   0.43277723258096173\n菩萨    765 9.471675214392045   3.6013747138664756  0.15910495734948696\n妖精    634 7.199672344836364   3.1817261900583627  0.13134411600669268\n徒弟    439 8.060695931687555   2.498555429145656   0.15553809897879026\n兄弟    284 7.845490050944376   2.93037668783551    0.16085578446909668\n宝贝    283 9.319672120946995   2.616164396748633   0.15108220492589827\n今日    282 6.714245517666122   2.1303069812971214  0.1774535638537181\n取经    263 7.539158811108032   2.663944888382171   0.10181178023912565\n如今    259 6.189824558880018   2.056188859866133   0.1780766096169519\n认得    223 6.357552004618085   2.9543379335926954  0.2326782564877803\n东土    212 8.422064766172811   3.326253983395916   0.14745277618775043\n孙大圣  202 6.022367813028454   2.4886576514017107  0.13128460061010055\n变作    189 7.554588851677638   3.0713596792578635  0.23452975920036348\n玉帝    189 8.912889336229961   2.973106046717708   0.27518689925240297\n土地    179 7.499845887083206   3.1206506190132566  0.2819944064037033\n欢喜    173 8.861086905995393   2.184918471204895   0.31727272727272726\n贫僧    170 7.400879436282184   2.0731236036504477  0.43277723258096173\n```\n\n#### 拉勾JD语料抽取结果\n```shell\n工作\t641962\t11.645208082774683\t4.083574124851783\t0.11247281022865935\n开发\t348538\t14.031184262140844\t4.37645153459778\t0.18409496065046307\n相关\t300517\t10.477758266443889\t5.038915743418073\t0.1758213331033888\n合作\t159688\t10.397674632948268\t3.9963476653135794\t0.19498851077798446\n专业\t158831\t10.712527000439824\t3.152041650598071\t0.2640750670241287\n测试\t158179\t13.65362883340751\t4.464104436545589\t0.18344308560677328\n互联网\t148818\t16.106992250086762\t3.9556191209604314\t0.407386403912951\n活动\t131099\t10.391243589427443\t3.9155422678129406\t0.20137250696976194\n维护\t120316\t12.681677655209691\t3.2400117935377266\t0.1960306406685237\n问题\t112116\t9.159871336778389\t2.314215135279833\t0.20283174185051037\n优化\t109563\t11.324180546618742\t4.331660381832997\t0.2456782591010779\n营销\t105845\t14.36850646150769\t5.097001962525406\t0.14961371773129828\n平台\t100783\t9.002815015607053\t4.443804901153697\t0.2877423571272965\n培训\t93204\t9.041659151637216\t3.8898570467819824\t0.13345998575160295\n资源\t90339\t8.651051691178928\t4.063430372719874\t0.14695817490494298\n相关专业\t87545\t8.988684686772165\t2.4897196388075598\t0.2905199904149232\n网站\t87182\t8.92184093707449\t5.465843476701055\t0.21266038137095059\n独立\t86111\t9.074141462752506\t3.1456261690072957\t0.19050261614079594\n一定\t83798\t8.335390354693924\t2.107303660112154\t0.26157299167679793\n流程\t83165\t9.321928094887362\t2.5509378861028074\t0.2063141084699957\n网络\t82742\t9.087462841250339\t4.681429111504988\t0.21266038137095059\n优秀\t74600\t9.370687406807217\t2.0756995478573135\t0.2899855507391353\n信息\t71009\t9.820178962415188\t4.2602697278449755\t0.18863532864443658\n媒体\t67533\t10.556506054671928\t4.615376861300178\t0.17976710334788937\n编写\t64337\t7.960001932068081\t3.482400585501417\t0.265625\n思维\t62351\t8.741466986401146\t2.4320664807326646\t0.15396736072031514\n规划\t59733\t7.851749041416057\t2.936854928368285\t0.14166201896263245\n移动\t59671\t10.10459875356437\t3.4421932833155653\t0.20137250696976194\n渠道\t59072\t9.513727595952437\t4.597891463808354\t0.23578595317725753\n关系\t58483\t8.348728154231077\t2.4369558675502927\t0.3170022612253688\n积极\t57295\t9.044394119358454\t2.763249521041074\t0.1746848469256496\n实施\t56645\t7.781359713524661\t4.371966846513886\t0.15944453739334113\n福利\t55732\t8.475733430966399\t2.4036919305145426\t0.20908952728378172\n其他\t55665\t8.434628227636725\t2.9614863103296867\t0.15943975441289332\n功能\t55087\t7.787902559391432\t4.1663586610392755\t0.18097560975609756\n代码\t52431\t7.88874324889826\t3.876917512626917\t0.2135697048449972\n微信\t49143\t8.945443836377912\t3.6868130380800643\t0.18215857916308253\n企业\t48799\t9.422064766172813\t5.568662443510237\t0.2905199904149232\n提升\t48446\t8.233619676759702\t3.7390647282620666\t0.29750778816199375\n质量\t47918\t10.861862340059153\t3.391825261582227\t0.10921827734437191\n人员\t47109\t7.774787059601174\t5.249783964892326\t0.13589632038101343\n数据库\t45445\t8.290018846932618\t4.123423571610193\t0.2640569395017794\n商务\t44047\t8.189824558880018\t3.44858516585648\t0.12901085044961344\n主动\t42628\t13.815583433851023\t2.5049637884195137\t0.1968791796700847\n创意\t41768\t14.396470993910388\t4.115068825929573\t0.30544056771141337\n工具\t40227\t9.927777962082342\t2.208874047820781\t0.11247281022865935\n等相关\t39230\t11.919608238603255\t3.0330398736413557\t0.1758213331033888\n提出\t38741\t10.179909090014934\t4.46446156782086\t0.13053040103492886\n各类\t38309\t8.344295907915816\t5.136417986953123\t0.3969948596283116\n操作\t37061\t9.06339508128851\t4.676836974292029\t0.23452975920036348\n收集\t36600\t8.800899899920305\t2.797691452951563\t0.11388512456999896\n过程\t36534\t8.214319120800766\t2.5633950372758565\t0.2063141084699957\n数据分析\t36081\t8.442943495848729\t3.5589033442862585\t0.2640569395017794\n```\n\n#### 全宋词抽取结果\n```shell\n何处\t388\t6.491853096329675\t3.3628674437455617\t0.6815015936725298\n东风\t286\t5.392317422778761\t4.458774408044057\t0.19724622030237582\n江南\t250\t6.409390936137703\t3.903802705407174\t0.10545138034778331\n春风\t237\t3.5849625007211565\t4.927775131630969\t0.16484505593416485\n相思\t225\t6.614709844115209\t4.358855443007008\t0.242072962836686\n千里\t218\t6.409390936137703\t4.4108660037595\t0.2562873368242496\n人间\t200\t5.357552004618084\t3.6298146463975085\t0.13589632038101343\n明月\t196\t5.357552004618084\t4.461698115330817\t0.2009720696427977\n归来\t195\t5.08746284125034\t4.510975805812117\t0.4260707923476106\n尊前\t190\t7.607330313749611\t3.7677180601390012\t0.1516088400320623\n相逢\t179\t7.426264754702098\t3.729594240735622\t0.2827298050139276\n芳草\t176\t7.409390936137703\t4.193709696939418\t0.10797973400886637\n多情\t175\t6.247927513443586\t3.8156445316213303\t0.3327408912022344\n阑干\t167\t9.30149619498255\t4.1027945328835855\t0.17564639607106747\n梅花\t159\t4.807354922057604\t4.829461592976214\t0.1725721995566835\n年年\t157\t3.8073549220576037\t3.401504022650184\t0.10157033077180087\n无人\t150\t2.807354922057604\t4.773999920722275\t0.35809310100061825\n如今\t148\t5.7279204545632\t2.4554158038937834\t0.1780766096169519\n回首\t145\t7.94251450533924\t3.197825274741958\t0.20080445544554457\n天涯\t142\t7.74819284958946\t4.087307754334477\t0.4339155749636099\n一枝\t135\t5.20945336562895\t3.5111675192832683\t0.2674922938432581\n当时\t134\t6.08746284125034\t3.2683525636568564\t0.14850198715988994\n流水\t132\t5.700439718141093\t4.024081009656002\t0.13549047394111163\n佳人\t131\t5.20945336562895\t3.0918026501936384\t0.22896958600345846\n西风\t128\t4.321928094887363\t4.310178372466687\t0.19724622030237582\n依旧\t125\t7.768184324776926\t3.8821144630683277\t0.1728525980911983\n故人\t122\t5.392317422778761\t2.9526098687901237\t0.2363130219610269\n今夜\t121\t5.554588851677638\t3.239568407653533\t0.2543231961836613\n少年\t120\t5.357552004618084\t2.8645866477158934\t0.23419345103365022\n春色\t120\t5.129283016944966\t4.576389958371988\t0.16484505593416485\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsing1ee%2Fdict_build","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsing1ee%2Fdict_build","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsing1ee%2Fdict_build/lists"}