{"id":16837851,"url":"https://github.com/hankcs/hanlp-lucene-plugin","last_synced_at":"2025-04-05T15:07:20.279Z","repository":{"id":57724808,"uuid":"41210523","full_name":"hankcs/hanlp-lucene-plugin","owner":"hankcs","description":"HanLP中文分词Lucene插件，支持包括Solr在内的基于Lucene的系统","archived":false,"fork":false,"pushed_at":"2020-10-13T09:04:23.000Z","size":75,"stargazers_count":297,"open_issues_count":20,"forks_count":97,"subscribers_count":26,"default_branch":"master","last_synced_at":"2025-03-29T14:08:48.603Z","etag":null,"topics":["chinese-text-segmentation","hanlp","lucene","nlp","solr","traditional-chinese"],"latest_commit_sha":null,"homepage":"http://www.hankcs.com/nlp/segment/full-text-retrieval-solr-integrated-hanlp-chinese-word-segmentation.html","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hankcs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-08-22T14:23:27.000Z","updated_at":"2025-03-27T06:53:25.000Z","dependencies_parsed_at":"2022-09-11T02:01:47.860Z","dependency_job_id":null,"html_url":"https://github.com/hankcs/hanlp-lucene-plugin","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hankcs%2Fhanlp-lucene-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hankcs%2Fhanlp-lucene-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hankcs%2Fhanlp-lucene-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hankcs%2Fhanlp-lucene-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hankcs","download_url":"https://codeload.github.com/hankcs/hanlp-lucene-plugin/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247353745,"owners_count":20925329,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chinese-text-segmentation","hanlp","lucene","nlp","solr","traditional-chinese"],"created_at":"2024-10-13T12:19:04.965Z","updated_at":"2025-04-05T15:07:20.225Z","avatar_url":"https://github.com/hankcs.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"hanlp-lucene-plugin\n========\n\nHanLP中文分词Lucene插件\n----------------------\n基于HanLP，支持包括Solr（7.x）在内的任何基于Lucene（7.x）的系统。\n\n## Maven\n\n```xml\n    \u003cdependency\u003e\n      \u003cgroupId\u003ecom.hankcs.nlp\u003c/groupId\u003e\n      \u003cartifactId\u003ehanlp-lucene-plugin\u003c/artifactId\u003e\n      \u003cversion\u003e1.1.7\u003c/version\u003e\n    \u003c/dependency\u003e\n```\n\n## Solr快速上手\n 1. 将[hanlp-portable.jar](https://search.maven.org/search?q=g:com.hankcs%20AND%20a:hanlp)和[hanlp-lucene-plugin.jar](https://github.com/hankcs/hanlp-lucene-plugin/releases)共两个jar放入```${webapp}/WEB-INF/lib```下。（或者使用```mvn package```对源码打包，拷贝```target/hanlp-lucene-plugin-x.x.x.jar```到```${webapp}/WEB-INF/lib```下）\n 1. 修改solr core的配置文件```${core}/conf/schema.xml```：\n\n```xml\n  \u003cfieldType name=\"text_cn\" class=\"solr.TextField\"\u003e\n      \u003canalyzer type=\"index\"\u003e\n          \u003ctokenizer class=\"com.hankcs.lucene.HanLPTokenizerFactory\" enableIndexMode=\"true\"/\u003e\n      \u003c/analyzer\u003e\n      \u003canalyzer type=\"query\"\u003e\n          \u003c!-- 切记不要在query中开启index模式 --\u003e\n          \u003ctokenizer class=\"com.hankcs.lucene.HanLPTokenizerFactory\" enableIndexMode=\"false\"/\u003e\n      \u003c/analyzer\u003e\n  \u003c/fieldType\u003e\n  \u003c!-- 业务系统中需要分词的字段都需要指定type为text_cn --\u003e\n  \u003cfield name=\"my_field1\" type=\"text_cn\" indexed=\"true\" stored=\"true\"/\u003e\n  \u003cfield name=\"my_field2\" type=\"text_cn\" indexed=\"true\" stored=\"true\"/\u003e\n```\n\n * 如果你的业务系统中有其他字段，比如location，summary之类，也需要一一指定其type=\"text_cn\"。切记，否则这些字段仍旧是solr默认分词器。\n * 另外，切记不要在query中开启indexMode，否则会影响PhaseQuery。indexMode只需在index中开启一遍即可。\n\n## 高级配置\n 目前本插件支持如下基于```schema.xml```的配置:\n\n| 配置项名称       | 功能   |  默认值  |\n| --------   | -----:  | :----:  |\n| algorithm   | [分词算法](https://github.com/hankcs/HanLP/blob/master/src/main/java/com/hankcs/hanlp/HanLP.java#L643) |   viterbi     |\n| enableIndexMode    | 设为索引模式（切勿在query中开启） |   true     |\n| enableCustomDictionary    | 是否启用用户词典 |   true     |\n| customDictionaryPath    | 用户词典路径(绝对路径或程序可以读取的相对路径,多个词典用空格隔开) |   null     |\n| enableCustomDictionaryForcing    | [用户词典高优先级](https://github.com/hankcs/HanLP/wiki/FAQ#%E4%B8%BA%E4%BB%80%E4%B9%88%E4%BF%AE%E6%94%B9%E4%BA%86%E8%AF%8D%E5%85%B8%E8%BF%98%E6%98%AF%E6%B2%A1%E6%9C%89%E6%95%88%E6%9E%9C) |   false     |\n| stopWordDictionaryPath    | 停用词词典路径 |   null     |\n| enableNumberQuantifierRecognize    | 是否启用数词和数量词识别 |   true     |\n| enableNameRecognize    | 开启人名识别 |   true     |\n| enableTranslatedNameRecognize    | 是否启用音译人名识别 |   false     |\n| enableJapaneseNameRecognize    | 是否启用日本人名识别 |   false     |\n| enableOrganizationRecognize    | 开启机构名识别 |   false     |\n| enablePlaceRecognize    | 开启地名识别 |   false     |\n| enableNormalization    | 是否执行字符正规化（繁体-\u003e简体，全角-\u003e半角，大写-\u003e小写） |   false     |\n| enableTraditionalChineseMode    | 开启精准繁体中文分词 |   false     |\n| enableDebug    | 开启调试模式 |   false     |\n\n 更高级的配置主要通过class path下的```hanlp.properties```进行配置，请阅读[HanLP自然语言处理包文档](https://github.com/hankcs/HanLP)以了解更多相关配置，如：\n\n0. 用户词典\n0. 词性标注\n0. 简繁转换\n0. ……\n\n## 停用词与同义词\n 推荐利用Lucene或Solr自带的filter实现，本插件不会越俎代庖。\n 一个示例配置如下：\n\n```xml\n    \u003c!-- text_cn字段类型: 指定使用HanLP分词器，同时开启索引模式。通过solr自带的停用词过滤器，使用\"stopwords.txt\"（默认空白）过滤。\n\t 在搜索的时候，还支持solr自带的同义词词典。--\u003e\n    \u003cfieldType name=\"text_cn\" class=\"solr.TextField\" positionIncrementGap=\"100\"\u003e\n      \u003canalyzer type=\"index\"\u003e\n        \u003ctokenizer class=\"com.hankcs.lucene.HanLPTokenizerFactory\" enableIndexMode=\"true\"/\u003e\n        \u003cfilter class=\"solr.StopFilterFactory\" ignoreCase=\"true\" words=\"stopwords.txt\" /\u003e\n        \u003c!-- 取消注释可以启用索引期间的同义词词典\n        \u003cfilter class=\"solr.SynonymFilterFactory\" synonyms=\"index_synonyms.txt\" ignoreCase=\"true\" expand=\"false\"/\u003e\n        --\u003e\n        \u003cfilter class=\"solr.LowerCaseFilterFactory\"/\u003e\n      \u003c/analyzer\u003e\n      \u003canalyzer type=\"query\"\u003e\n        \u003ctokenizer class=\"com.hankcs.lucene.HanLPTokenizerFactory\" enableIndexMode=\"false\"/\u003e\n        \u003cfilter class=\"solr.StopFilterFactory\" ignoreCase=\"true\" words=\"stopwords.txt\" /\u003e\n        \u003cfilter class=\"solr.SynonymFilterFactory\" synonyms=\"synonyms.txt\" ignoreCase=\"true\" expand=\"true\"/\u003e\n        \u003cfilter class=\"solr.LowerCaseFilterFactory\"/\u003e\n      \u003c/analyzer\u003e\n    \u003c/fieldType\u003e\n    \u003c!-- 业务系统中需要分词的字段都需要指定type为text_cn --\u003e\n    \u003cfield name=\"my_field1\" type=\"text_cn\" indexed=\"true\" stored=\"true\"/\u003e\n    \u003cfield name=\"my_field2\" type=\"text_cn\" indexed=\"true\" stored=\"true\"/\u003e\n```\n\n## 调用方法\n在Query改写的时候，可以利用HanLPAnalyzer分词结果中的词性等属性，如\n```java\nString text = \"中华人民共和国很辽阔\";\nfor (int i = 0; i \u003c text.length(); ++i)\n{\n    System.out.print(text.charAt(i) + \"\" + i + \" \");\n}\nSystem.out.println();\nAnalyzer analyzer = new HanLPAnalyzer();\nTokenStream tokenStream = analyzer.tokenStream(\"field\", text);\ntokenStream.reset();\nwhile (tokenStream.incrementToken())\n{\n    CharTermAttribute attribute = tokenStream.getAttribute(CharTermAttribute.class);\n    // 偏移量\n    OffsetAttribute offsetAtt = tokenStream.getAttribute(OffsetAttribute.class);\n    // 距离\n    PositionIncrementAttribute positionAttr = tokenStream.getAttribute(PositionIncrementAttribute.class);\n    // 词性\n    TypeAttribute typeAttr = tokenStream.getAttribute(TypeAttribute.class);\n    System.out.printf(\"[%d:%d %d] %s/%s\\n\", offsetAtt.startOffset(), offsetAtt.endOffset(), positionAttr.getPositionIncrement(), attribute, typeAttr.type());\n}\n```\n在另一些场景，支持以自定义的分词器（比如开启了命名实体识别的分词器、繁体中文分词器、CRF分词器等）构造HanLPTokenizer，比如：\n```java\ntokenizer = new HanLPTokenizer(HanLP.newSegment()\n                                    .enableJapaneseNameRecognize(true)\n                                    .enableIndexMode(true), null, false);\ntokenizer.setReader(new StringReader(\"林志玲亮相网友:确定不是波多野结衣？\"));\n```\n\n## 版权\n Apache License Version 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhankcs%2Fhanlp-lucene-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhankcs%2Fhanlp-lucene-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhankcs%2Fhanlp-lucene-plugin/lists"}