{"id":15027205,"url":"https://github.com/shibing624/similarity","last_synced_at":"2025-05-14T13:06:18.648Z","repository":{"id":40605204,"uuid":"73244481","full_name":"shibing624/similarity","owner":"shibing624","description":"similarity: Text similarity calculation Toolkit for Java. 文本相似度计算工具包，java编写，可用于文本相似度计算、情感分析等任务，开箱即用。","archived":false,"fork":false,"pushed_at":"2025-01-04T10:04:47.000Z","size":79826,"stargazers_count":1498,"open_issues_count":11,"forks_count":336,"subscribers_count":39,"default_branch":"master","last_synced_at":"2025-04-13T16:50:20.472Z","etag":null,"topics":["java","nlp","semantic","sentiment","sim-scores","similarity"],"latest_commit_sha":null,"homepage":"https://shibing624.github.io/similarity/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shibing624.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-11-09T02:02:01.000Z","updated_at":"2025-04-12T17:12:27.000Z","dependencies_parsed_at":"2025-01-04T19:00:25.588Z","dependency_job_id":"f13df7ae-0aab-4e27-8662-0553cb7af71a","html_url":"https://github.com/shibing624/similarity","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibing624%2Fsimilarity","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibing624%2Fsimilarity/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibing624%2Fsimilarity/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibing624%2Fsimilarity/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shibing624","download_url":"https://codeload.github.com/shibing624/similarity/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254149953,"owners_count":22022851,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["java","nlp","semantic","sentiment","sim-scores","similarity"],"created_at":"2024-09-24T20:05:57.508Z","updated_at":"2025-05-14T13:06:18.608Z","avatar_url":"https://github.com/shibing624.png","language":"Java","funding_links":[],"categories":["人工智能"],"sub_categories":[],"readme":"[![jitpack](https://jitpack.io/v/shibing624/similarity.svg)](https://jitpack.io/#shibing624/similarity)\n[![Stars](https://img.shields.io/github/stars/shibing624/similarity?style=social)](https://github.com/shibing624/similarity)\n[![License Apache 2.0](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)\n[![GitHub issues](https://img.shields.io/github/issues/shibing624/similarity.svg)](https://github.com/shibing624/similarity/issues)\n[![Wechat Group](https://img.shields.io/badge/wechat-group-green.svg?logo=wechat)](#Contact)\n\n# Similarity\nsimilarity, compute similarity score between text strings, Java written.\n\nsimilarity，相似度计算工具包，可用于文本相似度计算、情感倾向分析等，Java编写。\n\n**similarity**是由一系列算法组成的Java版相似度计算工具包，目标是传播自然语言处理中相似度计算方法。**similarity**具备工具实用、性能高效、架构清晰、语料时新、可自定义的特点。\n\n# Feature\n\n**similarity**提供下列功能：\n\n- 词语相似度计算\n  * **词林编码法相似度[推荐]**\n  * 汉语语义法相似度\n  * 知网词语相似度\n  * 字面编辑距离法\n  \n- 短语相似度计算\n  * **简单短语相似度[推荐]**\n    \n- 句子相似度计算\n  * **词性和词序结合法[推荐]**\n  * 编辑距离算法\n  * Gregor编辑距离法\n  * 优化编辑距离法\n  \n- 段落相似度计算\n  * **余弦相似度[推荐]**\n  * 编辑距离\n  * 欧几里得距离\n  * Jaccard相似性系数\n  * Jaro距离\n  * Jaro–Winkler距离\n  * 曼哈顿距离\n  * SimHash + 汉明距离\n  * Sørensen–Dice系数\n\n- 知网义原\n  * 词语义原树\n\n- 情感分析\n  * 正面倾向程度\n  * 负面倾向程度\n  * 情感倾向性\n  \n- 近似词\n  * word2vec\n\n\n\n在提供丰富功能的同时，**similarity**内部模块坚持低耦合、模型坚持惰性加载、词典坚持明文发布，使用方便，帮助用户训练自己的语料。\n\n# Usage\n引入Jar包\n\n### Maven\n\n```xml\n\u003crepositories\u003e\n  \u003crepository\u003e\n    \u003cid\u003ejitpack.io\u003c/id\u003e\n    \u003curl\u003ehttps://jitpack.io\u003c/url\u003e\n  \u003c/repository\u003e\n\u003c/repositories\u003e\n```\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.github.shibing624\u003c/groupId\u003e\n  \u003cartifactId\u003esimilarity\u003c/artifactId\u003e\n  \u003cversion\u003e1.1.6\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n### Gradle\ngradle的引入：[![jitpack](https://jitpack.io/v/shibing624/similarity.svg)](https://jitpack.io/#shibing624/similarity)\n\n\n### 使用示例\n```java\nimport org.xm.Similarity;\nimport org.xm.tendency.word.HownetWordTendency;\n\npublic class demo {\n    public static void main(String[] args) {\n        double result = Similarity.cilinSimilarity(\"电动车\", \"自行车\");\n        System.out.println(result);\n\n        String word = \"混蛋\";\n        HownetWordTendency hownetWordTendency = new HownetWordTendency();\n        result = hownetWordTendency.getTendency(word);\n        System.out.println(word + \"  词语情感趋势值：\" + result);\n    }\n}\n```\n\n## 功能演示\n\n### 1. 词语相似度计算\n\n文本长度：词语粒度\n\n**推荐**使用词林相似度：`org.xm.Similarity.cilinSimilarity`，是基于[同义词词林](https://blog.csdn.net/sinat_33741547/article/details/80016713)的相似度计算方法\n\nexample: [src/test/java/org.xm/WordSimilarityDemo.java](src/test/java/org/xm/WordSimilarityDemo.java)\n```java\npackage org.xm;\n\npublic class WordSimilarityDemo {\n\n    public static void main(String[] args) {\n        String word1 = \"教师\";\n        String word2 = \"教授\";\n        double cilinSimilarityResult = Similarity.cilinSimilarity(word1, word2);\n        double pinyinSimilarityResult = Similarity.pinyinSimilarity(word1, word2);\n        double conceptSimilarityResult = Similarity.conceptSimilarity(word1, word2);\n        double charBasedSimilarityResult = Similarity.charBasedSimilarity(word1, word2);\n\n        System.out.println(word1 + \" vs \" + word2 + \" 词林相似度值：\" + cilinSimilarityResult);\n        System.out.println(word1 + \" vs \" + word2 + \" 拼音相似度值：\" + pinyinSimilarityResult);\n        System.out.println(word1 + \" vs \" + word2 + \" 概念相似度值：\" + conceptSimilarityResult);\n        System.out.println(word1 + \" vs \" + word2 + \" 字面相似度值：\" + charBasedSimilarityResult);\n    }\n}\n```\n\n* result:\n\n![word_sim result](./docs/pic/word_sim.png)\n\n### 2. 短语相似度计算\n\n文本长度：短语粒度\n\n**推荐**使用短语相似度：`org.xm.Similarity.phraseSimilarity`，本质是通过两个短语具有的相同字符，和相同字符的位置计算其相似度的方法\n\nexample: [src/test/java/org.xm/PhraseSimilarityDemo.java](src/test/java/org/xm/PhraseSimilarityDemo.java)\n```java\npublic static void main(String[] args) {\n    String phrase1 = \"继续努力\";\n    String phrase2 = \"持续发展\";\n    double result = Similarity.phraseSimilarity(phrase1, phrase2);\n\n    System.out.println(phrase1 + \" vs \" + phrase2 + \" 短语相似度值：\" + result);\n}\n```\n\n\n* result:\n\n![phrase sim result](./docs/pic/phrase_sim.png)\n\n### 3. 句子相似度计算\n文本长度：句子粒度\n\n**推荐**使用词形词序句子相似度：`org.xm.similarity.morphoSimilarity`，一种既考虑两个句子相同文本字面，也考虑相同文本出现的前后顺序的相似度方法\n\nexample: [src/test/java/org.xm/SentenceSimilarityDemo.java](src/test/java/org/xm/SentenceSimilarityDemo.java)\n\n```java\npublic static void main(String[] args) {\n    String sentence1 = \"中国人爱吃鱼\";\n    String sentence2 = \"湖北佬最喜吃鱼\";\n\n    double morphoSimilarityResult = Similarity.morphoSimilarity(sentence1, sentence2);\n    double editDistanceResult = Similarity.editDistanceSimilarity(sentence1, sentence2);\n    double standEditDistanceResult = Similarity.standardEditDistanceSimilarity(sentence1,sentence2);\n    double gregeorEditDistanceResult = Similarity.gregorEditDistanceSimilarity(sentence1,sentence2);\n\n    System.out.println(sentence1 + \" vs \" + sentence2 + \" 词形词序句子相似度值：\" + morphoSimilarityResult);\n    System.out.println(sentence1 + \" vs \" + sentence2 + \" 优化的编辑距离句子相似度值：\" + editDistanceResult);\n    System.out.println(sentence1 + \" vs \" + sentence2 + \" 标准编辑距离句子相似度值：\" + standEditDistanceResult);\n    System.out.println(sentence1 + \" vs \" + sentence2 + \" gregeor编辑距离句子相似度值：\" + gregeorEditDistanceResult);\n}\n```\n\n* result:\n\n![sentence sim result](./docs/pic/sentence_sim.png)\n\n\n\n### 4. 段落文本相似度计算\n文本长度：段落粒度（一段话，25字符 \u003c length(text) \u003c 500字符）\n\n**推荐**使用词形词序句子相似度：`org.xm.similarity.text.CosineSimilarity`，一种考虑两个段落中相同的文本，经过切词，词频和词性权重加权，并用余弦计算相似度的方法\n\nexample: [src/test/java/org.xm/similarity/text/CosineSimilarityTest.java](src/test/java/org/xm/similarity/text/CosineSimilarityTest.java)\n\n\n```java\n@Test\npublic void getSimilarityScore() throws Exception {\n        String text1 = \"对于俄罗斯来说，最大的战果莫过于夺取乌克兰首都基辅，也就是现任总统泽连斯基和他政府的所在地。目前夺取基辅的战斗已经打响。\";\n        String text2 = \"迄今为止，俄罗斯的入侵似乎没有完全按计划成功执行——英国国防部情报部门表示，在乌克兰军队激烈抵抗下，俄罗斯军队已经损失数以百计的士兵。尽管如此，俄军在继续推进。\";\n        TextSimilarity cosSimilarity = new CosineSimilarity();\n        double score1 = cosSimilarity.getSimilarity(text1, text2);\n        System.out.println(\"cos相似度分值：\" + score1);\n\n        TextSimilarity editSimilarity = new EditDistanceSimilarity();\n        double score2 = editSimilarity.getSimilarity(text1, text2);\n        System.out.println(\"edit相似度分值：\" + score2);\n        }\n```\n\n* result:\n```shell\ncos相似度分值：0.399143\nedit相似度分值：0.0875\n```\n\n### 5. 基于义原树的情感分析\nexample: [src/test/java/org/xm/tendency/word/HownetWordTendencyTest.java](src/test/java/org/xm/tendency/word/HownetWordTendencyTest.java)\n\n```java\n@Test\npublic void getTendency() throws Exception {\n    HownetWordTendency hownet = new HownetWordTendency();\n    String word = \"美好\";\n    double sim = hownet.getTendency(word);\n    System.out.println(word + \":\" + sim);\n    System.out.println(\"混蛋:\" + hownet.getTendency(\"混蛋\"));\n}\n```\n* result:\n\n![tendency result](./docs/pic/tendency.png)\n\n本例是基于义原树的词语粒度情感极性分析，关于文本情感分析有[pytextclassifier](https://github.com/shibing624/pytextclassifier)，利用深度神经网络模型、SVM分类算法实现的效果更好。\n\n### 6. 近义词推荐\nexample: [src/test/java/org/xm/word2vec/Word2vecTest.java](src/test/java/org/xm/word2vec/Word2vecTest.java)\n\n```java\n@Test\npublic void testHomoionym() throws Exception {\n    List\u003cString\u003e result = Word2vec.getHomoionym(RAW_CORPUS_SPLIT_MODEL, \"武功\", 10);\n    System.out.println(\"武功 近似词：\" + result);\n}\n\n@Test\npublic void testHomoionymName() throws Exception {\n    String model = RAW_CORPUS_SPLIT_MODEL;\n    List\u003cString\u003e result = Word2vec.getHomoionym(model, \"乔帮主\", 10);\n    System.out.println(\"乔帮主 近似词：\" + result);\n\n    List\u003cString\u003e result2 = Word2vec.getHomoionym(model, \"阿朱\", 10);\n    System.out.println(\"阿朱 近似词：\" + result2);\n\n    List\u003cString\u003e result3 = Word2vec.getHomoionym(model, \"少林寺\", 10);\n    System.out.println(\"少林寺 近似词：\" + result3);\n}\n```\n\n* 训练过程:\n\n![word2vec train](./docs/pic/word2v.png)\n\n* result:\n\n![word2vec result](./docs/pic/word2v_ret.png)\n\nWord2vec词向量训练用的java版word2vec训练工具[Word2VEC_java](https://github.com/NLPchina/Word2VEC_java)，训练语料是小说天龙八部，通过词向量实现得到近义词。\n用户可以训练自定义语料，也可以用中文维基百科训练通用词向量。\n\n## Todo\n\n文本相似性度量\n\n- [x] 关键词匹配（TF-IDF、BM25）\n- [x] 浅层语义匹配（WordEmbed隐语义模型，用word2vec或glove词向量直接累加构造的句向量）\n- [x] 深度语义匹配模型（DSSM、CLSM、DeepMatch、MatchingFeatures、ARC-II、DeepMind见MatchZoo），BERT类语义匹配模型SentenceBERT、CoSENT见[text2vec](https://github.com/shibing624/text2vec)\n\n# Contact\n\n- Issue(建议)：[![GitHub issues](https://img.shields.io/github/issues/shibing624/similarity.svg)](https://github.com/shibing624/similarity/issues)\n- 邮件我：xuming: xuming624@qq.com\n- 微信我：\n  加我*微信号：xuming624, 备注：姓名-公司-NLP* 进NLP交流群。\n\n\u003cimg src=\"docs/wechat.jpeg\" width=\"200\" /\u003e\n\n\n# License\n\n\n授权协议为 [The Apache License 2.0](/LICENSE)，可免费用做商业用途。请在产品说明中附加*similarity*的链接和授权协议。\n\n\n# Contribute\n项目代码还很粗糙，如果大家对代码有所改进，欢迎提交回本项目，在提交之前，注意以下两点：\n\n- 在`test`添加相应的单元测试\n- 运行所有单元测试，确保所有单测都是通过的\n\n之后即可提交PR。\n\n# Reference\n\n* [DSSM] Po-Sen Huang, et al., 2013, Learning Deep Structured Semantic Models for Web Search using Clickthrough Data\n* [CLSM] Yelong Shen, et al, 2014, A Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval \n* [DeepMatch] Zhengdong Lu \u0026 Hang Li, 2013, A Deep Architecture for Matching Short Texts\n* [MatchingFeatures] Zongcheng Ji, et al., 2014, An Information Retrieval Approach to Short Text Conversation\n* [ARC-II] Baotian Hu, et al., 2015, Convolutional Neural Network Architectures for Matching Natural Language Sentences\n* [DeepMind] Aliaksei Severyn, et al., 2015, Learning to Rank Short Text Pairs with Convolutional Deep Neural Networks\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshibing624%2Fsimilarity","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshibing624%2Fsimilarity","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshibing624%2Fsimilarity/lists"}