{"id":28558589,"url":"https://github.com/yongzhuo/macropodus","last_synced_at":"2025-06-10T08:09:31.487Z","repository":{"id":39739247,"uuid":"231115996","full_name":"yongzhuo/Macropodus","owner":"yongzhuo","description":"自然语言处理工具Macropodus，基于Albert+BiLSTM+CRF深度学习网络架构，中文分词，词性标注，命名实体识别，新词发现，关键词，文本摘要，文本相似度，科学计算器，中文数字阿拉伯数字(罗马数字)转换，中文繁简转换，拼音转换。tookit(tool) of NLP，CWS(chinese word segnment)，POS(Part-Of-Speech Tagging)，NER(name entity recognition)，Find(new words discovery)，Keyword(keyword extraction)，Summarize(text summarization)，Sim(text similarity)，Calculate(scientific calculator)，Chi2num(chinese number to arabic number)","archived":false,"fork":false,"pushed_at":"2023-03-24T22:22:42.000Z","size":9507,"stargazers_count":662,"open_issues_count":5,"forks_count":92,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-06-10T08:09:26.332Z","etag":null,"topics":["albert","calulator","crf","cws","keyword","macropodus","ner","newword","nlp","segnment","similarity","text-summarization"],"latest_commit_sha":null,"homepage":"https://blog.csdn.net/rensihui","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yongzhuo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-12-31T16:14:40.000Z","updated_at":"2025-06-10T03:31:52.000Z","dependencies_parsed_at":"2023-01-22T09:45:37.302Z","dependency_job_id":"3488e6c8-472f-4c9e-92de-31cc6428be38","html_url":"https://github.com/yongzhuo/Macropodus","commit_stats":{"total_commits":17,"total_committers":3,"mean_commits":5.666666666666667,"dds":0.3529411764705882,"last_synced_commit":"1d7b8f9938cb8b6d7744e9caabc3eb41c8891283"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yongzhuo%2FMacropodus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yongzhuo%2FMacropodus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yongzhuo%2FMacropodus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yongzhuo%2FMacropodus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yongzhuo","download_url":"https://codeload.github.com/yongzhuo/Macropodus/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yongzhuo%2FMacropodus/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259033864,"owners_count":22795774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["albert","calulator","crf","cws","keyword","macropodus","ner","newword","nlp","segnment","similarity","text-summarization"],"created_at":"2025-06-10T08:09:20.649Z","updated_at":"2025-06-10T08:09:31.456Z","avatar_url":"https://github.com/yongzhuo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n    \u003cimg src=\"test/images/macropodus_logo.png\" width=\"480\"\\\u003e\n\u003c/p\u003e\n\n# [Macropodus](https://github.com/yongzhuo/Macropodus)\n![PyPI - Python Version](https://img.shields.io/badge/python-3.6-blue.svg) \n[![PyPI](https://img.shields.io/pypi/v/Macropodus)](https://pypi.org/project/Macropodus/)\n[![GitHub](https://img.shields.io/github/license/mashape/apistatus.svg)](https://img.shields.io/github/license/mashape/apistatus.svg) \n[![PyPI_downloads](https://img.shields.io/pypi/dm/Macropodus)](https://pypi.org/project/Macropodus/)\n[![Stars](https://img.shields.io/github/stars/yongzhuo/Macropodus?style=social)](https://github.com/yongzhuo/Macropodus/stargazers)\n[![Forks](https://img.shields.io/github/forks/yongzhuo/Macropodus.svg?style=social)](https://github.com/yongzhuo/Macropodus/network/members)\n[![Join the chat at https://gitter.im/yongzhuo/Macropodus](https://badges.gitter.im/yongzhuo/Macropodus.svg)](https://gitter.im/yongzhuo/Macropodus?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\u003e\u003e\u003e Macropodus是一个以Albert+BiLSTM+CRF网络架构为基础，用大规模中文语料训练的自然语言处理工具包。将提供中文分词、词性标注、命名实体识别、关键词抽取、文本摘要、新词发现、文本相似度、计算器、数字转换、拼音转换、繁简转换等常见NLP功能。\n\n\n## 目录\n\n* [安装](#安装)\n* [使用方式](#使用方式)\n* [参考/引用](#参考/引用)\n* [FAQ](#FAQ)\n\n\n# 安装 \n0. 注意事项\n    ```\n\t默认不安装nlg-yongzhuo, 如果需要该功能自行安装;\n\t默认不指定numpy, pandas, scikit-learn版本, 过高或者过低的版本可能不支持\n\t标准版本的依赖包详见 requirements-all.txt\n\t```\n1. 通过PyPI安装(模型文件另需下载, 详见[命名实体提取](#命名实体提取), [词性标注](#词性标注))：\n\t```\n\tpip install macropodus\n\t```\n2. 使用镜像源，例如：   \n\t```\n\tpip install -i https://pypi.tuna.tsinghua.edu.cn/simple macropodus\n\t```\n\n\n# 使用方式\n\n\n## 快速使用\n```python3\nimport macropodus\n\nsen_calculate = \"23 + 13 * (25+(-9-2-5-2*3-6/3-40*4/(2-3)/5+6*3))加根号144你算得几多\"\nsen_chi2num = \"三千零七十八亿三千零十五万零三百一十二点一九九四\"\nsen_num2chi = 1994.1994\nsen_roman2int = \"IX\"\nsen_int2roman = 132\nsent1 = \"PageRank算法简介\"\nsent2 = \"百度百科如是介绍他的思想:PageRank通过网络浩瀚的超链接关系来确定一个页面的等级。\"\nsummary = \"PageRank算法简介。\" \\\n           \"是上世纪90年代末提出的一种计算网页权重的算法! \" \\\n           \"当时，互联网技术突飞猛进，各种网页网站爆炸式增长。 \" \\\n           \"业界急需一种相对比较准确的网页重要性计算方法。 \" \\\n           \"是人们能够从海量互联网世界中找出自己需要的信息。 \" \\\n           \"百度百科如是介绍他的思想:PageRank通过网络浩瀚的超链接关系来确定一个页面的等级。 \" \\\n           \"Google把从A页面到B页面的链接解释为A页面给B页面投票。 \" \\\n           \"Google根据投票来源甚至来源的来源，即链接到A页面的页面。 \" \\\n           \"和投票目标的等级来决定新的等级。简单的说， \" \\\n           \"一个高等级的页面可以使其他低等级页面的等级提升。 \" \\\n           \"具体说来就是，PageRank有两个基本思想，也可以说是假设。 \" \\\n           \"即数量假设：一个网页被越多的其他页面链接，就越重）。 \" \\\n           \"质量假设：一个网页越是被高质量的网页链接，就越重要。 \" \\\n           \"总的来说就是一句话，从全局角度考虑，获取重要的信。 \"\n\n# 分词(词典最大概率分词DAG)\nwords = macropodus.cut(summary)\nprint(words)\n# 新词发现\nnew_words = macropodus.find(summary)\nprint(new_words)\n# 文本摘要\nsum = macropodus.summarize(summary)\nprint(sum)\n# 关键词抽取\nkeyword = macropodus.keyword(summary)\nprint(keyword)\n# 文本相似度\nsim = macropodus.sim(sent1, sent2)\nprint(sim)\n# tookit\n# 计算器\nscore_calcul = macropodus.calculate(sen_calculate)\nprint(score_calcul)\n# 中文数字与阿拉伯数字相互转化\nres_chi2num = macropodus.chi2num(sen_chi2num)\nprint(res_chi2num)\nres_num2chi = macropodus.num2chi(sen_num2chi)\nprint(res_num2chi)\n# 阿拉伯数字与罗马数字相互转化\nres_roman2int = macropodus.roman2int(sen_roman2int)\nprint(res_roman2int)\nres_int2roman = macropodus.int2roman(sen_int2roman)\nprint(res_int2roman)\n# 中文汉字转拼音\nres_pinyin = macropodus.pinyin(summary)\nprint(res_pinyin)\n# 中文繁简转化\nres_zh2han = macropodus.zh2han(summary)\nprint(res_zh2han)\nres_han2zh = macropodus.han2zh(res_zh2han)\nprint(res_han2zh)\n\n```\n\n\n## 中文分词\n\n各种分词方法\n```python3\nimport macropodus\n\n# 用户词典\nmacropodus.add_word(word=\"斗鱼科\")\nmacropodus.add_word(word=\"鲈形目\") # 不持久化, 当前有效\nmacropodus.save_add_words(word_freqs={\"喜斗\":32, \"护卵\":64, \"护幼\":132}) # 持久化保存到用户字典\nsent = \"斗鱼属，Macropodus (1801)，鲈形目斗鱼科的一属鱼类。本属鱼类通称斗鱼。因喜斗而得名。分布于亚洲东南部。中国有2种，即叉尾斗鱼，分布于长江及以南各省；叉尾斗鱼，分布于辽河到珠江流域。其喜栖居于小溪、河沟、池塘、稻田等缓流或静水中。雄鱼好斗，产卵期集草成巢，雄鱼口吐粘液泡沫，雌鱼产卵其中，卵浮性，受精卵在泡沫内孵化。雄鱼尚有护卵和护幼现象。\"\n\n# 分词\nsents = macropodus.cut_bidirectional(sent)\nprint(\"cut_bidirectional: \" + \" \".join(sents))\nsents = macropodus.cut_forward(sent)\nprint(\"cut_forward: \" + \" \".join(sents))\nsents = macropodus.cut_reverse(sent)\nprint(\"cut_reverse: \" + \" \".join(sents))\nsents = macropodus.cut_search(sent)\nprint(\"cut_search: \" + \" \".join(sents))\n# DAG\nsents = macropodus.cut_dag(sent)\nprint(\"cut_dag: \" + \" \".join(sents))\n\n```\n\n\n## 文本相似度\n\n  文本相似度主要使用词向量, 余弦相似度 或 jaccard相似度\n```python3\nimport macropodus\n\nsent1=\"叉尾斗鱼是一种观赏性动物\"\nsent2=\"中国斗鱼生性好斗,适应性强,能在恶劣的环境中生存\"\n           \n# 文本相似度(similarity)\nsents = macropodus.sim(sent1, sent2, type_sim=\"total\", type_encode=\"avg\")\nprint(sents)\nsents = macropodus.sim(sent1, sent2, type_sim=\"cosine\", type_encode=\"single\")\nprint(sents)\n\n```\n\n\n## 文本摘要\n\n  文本摘要方法有text_pronouns, text_teaser, word_sign, textrank, lead3, mmr, lda, lsi, nmf\n```python3\nimport macropodus\n\nsummary = \"PageRank算法简介。\" \\\n           \"是上世纪90年代末提出的一种计算网页权重的算法! \" \\\n           \"当时，互联网技术突飞猛进，各种网页网站爆炸式增长。 \" \\\n           \"业界急需一种相对比较准确的网页重要性计算方法。 \" \\\n           \"是人们能够从海量互联网世界中找出自己需要的信息。 \" \\\n           \"百度百科如是介绍他的思想:PageRank通过网络浩瀚的超链接关系来确定一个页面的等级。 \" \\\n           \"Google把从A页面到B页面的链接解释为A页面给B页面投票。 \" \\\n           \"Google根据投票来源甚至来源的来源，即链接到A页面的页面。 \" \\\n           \"和投票目标的等级来决定新的等级。简单的说， \" \\\n           \"一个高等级的页面可以使其他低等级页面的等级提升。 \" \\\n           \"具体说来就是，PageRank有两个基本思想，也可以说是假设。 \" \\\n           \"即数量假设：一个网页被越多的其他页面链接，就越重）。 \" \\\n           \"质量假设：一个网页越是被高质量的网页链接，就越重要。 \" \\\n           \"总的来说就是一句话，从全局角度考虑，获取重要的信。 \"\n           \n# 文本摘要(summarize, 默认接口)\nsents = macropodus.summarize(summary)\nprint(sents)\n\n# 文本摘要(summarization, 可定义方法, 提供9种文本摘要方法, 'lda', 'mmr', 'textrank', 'text_teaser')\nsents = macropodus.summarization(text=summary, type_summarize=\"lda\")\nprint(sents)\n\n```\n\n\n## 新词发现\n\n  新词发现主要使用凝固度, 左熵, 右熵, 词频等方案, 综合考虑\n```python3\nimport macropodus\n\nsummary = \"PageRank算法简介。\" \\\n           \"是上世纪90年代末提出的一种计算网页权重的算法! \" \\\n           \"当时，互联网技术突飞猛进，各种网页网站爆炸式增长。 \" \\\n           \"业界急需一种相对比较准确的网页重要性计算方法。 \" \\\n           \"是人们能够从海量互联网世界中找出自己需要的信息。 \" \\\n           \"百度百科如是介绍他的思想:PageRank通过网络浩瀚的超链接关系来确定一个页面的等级。 \" \\\n           \"Google把从A页面到B页面的链接解释为A页面给B页面投票。 \" \\\n           \"Google根据投票来源甚至来源的来源，即链接到A页面的页面。 \" \\\n           \"和投票目标的等级来决定新的等级。简单的说， \" \\\n           \"一个高等级的页面可以使其他低等级页面的等级提升。 \" \\\n           \"具体说来就是，PageRank有两个基本思想，也可以说是假设。 \" \\\n           \"即数量假设：一个网页被越多的其他页面链接，就越重）。 \" \\\n           \"质量假设：一个网页越是被高质量的网页链接，就越重要。 \" \\\n           \"总的来说就是一句话，从全局角度考虑，获取重要的信。 \"\n           \n# 新词发现(findword, 默认接口)\nsents = macropodus.find(text=summary, use_type=\"text\", use_avg=False, use_filter=False, use_output=True, freq_min=2, len_max=5, entropy_min=2.0, aggregation_min=3.2)\nprint(sents)\n\n```\n\n\n## 关键词\n\n  关键词抽取使用的是textrank, 边关系构建: 1. 字向量构建句向量; 2. 余弦相似度计算边得分\n```python3\nimport macropodus\n\nsent = \"斗鱼属，Macropodus (1801)，鲈形目斗鱼科的一属鱼类。本属鱼类通称斗鱼。因喜斗而得名。分布于亚洲东南部。中国有2种，即叉尾斗鱼，分布于长江及以南各省；叉尾斗鱼，分布于辽河到珠江流域。其喜栖居于小溪、河沟、池塘、稻田等缓流或静水中。雄鱼好斗，产卵期集草成巢，雄鱼口吐粘液泡沫，雌鱼产卵其中，卵浮性，受精卵在泡沫内孵化。雄鱼尚有护卵和护幼现象。\"\n# 关键词(keyword)\nsents = macropodus.keyword(sent)\nprint(sents)\n\n```\n\n## 命名实体提取\n   * ner, albert+bilstm+crf网络架构, 最大支持126个字符; \n   * 需要安装tensorflow==1.15.0, keras-bert==0.80.0, keras-adaptive-softmax==0.6.0(pip安装不默认下载, 1.15.0以下未实验, 1.13以上应该可以); \n   * 需要下载模型(pip安装不默认下载, 将ner_albert_people_1998覆盖到安装目录macropodus/data/model);\n   * 模型地址在[https://pan.baidu.com/s/1I3vydhmFEQ9nuPG2fDou8Q](https://pan.baidu.com/s/1I3vydhmFEQ9nuPG2fDou8Q), 提取码：rket;\n\n```python3\nimport macropodus\n\nsummary = \"美丽的广西是我国华南地区的一颗璀璨的明珠,山清水秀生态美,风生水起万象新。\"\nres_ner = macropodus.ner(summary)\nprint(res_ner)\nres_ners = macropodus.ners([summary])\nprint(res_ners)\n```\n\n## 词性标注\n   * pos tag, albert+bilstm+crf网络架构, 最大支持126个字符; \n   * 需要安装tensorflow==1.15.0, keras-bert==0.80.0, keras-adaptive-softmax==0.6.0(pip安装不默认下载, 1.15.0以下未实验, 1.13以上应该可以); \n   * 需要下载模型(pip安装不默认下载, 将tag_albert_people_1998覆盖到安装目录macropodus/data/model);\n   * 模型地址在[https://pan.baidu.com/s/1I3vydhmFEQ9nuPG2fDou8Q](https://pan.baidu.com/s/1I3vydhmFEQ9nuPG2fDou8Q), 提取码：rket;\n\n```python3\nimport macropodus\n\nsummary = \"美丽的广西是我国华南地区的一颗璀璨的明珠,山清水秀生态美,风生水起万象新。\"\nres_postag = macropodus.postag(summary)\nprint(res_postag)\nres_postags = macropodus.postags([summary])\nprint(res_postags)\n```\n\n## 常用小工具(tookit)\n\n  工具包括科学计算器, 中文繁体-简体转换, 阿拉伯-中文数字转换, 罗马数字-阿拉伯数字转换, 中文拼音\n```python3\nimport macropodus\n\nsen_calculate = \"23 + 13 * (25+(-9-2-5-2*3-6/3-40*4/(2-3)/5+6*3))加根号144你算得几多\"\nsen_chi2num = \"三千零七十八亿三千零十五万零三百一十二点一九九四\"\nsen_num2chi = 1994.1994\nsen_roman2num = \"IX\"\nsen_num2roman = 132\n# tookit, 科学计算器\nscore_calcul = macropodus.calculate(sen_calculate)\nprint(score_calcul)\n# tookit, 中文数字转阿拉伯\nres_chi2num = macropodus.chi2num(sen_chi2num)\nprint(res_chi2num)\n# tookit, 阿拉伯数字转中文\nres_num2chi = macropodus.num2chi(sen_num2chi)\nprint(res_num2chi)\n# tookit, 阿拉伯数字转罗马数字\nres_roman2num = macropodus.roman2num(sen_roman2num)\nprint(res_roman2num)\n# tookit, 罗马数字转阿拉伯数字\nres_num2roman = macropodus.num2roman(sen_num2roman)\nprint(res_num2roman)\n# 中文汉字转拼音\nres_pinyin = macropodus.pinyin(summary)\nprint(res_pinyin)\n# 中文繁体转简体\nres_zh2han = macropodus.zh2han(summary)\nprint(res_zh2han)\n# 中文简体转繁体\nres_han2zh = macropodus.han2zh(res_zh2han)\nprint(res_han2zh)\n\n```\n\n\n# 参考/引用\n* StringPreHandler(提取数字,纯数字):[https://github.com/zhanzecheng/Time_NLP](https://github.com/zhanzecheng/Time_NLP)\n* textrank_gensim: [https://github.com/RaRe-Technologies/gensim](https://github.com/RaRe-Technologies/gensim)\n* 最大概率(DAG-动态规划)词典分词: [https://github.com/fxsjy/jieba](https://github.com/fxsjy/jieba)\n* CRF(-未解决): [https://github.com/BrikerMan/Kashgari](https://github.com/BrikerMan/Kashgari)\n\n\n# Reference\nFor citing this work, you can refer to the present GitHub project. For example, with BibTeX:\n```\n@misc{Macropodus,\n    howpublished = {\\url{https://github.com/yongzhuo/Macropodus}},\n    title = {Macropodus},\n    author = {Yongzhuo Mo},\n    publisher = {GitHub},\n    year = {2020}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyongzhuo%2Fmacropodus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyongzhuo%2Fmacropodus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyongzhuo%2Fmacropodus/lists"}