{"id":24716801,"url":"https://github.com/dovolopor-research/cnlp","last_synced_at":"2025-07-17T13:34:04.061Z","repository":{"id":62563151,"uuid":"142736107","full_name":"dovolopor-research/cnlp","owner":"dovolopor-research","description":"🔥 专注于中文的「自然语言处理框架」：中文分词；平衡类别；数据集划分...","archived":false,"fork":false,"pushed_at":"2020-11-14T05:20:05.000Z","size":5182,"stargazers_count":13,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-07T01:05:14.468Z","etag":null,"topics":["ai","cnlp","deep-learning","machine-learning","natural-language-processing","neural-network","nlp","nlp-library","rnn"],"latest_commit_sha":null,"homepage":"https://cnlp.dovolopor.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dovolopor-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-29T06:19:11.000Z","updated_at":"2024-11-05T08:21:01.000Z","dependencies_parsed_at":"2022-11-03T15:45:25.797Z","dependency_job_id":null,"html_url":"https://github.com/dovolopor-research/cnlp","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dovolopor-research/cnlp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dovolopor-research%2Fcnlp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dovolopor-research%2Fcnlp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dovolopor-research%2Fcnlp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dovolopor-research%2Fcnlp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dovolopor-research","download_url":"https://codeload.github.com/dovolopor-research/cnlp/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dovolopor-research%2Fcnlp/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265611356,"owners_count":23797872,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","cnlp","deep-learning","machine-learning","natural-language-processing","neural-network","nlp","nlp-library","rnn"],"created_at":"2025-01-27T09:14:21.727Z","updated_at":"2025-07-17T13:34:04.039Z","avatar_url":"https://github.com/dovolopor-research.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://ailln.oss-cn-hangzhou.aliyuncs.com/github/cnlp/cnlp-logo-v4.png\" width=\"60%\"\u003e\n  \u003cbr\u003e\n  \u003cimg src=\"https://img.shields.io/pypi/v/cnlp.svg\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/license-MIT-green.svg\"\u003e\n  \u003cimg src=\"https://img.shields.io/github/stars/dovolopor-research/cnlp.svg\"\u003e\n\u003c/div\u003e\n\n# cnlp = cn + nlp\n\n🔥 专注于中文的「自然语言处理框架」\n\n## 1 功能\n\n1. [cnlp.data](https://cnlp.dovolopor.com/api/#_1-cnlp-data): 一个通用的文本处理器。\n2. [cnlp.dataset](https://cnlp.dovolopor.com/api/#_2-cnlp-dataset): 提供常见的数据集加载。\n3. [cnlp.model](https://cnlp.dovolopor.com/api/#_3-cnlp-model): 提供常见的模型。\n\n## 2 安装\n\n### 2.1 pip 安装\n\n```bash\npip install cnlp\n```\n\n### 2.2 手动安装\n\n```bash\ngit clone https://github.com/dovolopor-research/cnlp.git\ncd cnlp \u0026\u0026 python setup.py install\n```\n\n## 3 快速上手\n\n### 中文分词\n\n分词是中文自然语言处理最基础的任务之一，我们先来看个分词的例子。\n\n```python\nfrom cnlp.data import Tokenizer\n\nt = Tokenizer()\n\n# 默认方式为有向无环图分词\nresult = t.cut(\"为中华之崛起而读书\")\n# result:\n# ['为', '中华', '之', '崛起', '而', '读书']\n\n# 机械分词\nresult = t.cut(\"为中华之崛起而读书\", methods=\"mechanical\")\n# result:\n# ['为', '中华', '之', '崛起', '而', '读书']\n\n# 获取偏旁\nresult = t.radical(\"为中华之崛起而读书\")\n# result:\n# ['丶', '丨', '十', '丶', '@UNK@', '走', '而', '讠', '乛']\n\n# 获取部首\nresult = t.stroke(\"为中华之崛起而读书\")\n# result:\n# [['点', '撇', '横折竖钩', '点'], ['竖', '横折', '横', '竖'], ['撇', '竖', '撇', '竖弯横钩', '横', '竖'],\n# ['点', '横撇', '捺'], '@UNK@', ['横', '竖', '横', '竖', '横', '撇', '撇', '竖', '横折竖钩', '竖', '竖'],\n# ['点', '横折提', '横', '竖', '横撇', '点', '点', '横', '撇', '点'], ['横折', '横折竖钩', '竖', '点']]\n```\n\n### 类别不平衡\n\n数据集中的类别不平衡是在实际数据集中经常遇到的问题，通常采用**过采样扩充**或者**欠采样缩减**的方式解决。\n\n```python\nfrom cnlp.data import sampling\n\n# 类别 pos:3 neg:2 unk:1\norigin_dataset = [\n  [\"我好开心！\", \"pos\"],\n  [\"今天心情不错啊～\", \"pos\"],\n  [\"学会了 cnlp，真棒！\", \"pos\"],\n  [\"我心态崩了啊！\", \"neg\"],\n  [\"哎，又加班。。。\", \"neg\"],\n  [\"吃雪糕中\", \"unk\"]\n]\n\n# 默认为过采样，都扩充到 3\ndataset = sampling(origin_dataset)\n# dataset:\n# [\n#   [\"我好开心！\", \"pos\"],\n#   [\"今天心情不错啊～\", \"pos\"],\n#   [\"学会了 cnlp，真棒！\", \"pos\"],\n#   [\"我心态崩了啊！\", \"neg\"],\n#   [\"哎，又加班。。。\", \"neg\"],\n#   [\"我心态崩了啊！\", \"neg\"],\n#   [\"吃雪糕中\", \"unk\"],\n#   [\"吃雪糕中\", \"unk\"],\n#   [\"吃雪糕中\", \"unk\"]\n# ]\n\n# 也可以使用欠采样，都缩减到1\n# 这种方式不推荐，容易使模型过拟合\ndataset = sampling(origin_dataset, mode=\"under\")\n# dataset:\n# [\n#   [\"学会了 cnlp，真棒！\", \"pos\"],\n#   [\"我心态崩了啊！\", \"neg\"],\n#   [\"吃雪糕中\", \"unk\"]\n# ]\n```\n\n### 数据集划分\n\n数据集划分是训练模型前的常见操作，这里也提供了一个基本的方法。\n\n```python\nfrom cnlp.data import split\n\norigin_dataset = [\n  \"今天天气不错啊\",\n  \"我想吃烧烤\",\n  \"地球人就知道吃\",\n  \"说的好像你们火星人不爱吃一样\",\n  \"没错啊！\",\n  \"好吧\",\n  \"你赢了\",\n  \"那请你离开地球\",\n  \"快走\",\n  \"快快走\"\n]\n\n# 默认以 train:test = 7:3 划分\ntrain_data, test_data = split(origin_dataset)\n# train_data:\n# ['今天天气不错啊', '我想吃烧烤', '地球人就知道吃', '说的好像你们火星人不爱吃一样', '没错啊！', '好吧', '你赢了']\n# test_data:\n# ['那请你离开地球', '快走', '快快走']\n\n# 支持划分 train val test 三种数据集\ntrain_data, val_data, test_data = split(origin_dataset, [0.5, 0.3, 0.2])\n# train_data:\n# ['今天天气不错啊', '我想吃烧烤', '地球人就知道吃', '说的好像你们火星人不爱吃一样', '没错啊！'],\n# val_data:\n# ['好吧', '你赢了', '那请你离开地球']\n# test_data:\n# ['快走', '快快走']\n\n# 支持打乱(shuffle)\ntrain_data, val_data, test_data = split(origin_dataset, [0.5, 0.3, 0.2], shuffle=True)\n# train_data:\n# ['你赢了', '说的好像你们火星人不爱吃一样', '快走', '今天天气不错啊', '那请你离开地球']\n# val_data:\n# ['我想吃烧烤', '快快走', '地球人就知道吃']\n# test_data:\n# ['没错啊！', '好吧']\n```\n\n## 4 文档\n\n请在 [https://cnlp.dovolopor.com](https://cnlp.dovolopor.com) 中查看它。\n\n## 5 贡献\n\n如果您计划为此项目提供新功能，实用程序功能或扩展，请首先打开一个 [Issues](https://github.com/dovolopor-research/cnlp/issues) 并与我们讨论该功能。\n\n## 6 许可证\n\n[![](https://award.dovolopor.com?lt=License\u0026rt=MIT\u0026rbc=green)](./LICENSE)\n\n## 7 交流\n\n欢迎添加微信号：`Ailln_`，备注「cnlp」，我邀请你进入交流群。\n\n## 8 参考\n\n- [jieba 分词](https://github.com/fxsjy/jieba)\n- [Torch Text](https://github.com/pytorch/text)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdovolopor-research%2Fcnlp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdovolopor-research%2Fcnlp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdovolopor-research%2Fcnlp/lists"}