{"id":17176156,"url":"https://github.com/upbit/cn_segment","last_synced_at":"2025-04-13T17:08:50.182Z","repository":{"id":21155724,"uuid":"24458229","full_name":"upbit/cn_segment","owner":"upbit","description":"Chinese word segmentation based on statistical methods (for Python)","archived":false,"fork":false,"pushed_at":"2015-08-25T02:48:44.000Z","size":22016,"stargazers_count":7,"open_issues_count":0,"forks_count":7,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-13T17:08:44.201Z","etag":null,"topics":["python"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/upbit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-09-25T13:04:30.000Z","updated_at":"2021-02-10T02:26:46.000Z","dependencies_parsed_at":"2022-07-29T18:40:04.091Z","dependency_job_id":null,"html_url":"https://github.com/upbit/cn_segment","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/upbit%2Fcn_segment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/upbit%2Fcn_segment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/upbit%2Fcn_segment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/upbit%2Fcn_segment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/upbit","download_url":"https://codeload.github.com/upbit/cn_segment/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248750107,"owners_count":21155686,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python"],"created_at":"2024-10-14T23:59:18.662Z","updated_at":"2025-04-13T17:08:50.161Z","avatar_url":"https://github.com/upbit.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## 统计学中文分词的Python版本\n\n参照《数据之美》第14章(Word Segmentation)，和《数学之美》中统计学分词方法，实现的最小统计学分词脚本。其实早就搁置在一边了，最近和朋友聊到中文分词才发现这个东西可能有人需要，放出来共享下。\n\n~~~ python\n#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\nimport sys\n\nimport re, string, random, glob, operator, heapq\nfrom collections import defaultdict\nfrom math import log10\n\ndef memo(f):\n\t\"Memoize function f.\"\n\ttable = {}\n\tdef fmemo(*args):\n\t\tif args not in table:\n\t\t\ttable[args] = f(*args)\n\t\treturn table[args]\n\tfmemo.memo = table\n\treturn fmemo\n\n################ Word Segmentation (p. 223)\n\n@memo\ndef segment(text):\n\t\"Return a list of words that is the best segmentation of text.\"\n\tif not text:\n\t\treturn []\n\n\t#candidates = ([first]+segment(rem) for first,rem in splits(text))\t\n\tcandidates = []\n\tfor first,rem in splits(text):\n\t\tcandidates.append([first]+segment(rem))\n\n\treturn max(candidates, key=Pwords)\n\ndef splits(text, L=20):\n\t\"Return a list of all possible (first, rem) pairs, len(first)\u003c=L.\"\n\treturn [(text[:i+1], text[i+1:]) \n\t\t\tfor i in range(min(len(text), L))]\n\ndef Pwords(words): \n\t\"The Naive Bayes probability of a sequence of words.\"\n\treturn product(Pw(w) for w in words)\n\ndef product(nums):\n\t\"Return the product of a sequence of numbers.\"\n\treturn reduce(operator.mul, nums, 1)\n\nclass Pdist(dict):\n\t\"A probability distribution estimated from counts in datafile.\"\n\tdef __init__(self, data=[], N=None, missingfn=None):\n\t\tfor key,count in data:\n\t\t\tself[key] = self.get(key, 0) + int(count)   # 映射到map并去重\n\t\tself.N = float(N or sum(self.itervalues()))\n\t\tself.missingfn = missingfn or (lambda k, N: 1./N)\n\tdef __call__(self, key): \n\t\tif key in self: return self[key]/self.N  \n\t\telse: return self.missingfn(key, self.N)\n\ndef datafile(name, sep='\\t'):\n\t\"Read key,value pairs from file.\"\n\tfor line in file(name):\n\t\tyield line.split(sep)\n\ndef avoid_long_words(key, N):\n\t\"Estimate the probability of an unknown word.\"\n\treturn 10./(N * 10**len(key))\n\n# global\nN = 43514267000000 ## Number of tokens\nPw = Pdist(datafile('utf8_frequency_dict.txt', ' '), N, avoid_long_words)\n\ndef main():\n\t# 注意: segment()输入的字符集，需要和字典匹配\n\tword_list = segment('研究生教育研究生命起源不能改变研究生命运')\n\t\n\t# 输出：研究生 教育 研究 生命 起源 不能 改变 研究生 命运\n\tprint \" \".join(word_list)\n\nif __name__ == '__main__':\n\tmain()\n~~~\n\n基本上只是将[Beautiful Data](books.google.com.hk/books?isbn=144937929X)14章Word Segmentation的segment.py改了下，使之能支持中文。性能上还有很多可优化的空间，比如在求max(candidates, key=Pwords)的过程上，不过为了保持原来的简洁，实在不忍动手糟蹋...\n\n另外附带一个算TF/IDF的例子，帮助提取关键词：\n\n~~~\n研究生 教育 研究 生命 起源 不能 改变 研究生 命运\n\nTF/IDF:\n  研究生 0.0938609996672\n  起源 0.141864047156\n  教育 2.42227567084\n  命运 3.52080771578\n  研究 8.07468710161\n  改变 9.70901893519\n  生命 10.7189498685\n  不能 39.934120938\n~~~\n\n关于隐马尔可夫链的词性标注，可以看之前的另一个项目：[Hidden Markov Model](https://github.com/upbit/HiddenMarkovModel)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fupbit%2Fcn_segment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fupbit%2Fcn_segment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fupbit%2Fcn_segment/lists"}