{"id":19465196,"url":"https://github.com/luozhouyang/autophrasex","last_synced_at":"2025-04-09T14:06:35.219Z","repository":{"id":39593730,"uuid":"313813975","full_name":"luozhouyang/AutoPhraseX","owner":"luozhouyang","description":"Automated Phrase Mining from Massive Text Corpora in Python.","archived":false,"fork":false,"pushed_at":"2021-05-23T05:55:42.000Z","size":93,"stargazers_count":171,"open_issues_count":4,"forks_count":37,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-02T07:48:51.346Z","etag":null,"topics":["autophrase","phrase-extraction","phrase-mining","text-mining"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/luozhouyang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-18T03:35:59.000Z","updated_at":"2025-03-27T07:33:12.000Z","dependencies_parsed_at":"2022-07-11T05:34:21.357Z","dependency_job_id":null,"html_url":"https://github.com/luozhouyang/AutoPhraseX","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luozhouyang%2FAutoPhraseX","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luozhouyang%2FAutoPhraseX/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luozhouyang%2FAutoPhraseX/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luozhouyang%2FAutoPhraseX/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/luozhouyang","download_url":"https://codeload.github.com/luozhouyang/AutoPhraseX/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248054227,"owners_count":21039952,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autophrase","phrase-extraction","phrase-mining","text-mining"],"created_at":"2024-11-10T18:19:10.131Z","updated_at":"2025-04-09T14:06:35.188Z","avatar_url":"https://github.com/luozhouyang.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AutoPhraseX\n\n![Python package](https://github.com/luozhouyang/autophrasex/workflows/Python%20package/badge.svg)\n[![PyPI version](https://badge.fury.io/py/autophrasex.svg)](https://badge.fury.io/py/autophrasex)\n[![Python](https://img.shields.io/pypi/pyversions/autophrasex.svg?style=plastic)](https://badge.fury.io/py/autophrasex)\n\n\nAutomated Phrase Mining from Massive Text Corpora in Python.\n\n\n实现思路参考 [shangjingbo1226/AutoPhrase](https://github.com/shangjingbo1226/AutoPhrase)，并不完全一致。\n\n## 安装\n\n```bash\npip install -U autophrasex\n```\n\n## 基本使用\n\n```python\nfrom autophrasex import *\n\n# 构造autophrase\nautophrase = AutoPhrase(\n    reader=DefaultCorpusReader(tokenizer=JiebaTokenizer()),\n    selector=DefaultPhraseSelector(),\n    extractors=[\n        NgramsExtractor(N=4), \n        IDFExtractor(), \n        EntropyExtractor()\n    ]\n)\n\n# 开始挖掘\npredictions = autophrase.mine(\n    corpus_files=['data/answers.10000.txt'],\n    quality_phrase_files='data/wiki_quality.txt',\n    callbacks=[\n        LoggingCallback(),\n        ConstantThresholdScheduler(),\n        EarlyStopping(patience=2, min_delta=3)\n    ])\n\n# 输出挖掘结果\nfor pred in predictions:\n    print(pred)\n\n```\n\n\n## 高级用法\n\n本项目的各个关键步骤都是可以扩展的，所以大家可以自由实现自己的逻辑。\n\n本项目大体上可以氛围以下几个主要模块：\n\n* `tokenizer`分词器模块\n* `reader`语料读取模块\n* `selector`高频短语的选择模块\n* `extractors`特征抽取器，用于抽取分类器所需要的特征\n* `callbacks`挖掘周期的回调模块\n\n以下是每个模块的高级使用方法。\n\n### tokenizer\n\n`tokenizer`用于文本分词，用户可以继承`AbstractTokenizer`实现自己的分词器。本库自带`JiebaTokenizer`。\n\n例如，你可以使用`baidu/LAC`来进行中文分词。你可以这样实现分词器：\n\n```python\n# pip install lac\n\nclass BaiduLacTokenizer(AbstractTokenizer):\n\n    def __init__(self, custom_vocab_path=None, model_path=None, mode='seg', use_cuda=False, **kwargs):\n        self.lac = LAC(model_path=model_path, mode=mode, use_cuda=use_cuda)\n        if custom_vocab_path:\n            self.lac.load_customization(custom_vocab_path)\n\n    def tokenize(self, text, **kwargs):\n        text = self._uniform_text(text, **kwargs)\n        results = self.lac.run(text)\n        results = [x.strip() for x in results if x.strip()]\n        return results\n```\n\n然后在构建`reader`的使用使用`BaiduLacTokenizer`:\n```python\nreader = DefaultCorpusReader(tokenizer=BaiduLacTokenizer())\n```\n\n### reader\n\n`reader`用于读取语料，用户可以继承`AbstractCorpusReader`实现自己的Reader。本库自带`DefaultCorpusReader`。\n\n因为目前的`extractor`其实是依赖`reader`的（具体来说是`extractor`实现了`reader`的生命周期回调接口），因此想要重写`reader`，在有些情况下需要同时更改`extractor`的实现，此时自定义成本比较大，暂时不推荐重写`reader`。\n\n### selector\n\n`selector`用于选择高频Phrase，用户可以继承`AbstractPhraseSelector`实现自己的Phrase选择器。本库自带`DefaultPhraseSelector`。\n\n`selector`可以拥有多个`phrase_filter`，用于实现Phrase的过滤。关于`phrase_filter`本库提供了开放的接口，用户可以继承`AbstractPhraseFilter`实现自己的过滤器。本库自带了默认的过滤器`DefaultPhraseFilter`，并且在默认情况下使用。\n\n如果你想要禁用默认的过滤器，转而使用自己实现的过滤器，可以在构造`selector`的时候设置：\n\n```python\n# 自定义过滤器\nclass MyPhraseFilter(AbstractPhraseFilter):\n\n    def apply(self, pair, **kwargs):\n        phrase, freq = pair\n        # return True to filter this phrase\n        if is_verb(phrase):\n            return True\n        return False\n\nselector = DefaultPhraseSelector(\n    phrase_filters=[MyPhraseFilter()], \n    use_default_phrase_filters=False\n)\n```\n\n考虑到有些过滤过程，使用按批处理可以显著提升速度(例如使用深度学习模型计算词性)，`phrase_filter`提供了一个`batch_apply`方法。\n\n举个例子，使用`baidu/LAC`来计算词性，从而实现Phrase的过滤：\n\n```python\n\nclass VerbPhraseFilter(AbstractPhraseFilter):\n\n    def __init__(self, batch_size=100):\n        super().__init__()\n        self.lac = LAC()\n        self.batch_size = batch_size\n\n    def batch_apply(self, batch_pairs, **kwargs):\n        predictions = []\n        for i in range(0, len(batch_pairs), self.batch_size):\n            batch_texts = [x[0] for x in batch_pairs[i: i + self.batch_size]]\n            batch_preds = self.lac.run(batch_texts)\n            predictions.extend(batch_preds)\n        candidates = []\n        for i in range(len(predictions)):\n            _, pos_tags = predictions[i]\n            if any(pos in ['v', 'vn', 'vd'] for pos in pos_tags):\n                continue\n            candidates.append(batch_pairs[i])\n        return candidates\n\nselector = DefaultPhraseSelector(\n    phrase_filters=[VerbPhraseFilter()], \n    use_default_phrase_filters=False\n)\n```\n\n### extractor\n\n`extractor`用于抽取分类器的特征。特征抽取器会在`reader`读取语料的时候进行必要信息的统计。因此`extractor`实现了`reader`的回调接口，所以在自定义特征抽取器的时候，需要同时继承`AbstractCorpusReadCallback`和`AbstractFeatureExtractor`。\n\n本库自带了以下几个特征抽取器：\n\n* `NgramExtractor`，`n-gram`特征抽取器，可以计算phrase的`pmi`特征\n* `IDFExtractor`，`idf`特征抽取器，可以计算phrase的`doc_freq`、`idf`特征\n* `EntropyExtractor`，`熵`特征抽取器，可以计算phrase的`左右熵`特征\n\n上述自带的特征抽取器，都是基于`n-gram`统计的，因此都支持`ngram`的选择，也就是都可以自定义`ngram_filter`来过滤不需要统计的`ngram`。本库自带了`DefaultNgramFilter`，并且默认启用。用户可以实现自己的`ngram_filter`来灵活选取合适的`ngram`。\n\n举个例子，我需要过滤掉`包含标点符号`的`ngram`：\n\n```python\nCHARACTERS = set('!\"#$%\u0026\\'()*+,-./:;?@[\\\\]^_`{|}~ \\t\\n\\r\\x0b\\x0c，。？：“”【】「」')\n\nclass MyNgramFilter(AbstractNgramFiter):\n\n    def apply(self, ngram, **kwargs):\n        if any(x in CHARACTERS for x in ngram):\n            return True\n        return False\n\nautophrase = AutoPhrase(\n    reader=DefaultCorpusReader(tokenizer=JiebaTokenizer()),\n    selector=DefaultPhraseSelector(),\n    extractors=[\n        NgramsExtractor(N=4, ngram_filters=[MyNgramFilter()]), \n        IDFExtractor(ngram_filters=[MyNgramFilter()]), \n        EntropyExtractor(ngram_filters=[MyNgramFilter()]),\n    ]\n)\n# 开始挖掘\n...\n```\n\n用户可以继承`AbstractFeatureExtractor`实现自己的特征计算。只需要在构建autophrase实例的时候，把这些特征计算器传入即可，不需要做其他任何额外操作。\n\n举个例子，我增加一个`phrase是否是unigram`的特征：\n\n```python\nclass UnigramFeatureExtractor(AbstractFeatureExtractor，AbstractCorpusReadCallback):\n\n    def __init__(self, **kwargs):\n        super().__init__(**kwargs)\n\n    def extract(self, phrase, **kwargs):\n        parts = phrase.split(' ')\n        features = {\n            'is_unigram': 1 if len(parts) == 1 else 0\n        }\n        return features\n\n\nautophrase = AutoPhrase(\n    reader=DefaultCorpusReader(tokenizer=JiebaTokenizer()),\n    selector=DefaultPhraseSelector(),\n    extractors=[\n        NgramsExtractor(N=N), \n        IDFExtractor(), \n        EntropyExtractor(),\n        UnigramFeatureExtractor(),\n    ]\n)\n\n# 可以开始挖掘了\n...\n```\n\n### callback\n\n`callback`回调接口，可以提供phrase挖掘过程中的生命周期监听，并且实现一些稍微复杂的功能，例如`EarlyStopping`、`判断阈值Schedule`等。\n\n本库自带以下回调：\n\n* `LoggingCallback`提供挖掘过程的日志信息打印\n* `ConstantThresholdScheduler`在训练过程中调整阈值的回调\n* `EarlyStopping`早停，在指标没有改善的情况下停止训练\n\n用户可以自己继承`Callback`实现自己的逻辑。\n\n\n## 结果示例\n\n\u003e 以下结果属于本库比较早期的测试效果，目前本库的代码更新比较大，返回结果和下述内容不太一致。仅供参考。\n\n新闻语料上的抽取结果示例：\n\n```bash\n成品油价格, 0.992766816097071\n股份制银行, 0.992766816097071\n公务船, 0.992766816097071\n中国留学生, 0.992766816097071\n贷款基准, 0.992766816097071\n欧足联, 0.992766816097071\n新局面, 0.992766816097071\n淘汰赛, 0.992766816097071\n反动派, 0.992766816097071\n生命危险, 0.992766816097071\n新台阶, 0.992766816097071\n知名度, 0.992766816097071\n新兴产业, 0.9925660976153782\n安全感, 0.9925660976153782\n战斗力, 0.9925660976153782\n战略性, 0.9925660976153782\n私家车, 0.9925660976153782\n环球网, 0.9925660976153782\n副校长, 0.9925660976153782\n流行语, 0.9925660976153782\n债务危机, 0.9925660976153782\n保险资产, 0.9920376397372204\n保险机构, 0.9920376397372204\n豪华车, 0.9920376397372204\n环境质量, 0.9920376397372204\n瑞典队, 0.9919345469537152\n交强险, 0.9919345469537152\n马卡报, 0.9919345469537152\n生产力, 0.9911077251879798\n```\n\n医疗对话语料的抽取示例：\n\n```bash\n左眉弓, 1.0\n支原体, 1.0\nmri, 1.0\n颈动脉, 0.9854149008885851\n结核病, 0.9670815675552518\n手术室, 0.9617546444783288\n平扫示, 0.9570324222561065\n左手拇指, 0.94\n双膝关节, 0.94\n右手中指, 0.94\n拇指末节, 0.94\ncm皮肤, 0.94\n肝胆脾, 0.94\n抗体阳性, 0.94\nigm抗体阳性, 0.94\n左侧面颊, 0.94\n膀胱结石, 0.94\n左侧基底节, 0.94\n腰椎正侧, 0.94\n软组织肿胀, 0.94\n手术瘢痕, 0.94\n枕顶部, 0.94\n左膝关节正侧, 0.94\n膝关节正侧位, 0.94\n腰椎椎体, 0.94\n承德市医院, 0.94\n性脑梗塞, 0.94\n颈椎dr, 0.94\n泌尿系超声, 0.94\n双侧阴囊, 0.94\n右颞部, 0.94\n肺炎支原体, 0.94\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluozhouyang%2Fautophrasex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fluozhouyang%2Fautophrasex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluozhouyang%2Fautophrasex/lists"}