{"id":15038781,"url":"https://github.com/yongzhuo/nlp_xiaojiang","last_synced_at":"2025-05-15T16:08:16.801Z","repository":{"id":40614176,"uuid":"180307968","full_name":"yongzhuo/nlp_xiaojiang","owner":"yongzhuo","description":"自然语言处理（nlp），小姜机器人（闲聊检索式chatbot），BERT句向量-相似度（Sentence Similarity），XLNET句向量-相似度（text xlnet embedding），文本分类（Text classification）， 实体提取（ner，bert+bilstm+crf），数据增强（text augment, data enhance），同义句同义词生成，句子主干提取（mainpart），中文汉语短文本相似度，文本特征工程，keras-http-service调用","archived":false,"fork":false,"pushed_at":"2021-09-23T05:44:27.000Z","size":24490,"stargazers_count":1534,"open_issues_count":2,"forks_count":392,"subscribers_count":40,"default_branch":"master","last_synced_at":"2025-05-12T08:05:53.599Z","etag":null,"topics":["bert","chatbot","chinese","data-augmentation","distance","enhance","feature","nlp","text-augment","text-classification","xlnet"],"latest_commit_sha":null,"homepage":"https://blog.csdn.net/rensihui","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yongzhuo.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-04-09T07:17:05.000Z","updated_at":"2025-05-11T22:55:01.000Z","dependencies_parsed_at":"2022-07-18T04:46:39.892Z","dependency_job_id":null,"html_url":"https://github.com/yongzhuo/nlp_xiaojiang","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yongzhuo%2Fnlp_xiaojiang","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yongzhuo%2Fnlp_xiaojiang/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yongzhuo%2Fnlp_xiaojiang/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yongzhuo%2Fnlp_xiaojiang/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yongzhuo","download_url":"https://codeload.github.com/yongzhuo/nlp_xiaojiang/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254374475,"owners_count":22060611,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","chatbot","chinese","data-augmentation","distance","enhance","feature","nlp","text-augment","text-classification","xlnet"],"created_at":"2024-09-24T20:40:10.284Z","updated_at":"2025-05-15T16:08:16.760Z","avatar_url":"https://github.com/yongzhuo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nlp_xiaojiang\n\n\n# AugmentText\n    - 回译（效果比较好）\n    - EDA（同义词替换、插入、交换和删除）（效果还行）\n    - HMM-marko（质量较差）\n    - syntax（依存句法、句法、语法书）（简单句还可）\n    - seq2seq（深度学习同义句生成，效果不理想，seq2seq代码大都是 [https://github.com/qhduan/just_another_seq2seq] 的，效果不理想）\n    - 预训练(UNILM生成、开源模型回译)\n    \n# ChatBot\n    - 检索式ChatBot\n        - 像ES那样直接检索(如使用fuzzywuzzy)，只能字面匹配\n        - 构造句向量，检索问答库，能够检索有同义词的句子\n    - 生成式ChatBot（todo）\n        - seq2seq\n        - GAN\n\n# ClassificationText\n    - bert+bi-lstm(keras) approach 0.78~0.79% acc of weBank Intelligent Customer Service Question Matching Competition\n    - bert + text-cnn(keras) approach 0.78~0.79% acc of weBank Intelligent Customer Service Question Matching Competition\n    - bert + r-cnn(keras) approach 0.78~0.79% acc of weBank Intelligent Customer Service Question Matching Competition\n    - bert + avt-cnn(keras) approach 0.78~0.79% acc of weBank Intelligent Customer Service Question Matching Competition\n\n# Ner\n    - bert命名实体提取(bert12层embedding + bilstm + crf)\n        - args.py(配置一些参数)\n        - keras_bert_embedding.py(bert embedding)\n        - keras_bert_layer.py(layer层, 主要有CRF和NonMaskingLayer)\n        - keras_bert_ner_bi_lstm.py(主函数, 定义模型、数据预处理和训练预测等)\n        - layer_crf_bojone.py(CRF层, 未使用)\n\n# FeatureProject\n    - bert句向量、文本相似度\n        - bert/extract_keras_bert_feature.py:提取bert句向量特征\n        - bert/tet_bert_keras_sim.py:测试xlnet句向量cosin相似度\n    - xlnet句向量、文本相似度\n        - xlnet/extract_keras_xlnet_feature.py:提取bert句向量特征\n        - xlnet/tet_xlnet_keras_sim.py:测试bert句向量cosin相似度\n    - normalization_util指的是数据归一化\n        - 0-1归一化处理\n        - 均值归一化\n        - sig归一化处理\n    - sim feature（ML）\n        - distance_text_or_vec:各种计算文本、向量距离等\n        - distance_vec_TS_SS：TS_SS计算词向量距离\n        - cut_td_idf：将小黄鸡语料和gossip结合\n        - sentence_sim_feature：计算两个文本的相似度或者距离，例如qq（问题和问题），或者qa（问题和答案）\n\n# run(可以在win10下,pycharm下运行)\n  - 1.创建tf-idf文件等（运行2需要先跑1）:      \n                                       ```\n                                       python cut_td_idf.py\n                                       ```\n  - 2.计算两个句子间的各种相似度，先计算一个预定义的，然后可输入自定义的（先跑1）:  \n                                       ```\n                                       python sentence_sim_feature.py\n                                       ```\n  - 3.chatbot_1跑起来(fuzzy检索-没)（独立）：    \n                                       ```\n                                       python chatbot_fuzzy.py\n                                       ```\n  - 4.chatbot_2跑起来(句向量检索-词)（独立）：    \n                                       ```\n                                       python chatbot_sentence_vec_by_word.py\n                                       ```\n  - 5.chatbot_3跑起来(句向量检索-字)（独立）：    \n                                       ```\n                                       python chatbot_sentence_vec_by_char.py\n                                       ```\n  - 6.数据增强（eda)：                     python enhance_eda.py\n  - 7.数据增强（marko）:                   python enhance_marko.py\n  - 8.数据增强（translate_account）:       python translate_tencent_secret.py\n  - 9.数据增强（translate_tools）:         python translate_translate.py\n  - 10.数据增强（translate_web）:          python translate_google.py\n  - 11.数据增强（augment_seq2seq）:        先跑 python extract_char_webank.py生成数据，\n                                          再跑 python train_char_anti.py\n                                          然后跑 python predict_char_anti.py\n  - 12.特征计算(bert)（提取特征、计算相似度）: \n                      ```\n                      run extract_keras_bert_feature.py\n                      run tet_bert_keras_sim.py\n                      ```\n                      \n# Data\n    - chinese_L-12_H-768_A-12（谷歌预训练好的模型）\n       github项目中只是上传部分数据，需要的前往链接: https://pan.baidu.com/s/1I3vydhmFEQ9nuPG2fDou8Q 提取码: rket\n       解压后就可以啦\n    - chinese_xlnet_mid_L-24_H-768_A-12(哈工大训练的中文xlnet, mid, 24层, wiki语料+通用语料)\n        - 下载地址[https://github.com/ymcui/Chinese-PreTrained-XLNet](https://github.com/ymcui/Chinese-PreTrained-XLNet)\n    - chinese_vector\n        github项目中只是上传部分数据，需要的前往链接: https://pan.baidu.com/s/1I3vydhmFEQ9nuPG2fDou8Q 提取码: rket\n        - 截取的部分word2vec训练词向量（自己需要下载全效果才会好）\n        - w2v_model_wiki_char.vec、w2v_model_wiki_word.vec都只有部分，词向量w2v_model_wiki_word.vec可以用这个下载地址的替换[https://pan.baidu.com/s/14JP1gD7hcmsWdSpTvA3vKA](https://pan.baidu.com/s/14JP1gD7hcmsWdSpTvA3vKA)\n\n    - corpus\n        github项目中只是上传部分数据，需要的前往链接: https://pan.baidu.com/s/1I3vydhmFEQ9nuPG2fDou8Q 提取码: rket\n        - ner(train、dev、test----人民日报语料)\n        - webank(train、dev、test)\n        - 小黄鸡和gossip问答预料（数据没清洗）,chicken_and_gossip.txt\n        - 微众银行和支付宝文本相似度竞赛数据， sim_webank.csv\n    - sentence_vec_encode_char\n        - 1.txt（字向量生成的前100000句向量）\n    - sentence_vec_encode_word\n        - 1.txt（词向量生成的前100000句向量）\n    - tf_idf（chicken_and_gossip.txt生成的tf-idf）\n    \n# requestments.txt\n    - python_Levenshtei\n        - 调用Levenshtein，我的python是3.6，\n        - 打开其源文件: https://www.lfd.uci.edu/~gohlke/pythonlibs/\n        - 查找python_Levenshtein-0.12.0-cp36-cp36m-win_amd64.whl下载即可\n    - pyemd\n    - pyhanlp\n        - 下好依赖JPype1-0.6.3-cp36-cp36m-win_amd64.whl\n  \n# 参考/感谢\n* eda_chinese：[https://github.com/zhanlaoban/eda_nlp_for_Chinese](https://github.com/zhanlaoban/eda_nlp_for_Chinese)\n* 主谓宾提取器：[https://github.com/hankcs/MainPartExtractor](https://github.com/hankcs/MainPartExtractor)\n* HMM生成句子：[https://github.com/takeToDreamLand/SentenceGenerate_byMarkov](https://github.com/takeToDreamLand/SentenceGenerate_byMarkov)\n* 同义词等：[https://github.com/fighting41love/funNLP/tree/master/data/](https://github.com/fighting41love/funNLP/tree/master/data/)\n* 小牛翻译：[http://www.niutrans.com/index.html](http://www.niutrans.com/index.html)\n    \n# 其他资料\n* bert(keras):[https://github.com/CyberZHG/keras-bert](https://github.com/CyberZHG/keras-bert)\n* NLP数据增强汇总:[https://github.com/quincyliang/nlp-data-augmentation](https://github.com/quincyliang/nlp-data-augmentation)\n* 知乎NLP数据增强话题:[https://www.zhihu.com/question/305256736/answer/550873100](https://www.zhihu.com/question/305256736/answer/550873100)\n* chatbot_seq2seq_seqGan（比较好用）：[https://github.com/qhduan/just_another_seq2seq](https://github.com/qhduan/just_another_seq2seq)\n* 自己动手做聊天机器人教程: [https://github.com/warmheartli/ChatBotCourse](https://github.com/warmheartli/ChatBotCourse)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyongzhuo%2Fnlp_xiaojiang","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyongzhuo%2Fnlp_xiaojiang","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyongzhuo%2Fnlp_xiaojiang/lists"}