{"id":18918080,"url":"https://github.com/dengbocong/nlp-dialogue","last_synced_at":"2025-04-15T09:32:19.270Z","repository":{"id":108531976,"uuid":"327925634","full_name":"DengBoCong/nlp-dialogue","owner":"DengBoCong","description":"A full-process dialogue system that can be deployed online","archived":false,"fork":false,"pushed_at":"2022-06-20T12:45:18.000Z","size":198581,"stargazers_count":98,"open_issues_count":0,"forks_count":25,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-28T19:39:27.259Z","etag":null,"topics":["bot","bots","chatbot","conversational-ai","deep-learning","machine-learning","natural-language-processing","nlp","nlu"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DengBoCong.png","metadata":{"files":{"readme":"README.CN.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-01-08T14:29:44.000Z","updated_at":"2024-10-18T02:55:40.000Z","dependencies_parsed_at":"2023-03-26T01:17:52.802Z","dependency_job_id":null,"html_url":"https://github.com/DengBoCong/nlp-dialogue","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DengBoCong%2Fnlp-dialogue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DengBoCong%2Fnlp-dialogue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DengBoCong%2Fnlp-dialogue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DengBoCong%2Fnlp-dialogue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DengBoCong","download_url":"https://codeload.github.com/DengBoCong/nlp-dialogue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249043006,"owners_count":21203398,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bot","bots","chatbot","conversational-ai","deep-learning","machine-learning","natural-language-processing","nlp","nlu"],"created_at":"2024-11-08T10:29:33.073Z","updated_at":"2025-04-15T09:32:17.806Z","avatar_url":"https://github.com/DengBoCong.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NLP-Dialogue | Still Work\n\n[![Blog](https://img.shields.io/badge/blog-@DengBoCong-blue.svg?style=social)](https://www.zhihu.com/people/dengbocong)\n[![Paper Support](https://img.shields.io/badge/paper-repo-blue.svg?style=social)](https://github.com/DengBoCong/nlp-paper)\n![Stars Thanks](https://img.shields.io/badge/Stars-thanks-brightgreen.svg?style=social\u0026logo=trustpilot)\n![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=social\u0026logo=appveyor)\n\n# 项目正在优化架构，可执行代码已经标记tag\n\n一个能够部署执行的全流程对话系统\n+ TensorFlow模型\n   + Transformer\n   + Seq2Seq\n   + SMN检索式模型\n   + Scheduled Sampling的Transformer\n   + GPT2\n   + Task Dialogue\n+ Pytorch模型\n   + Transformer\n   + Seq2Seq\n\n# 项目说明\n\n本项目奔着构建一个能够在线部署对话系统，同时包含开放域和面向任务型两种对话系统，针对相关模型进行复现，论文阅读笔记放置另一个项目：[nlp-paper](https://github.com/DengBoCong/nlp-paper)，项目中使用TensorFlow和Pytorch进行实现。\n\n# 语料\n仓库中的[data](https://github.com/DengBoCong/nlp-dialogue/tree/main/dialogue/data)目录下放着各语料的玩具数据，可用于验证系统执行性，完整语料以及Paper可以在[这里](https://github.com/DengBoCong/nlp-paper)查看\n\n+ LCCC\n+ CrossWOZ\n+ 小黄鸡\n+ 豆瓣\n+ Ubuntu\n+ 微博\n+ 青云\n+ 贴吧\n\n# 执行说明\n\n+ Linux执行run.sh，项目工程目录检查执行check.sh（或check.py）\n+ 根目录下的actuator.py为总执行入口，通过调用如下指令格式执行（执行前注意安装requirements.txt）：\n```\npython actuator.py --version [Options] --model [Options] ...\n```\n+ 通过根目录下的actuator.py进行执行时，`--version`、`--model`和`--act`为必传参数，其中`--version`为代码版本`tf/torch`，`--model`为执行对应的模型`transformer/smn...`，而act为执行模式（缺省状态下为`pre_treat`模式），更详细指令参数参见各模型下的`actuator.py`或config目录下的对应json配置文件。\n+ `--act`执行模式说明如下：\n   + pre_treat模式为文本预处理模式，如果在没有分词结果集以及字典的情况下，需要先运行pre_treat模式\n   + train模式为训练模式\n   + evaluate模式为指标评估模式\n   + chat模式为对话模式，chat模式下运行时，输入ESC即退出对话。\n+ 正常执行顺序为pre_treat-\u003etrain-\u003eevaluate-\u003echat\n+ 各模型下单独有一个actuator.py，可以绕开外层耦合进行执行开发，不过执行时注意调整工程目录路径\n\n# 目录结构说明\n+ dialogue下为相关模型的核心代码放置位置，方便日后进行封装打包等\n   + checkpoints为检查点保存位置\n   + config为配置文件保存目录\n   + data为原始数据储存位置，同时，在模型执行过程中产生的中间数据文件也保存在此目录下\n   + models为模型保存目录\n   + tensorflow及pytorch放置模型构建以及各模组执行的核心代码\n   + preprocess_corpus.py为语料处理脚本，对各语料进行单轮和多轮对话的处理，并规范统一接口调用\n   + read_data.py用于load_dataset.py的数据加载格式调用\n   + metrics.py为各项指标脚本\n   + tools.py为工具脚本，保存有分词器、日志操作、检查点保存/加载脚本等\n+ docs下放置文档说明，包括模型论文阅读笔记\n+ docker（mobile）用于服务端（移动终端）部署脚本\n+ server为UI服务界面，使用flask进行构建使用，执行对应的server.py即可\n+ tools为预留工具目录\n+ actuator.py（run.sh）为总执行器入口\n+ check.py（check.sh）为工程目录检查脚本\n\n\n# SMN模型运行说明\nSMN检索式对话系统使用前需要准备solr环境，solr部署系统环境推荐Linux，工具推荐使用容器部署(推荐Docker)，并准备：\n+ Solr(8.6.3)\n+ pysolr(3.9.0)\n\n以下提供简要说明，更详细可参见文章：[搞定检索式对话系统的候选response检索--使用pysolr调用Solr](https://zhuanlan.zhihu.com/p/300165220)\n## Solr环境\n需要保证solr在线上运行稳定，以及方便后续维护，请使用DockerFile进行部署，DockerFile获取地址：[docker-solr](https://github.com/docker-solr/docker-solr)\n\n仅测试模型使用，可使用如下最简构建指令：\n```\ndocker pull solr:8.6.3\n# 然后启动solr\ndocker run -itd --name solr -p 8983:8983 solr:8.6.3\n# 然后创建core核心选择器，这里取名smn(可选)\ndocker exec -it --user=solr solr bin/solr create_core -c smn\n```\n\n关于solr中分词工具有IK Analyzer、Smartcn、拼音分词器等等，需要下载对应jar，然后在Solr核心配置文件managed-schema中添加配置。\n\n**特别说明**：如果使用TF-IDF，还需要在managed-schema中开启相似度配置。\n## Python中使用说明\n线上部署好Solr之后，在Python中使用pysolr进行连接使用：\n```\npip install pysolr\n```\n\n添加索引数据（一般需要先安全检查）方式如下。将回复数据添加索引，responses是一个json,形式如：[{},{},{},...]，里面每个对象构建按照你回复的需求即可：\n```\nsolr = pysolr.Solr(url=solr_server, always_commit=True, timeout=10)\n# 安全检查\nsolr.ping()\nsolr.add(docs=responses)\n```\n\n查询方式如下，以TF-IDF查询所有语句query语句方式如下：\n```\n{!func}sum(product(idf(utterance,key1),tf(utterance,key1),product(idf(utterance,key2),tf(utterance,key2),...)\n```\n\n使用前需要先将数据添加至Solr，在本SMN模型中使用，先执行pre_treat模式即可。\n\n# Demo概览\n\u003cimg align=\"center\" height=\"400\" src=\"https://github.com/DengBoCong/nlp-dialogue/blob/main/assets/main.png\"\u003e\n\u003cimg align=\"center\" height=\"400\" src=\"https://github.com/DengBoCong/nlp-dialogue/blob/main/assets/chat.png\"\u003e\n\n# 参考代码和文献\n1. [Attention Is All You Need](https://arxiv.org/pdf/1706.03762.pdf) | [阅读笔记](https://zhuanlan.zhihu.com/p/250946855)：Transformer的开山之作，值得精读 | Ashish et al,2017\n2. [Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-Based Chatbots](https://arxiv.org/pdf/1612.01627v2.pdf) | [阅读笔记](https://zhuanlan.zhihu.com/p/270554147)：SMN检索式对话模型，多层多粒度提取信息 | Devlin et al,2018\n3. [Massive Exploration of Neural Machine Translation Architectures](https://arxiv.org/pdf/1703.03906.pdf) | [阅读笔记](https://zhuanlan.zhihu.com/p/328801239)：展示了以NMT架构超参数为例的首次大规模分析，实验为构建和扩展NMT体系结构带来了新颖的见解和实用建议。 | Denny et al,2017\n4. [Scheduled Sampling for Transformers](https://arxiv.org/pdf/1906.07651.pdf) | [阅读笔记](https://zhuanlan.zhihu.com/p/267146739)：在Transformer应用Scheduled Sampling | Mihaylova et al,2019\n\n# License\nLicensed under the Apache License, Version 2.0. Copyright 2021 DengBoCong. [Copy of the license]().","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdengbocong%2Fnlp-dialogue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdengbocong%2Fnlp-dialogue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdengbocong%2Fnlp-dialogue/lists"}