{"id":13754266,"url":"https://github.com/xv44586/ccf_2020_qa_match","last_synced_at":"2025-09-13T01:09:34.575Z","repository":{"id":57845745,"uuid":"309606014","full_name":"xv44586/ccf_2020_qa_match","owner":"xv44586","description":"ccf 2020 qa match competition top1","archived":false,"fork":false,"pushed_at":"2021-01-28T06:46:29.000Z","size":2144,"stargazers_count":266,"open_issues_count":4,"forks_count":83,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-09T22:16:17.337Z","etag":null,"topics":["bert","ccf","keras","top1"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xv44586.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-03T07:23:50.000Z","updated_at":"2024-12-18T09:29:52.000Z","dependencies_parsed_at":"2022-09-08T04:21:03.268Z","dependency_job_id":null,"html_url":"https://github.com/xv44586/ccf_2020_qa_match","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xv44586%2Fccf_2020_qa_match","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xv44586%2Fccf_2020_qa_match/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xv44586%2Fccf_2020_qa_match/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xv44586%2Fccf_2020_qa_match/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xv44586","download_url":"https://codeload.github.com/xv44586/ccf_2020_qa_match/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248119287,"owners_count":21050755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","ccf","keras","top1"],"created_at":"2024-08-03T09:01:52.593Z","updated_at":"2025-04-09T22:16:22.775Z","avatar_url":"https://github.com/xv44586.png","language":"Python","funding_links":[],"categories":["机器阅读理解"],"sub_categories":["其他_文本生成、文本对话"],"readme":"# Update\n基于当前repo 优化后，A/B 榜皆是Top1，~~代码整理中，后续会陆续放上来！~~  \n\n\u003cstrong\u003e总结博客：\u003c/strong\u003e[ccf问答匹配比赛（下）：如何只用“bert”夺冠](https://xv44586.github.io/2021/01/20/ccf-qa-2/)\n\n# 优化思路\n## Post training\n### mlm\n提升mlm任务中的mask策略，提升难度，提高下游性能：挖掘新词，加入词典，whole word mask + dynamic mask\n* 挖掘新词\n```bash\npython new_words_mining.py \n```\n### nsp\n句子级别的任务是有用的，不过替换为SOP/AOP: query-answer pair时互换位置(sop)，query-answer-list时，只打乱answer-list的顺序（aop)\n\n### model-adaptive\npost training的样本格式与下游一致，也能带来提升（区别RoBERTa 中的结论）\n\n完整post training代码为两份：query-answer pair 与 query-answerA-list两种方式：\n```bash\npython popint-post-training-wwm-sop.py\npython pair-post-training-wwm-sop.py\n```\n\nPS: post training 后，bert 后接复杂分类层（CNN/RNN/DGCNN/...)基本不会带来提升  \n![post training result](./img/post-training.png)\n\n## 融入知识\n融入知识主要两种方式：bert 的Embedding层融入与transformer output层融入:\n* embedding层融合  \n![external-embedding-bottom](./img/bottom-embedding.png)\n* transformer output 层融合  \n![top-embedding](./img/top-embedding.png)  \n\n融入的知识使用的gensim 训练的word2vec(dims=100)，不过两种方式多次实验后都没带来提升：\n```bash\npython pair-external-embedding.py\n```\n如何切换融入的方式，请查看代码后自行修改\n\n## 对比学习\n引入对比学习尝试提高模型性能，对比学习主要有两种方式：自监督对比学习与监督对比学习：\n* 自监督对比学习  \n通过互换QA位置，并随机mask 10%的token来构建一对view，view之间互为正例：\n* loss  \n![自监督对比学习loss](./img/ssc-loss.png)\n* model  \n![自监督对比学习模型](./img/ssc.png)\n\n* 监督对比学习  \n将相同label的样本视为互为正例：\n* loss  \n![监督对比学习loss](./img/sc-loss.png)\n* model  \n![监督对比学习模型](./img/sc.png)\n\n执行自监督对比代码：\n```bash\npython pair-data-augment-contrstive-learning.py \n```\n执行监督对比学习代码：\n```bash\npython pair-supervised-contrastive-learning.py\n```\n\n## 自蒸馏\n自蒸馏即Teacher 与 Student 为同一个模型，Teacher训练一次后，在train data上打上soften labels，然后迁移至Student 模型。\n```bash\npython pair-self-kd.py\n```\n\n## 对抗训练\n使用FGM方法对EMbedding进行扰动：\n```bash\npython pair-adversarial-train.py\n```\n\n## 数据增强\n数据增强主要尝试了两种方式：EDA与伪标签。\n* EDA\n随机删除/随机替换/随机插入/随机重复，操作比例10%，每个样本生成4个新样本\n词向量质量低，所以使用从当前句子随机选取一个词作为同义词进行操作\n\n* 伪标签\n用已训练的模型对test data打上标签加入训练集\n\nTips：\n数据增强时用已训练模型进行过滤，将低置信度（\u003c0.7)的样本过滤掉，避免引入错误标签样本；此外，伪标签时，要结合数据比例，过多的测试数据提前进入训练集，最终的结果只会与“伪标签”一致，反而无法带来提升。\n\n## shuffle\n在query-answer-list 样本格式下，解码时对answer-list进行全排列，然后投票。不过此次比赛的数据顺序很重要，乱序后结果较差，没带来提升\n\n\n# 总结\n![](./img/summary.png)\n\n**-----------------------------------2020.01.18------------------------------------------------------------------**\n\n# 比赛\n贝壳找房-房产行业聊天问答匹配， 比赛地址[https://www.datafountain.cn/competitions/474/datasets](https://www.datafountain.cn/competitions/474/datasets)\n\n总结博客：[ccf问答匹配](https://xv44586.github.io/2020/11/08/ccf-qa/)\n\n# 简单说明\n样本为一个问题多个回答，其中回答有的是针对问题的回答（1），也有不是的（0），其中回答是按顺序排列的。即：\nquery1: [(answer1, 0), (answer2, 1),...]\n任务是对每个回答进行分类，判断是不是针对问题的回答。\n\n# pretrain model weights\n预训练模型使用的是华为开源的[nezha-base-wwm](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/NEZHA-TensorFlow)\n\n# Baseline\n## 思路一：\n不考虑回答之间的顺序关系，将其拆为query-answer 对，然后进行判断。\n比如现在的样本是: {query: \"房子几年了\", answers: [(\"二年了\", 1), (\"楼层靠中间\"， 0)]},此时我们将其拆分为单个query-answer pair，即：\n[{query: \"房子几年了\", answer: \"二年了\", label: 1}, {query: \"房子几年了\", answer: \"楼层靠中间\", label: 0}]\n\n![pair match](./img/pair.png)\n\n代码实现：[pair_match](https://github.com/xv44586/ccf_2020_qa_match/ccf_2020_qa_match_pair.py)\n\n单模型提交f1: 0.752\n\n## 思路二：\n考虑对话连贯性，同时考虑其完整性，将所有回答顺序拼接后再与问题拼接，组成query-answer1-answer2，然后针对每句回答进行分类。\n上面的例子将被组成样本：{query: \"房子几年了\", answer: \"两年了[SEP]楼层靠中间[SEP]\", label: [mask, mask, mask, 0, mask, mask, mask,mask,mask, 0]}\n即：将每句回答后面的[SEP] 作为最终的特征向量，然后去做二分类。\n\n![](./img/point.png)\n\n代码实现：[match_point](https://github.com/xv44586/ccf_2020_qa_match/ccf_2020_qa_match_point.py)\n\n单模型提交f1: 0.75\n\n## 思路三：\nPattern-Exploiting Training(PET)，即增加一个pattern，将任务转换为MLM任务，然后通过pattern的得分来判断对应的类别。\n如本次样本可以添加一个前缀pattern：\"简接回答问题\"/\"直接回答问题\"，分别对应label 0/1,pattern的得分只需看第一个位置中\"间\"/\"直\" 两个token的概率谁高即可。\n此外，训练时还可以借助bert的预训练任务中的mlm任务增强模型的泛化能力。更详细的请介绍请查阅[文本分类秒解](https://xv44586.github.io/2020/10/25/pet/)\n\n对于本次样本，对应的示意图如下：\n\n![](./img/pet.png)\n\n对应代码实现：[pet classification](https://github.com/xv44586/ccf_2020_qa_match/ccf_2020_qa_match_pet.py)\n\n单模型提交f1: 0.76+\n\n# 思路四\n由于bert 不同的transformer 层提取到的语义粒度不同，而不同粒度的信息对分类来说起到的作用也不同，所以可以concat所以粒度的语义信息，拼接后作为特征进行分类。\n\n对应于本次样本，示意图如下：\n![](./img/concat.png)\n\n对应代码实现：[concat classification](https://github.com/xv44586/ccf_2020_qa_match/ccf_2020_qa_match_concat.py)\n单模型提交f1: 0.75+\n\n# tips\n贴几篇感觉有启发的关于文本分类的论文\n\n* [Universal Language Model Fine-tuning for Text Classification](http://arxiv.org/abs/1801.06146)\n* [How to Fine-Tune BERT for Text Classification?](http://arxiv.org/abs/1905.05583)\n* [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](http://arxiv.org/abs/2004.10964)\n* [Enriching BERT with Knowledge Graph Embeddings for Document Classification](http://arxiv.org/abs/1909.08402)\n* [Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT model](https://arxiv.org/pdf/2008.06460.pdf)\n* [Contrastive Self-Supervised Learning](http://ankeshanand.com/blog/2020/01/26/contrative-self-supervised-learning.html)\n* [A Survey on Contrastive Self-supervised Learning](http://arxiv.org/abs/2011.00362)\n* [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](http://arxiv.org/abs/2011.01403)\n* [Self-Attention with Relative Position Representations](http://arxiv.org/abs/1803.02155)\n* [RoBERTa: A Robustly Optimized BERT Pretraining Approach](http://arxiv.org/abs/1907.11692)\n* [NEZHA: Neural Contextualized Representation for Chinese Language Understanding](http://arxiv.org/abs/1909.00204)\n* [对抗训练浅谈：意义、方法和思考（附Keras实现）](https://kexue.fm/archives/7234)\n* [Train No Evil: Selective Masking for Task-Guided Pre-Training](https://arxiv.org/abs/2004.09733)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxv44586%2Fccf_2020_qa_match","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxv44586%2Fccf_2020_qa_match","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxv44586%2Fccf_2020_qa_match/lists"}