{"id":13535055,"url":"https://github.com/qiangsiwei/bert_distill","last_synced_at":"2025-04-02T00:32:13.476Z","repository":{"id":45507066,"uuid":"202273145","full_name":"qiangsiwei/bert_distill","owner":"qiangsiwei","description":"BERT distillation（基于BERT的蒸馏实验 ）","archived":false,"fork":false,"pushed_at":"2020-07-30T02:21:43.000Z","size":30324,"stargazers_count":309,"open_issues_count":7,"forks_count":87,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-11-02T23:32:07.638Z","etag":null,"topics":["bert","classification","distillation","nlp"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/qiangsiwei.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-14T04:13:27.000Z","updated_at":"2024-10-08T07:54:05.000Z","dependencies_parsed_at":"2022-07-15T10:30:34.915Z","dependency_job_id":null,"html_url":"https://github.com/qiangsiwei/bert_distill","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qiangsiwei%2Fbert_distill","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qiangsiwei%2Fbert_distill/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qiangsiwei%2Fbert_distill/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qiangsiwei%2Fbert_distill/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/qiangsiwei","download_url":"https://codeload.github.com/qiangsiwei/bert_distill/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246735238,"owners_count":20825217,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","classification","distillation","nlp"],"created_at":"2024-08-01T08:00:49.148Z","updated_at":"2025-04-02T00:32:08.443Z","avatar_url":"https://github.com/qiangsiwei.png","language":"Python","funding_links":[],"categories":["BERT Deploy Tricks:"],"sub_categories":[],"readme":"基于BERT的蒸馏实验\n================\n\n参考论文《Distilling Task-Specific Knowledge from BERT into Simple Neural Networks》\n\n分别采用keras和pytorch基于textcnn和bilstm(gru)进行了实验\n\n实验数据分割成 1（有标签训练）：8（无标签训练）：1（测试）\n\n在情感2分类clothing的数据集上初步结果如下：\n\n - 小模型（textcnn \u0026 bilstm）准确率在 0.80 ~ 0.81\n\n - BERT模型 准确率在 0.90 ~ 0.91\n\n - 蒸馏模型 准确率在 0.87 ~ 0.88\n\n实验结果与论文结论基本一致，与预期相符\n\n后续将尝试其他更有效的蒸馏方案\n\n## 使用方法\n\n首先finetune BERT\n```bash\npython ptbert.py\n```\n\n然后把BERT的知识蒸馏到小模型里\n\n需要先解压`data/cache/word2vec.gz`\n\n然后\n```bash\npython distill.py\n```\n\n调整文件中的`use_aug`及以下的参数可以使用论文中提到的其中两种数据增强方式(masking, n-gram sampling)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqiangsiwei%2Fbert_distill","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqiangsiwei%2Fbert_distill","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqiangsiwei%2Fbert_distill/lists"}