{"id":13436467,"url":"https://github.com/ZhuiyiTechnology/WoBERT","last_synced_at":"2025-03-18T21:30:35.939Z","repository":{"id":41038350,"uuid":"296283584","full_name":"ZhuiyiTechnology/WoBERT","owner":"ZhuiyiTechnology","description":"以词为基本单位的中文BERT","archived":false,"fork":false,"pushed_at":"2021-11-18T03:38:20.000Z","size":59,"stargazers_count":457,"open_issues_count":15,"forks_count":69,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-10-27T20:19:50.164Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ZhuiyiTechnology.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-09-17T09:42:09.000Z","updated_at":"2024-10-24T00:19:00.000Z","dependencies_parsed_at":"2022-08-10T01:34:50.983Z","dependency_job_id":null,"html_url":"https://github.com/ZhuiyiTechnology/WoBERT","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZhuiyiTechnology%2FWoBERT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZhuiyiTechnology%2FWoBERT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZhuiyiTechnology%2FWoBERT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZhuiyiTechnology%2FWoBERT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ZhuiyiTechnology","download_url":"https://codeload.github.com/ZhuiyiTechnology/WoBERT/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244310317,"owners_count":20432514,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T03:00:48.760Z","updated_at":"2025-03-18T21:30:34.522Z","avatar_url":"https://github.com/ZhuiyiTechnology.png","language":"Python","funding_links":[],"categories":["Pretrained Language Model","Python","BERT优化","Uncategorized"],"sub_categories":["Repository","大语言对话模型及数据","Uncategorized"],"readme":"# WoBERT\n以词为基本单位的中文BERT（Word-based BERT）\n\n## 详情\n\nhttps://kexue.fm/archives/7758\n\n## 训练\n\n目前开源的WoBERT是Base版本，在哈工大开源的[RoBERTa-wwm-ext](https://github.com/ymcui/Chinese-BERT-wwm)基础上进行继续预训练，预训练任务为MLM。初始化阶段，将每个词用BERT自带的Tokenizer切分为字，然后用字embedding的平均作为词embedding的初始化。模型使用单张24G的RTX训练了100万步（大概训练了10天），序列长度为512，学习率为5e-6，batch_size为16，累积梯度16步，相当于batch_size=256训练了6万步左右。训练语料大概是30多G的通用型语料。\n\n此外，我们还提供了WoNEZHA，这是基于华为开源的[NEZHA](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/NEZHA-TensorFlow)进行再预训练的，训练细节跟WoBERT基本一样。NEZHA的模型结构跟BERT相似，不同的是它使用了相对位置编码，而BERT用的是绝对位置编码，因此理论上NEZHA能处理的文本长度是无上限的。这里提供以词为单位的WoNEZHA，就是让大家多一个选择。\n\n**2021年03月03日：**  新增WoBERT Plus模型，以RoBERTa-wwm-ext为基础，中文MLM式预训练，重新构建词表（比已经开源的WoBERT更完善），30+G语料，maxlen=512，batch_size=256、lr=1e-5训练了25万步（4 * TITAN RTX，累积4步梯度，是之前的WoBERT的4倍），每1000步耗时约1580s，共训练了18天，训练acc约64%，训练loss约1.80。\n\n## 依赖\n```bash\npip install bert4keras==0.8.8\n```\n\n## 下载\n\n- **WoBERT**: [chinese_wobert_L-12_H-768_A-12.zip](https://open.zhuiyi.ai/releases/nlp/models/zhuiyi/chinese_wobert_L-12_H-768_A-12.zip)\n- **WoNEZHA**: [chinese_wonezha_L-12_H-768_A-12.zip](https://open.zhuiyi.ai/releases/nlp/models/zhuiyi/chinese_wonezha_L-12_H-768_A-12.zip)\n- **WoBERT\u003csup\u003e+\u003c/sup\u003e**: [chinese_wobert_plus_L-12_H-768_A-12.zip](https://open.zhuiyi.ai/releases/nlp/models/zhuiyi/chinese_wobert_plus_L-12_H-768_A-12.zip)\n\n## 评测\n\n\n| | IFLYTEK | TNEWS |\n|:--:|:--:|:--:|\n| BERT | 60.31 | 56.94 | \n| WoBERT | 61.15 | 57.05 |\n| WoBERT Plus | **61.92** | **58.20** |\n\n## 引用\n\nBibtex：\n\n```tex\n@techreport{zhuiyiwobert,\n  title={WoBERT: Word-based Chinese BERT model - ZhuiyiAI},\n  author={Jianlin Su},\n  year={2020},\n  url=\"https://github.com/ZhuiyiTechnology/WoBERT\",\n}\n```\n\n## 联系\n\n邮箱：ai@wezhuiyi.com\n追一科技：https://zhuiyi.ai\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FZhuiyiTechnology%2FWoBERT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FZhuiyiTechnology%2FWoBERT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FZhuiyiTechnology%2FWoBERT/lists"}