{"id":13615352,"url":"https://github.com/ZhuiyiTechnology/pretrained-models","last_synced_at":"2025-04-13T21:30:39.722Z","repository":{"id":37339809,"uuid":"233508721","full_name":"ZhuiyiTechnology/pretrained-models","owner":"ZhuiyiTechnology","description":"Open Language Pre-trained Model Zoo","archived":false,"fork":false,"pushed_at":"2021-11-18T03:35:49.000Z","size":157,"stargazers_count":987,"open_issues_count":14,"forks_count":136,"subscribers_count":15,"default_branch":"master","last_synced_at":"2024-11-07T23:39:20.010Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ZhuiyiTechnology.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-01-13T04:07:56.000Z","updated_at":"2024-11-01T01:43:50.000Z","dependencies_parsed_at":"2022-07-19T00:32:11.983Z","dependency_job_id":null,"html_url":"https://github.com/ZhuiyiTechnology/pretrained-models","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZhuiyiTechnology%2Fpretrained-models","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZhuiyiTechnology%2Fpretrained-models/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZhuiyiTechnology%2Fpretrained-models/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZhuiyiTechnology%2Fpretrained-models/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ZhuiyiTechnology","download_url":"https://codeload.github.com/ZhuiyiTechnology/pretrained-models/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248785908,"owners_count":21161372,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T20:01:12.322Z","updated_at":"2025-04-13T21:30:39.438Z","avatar_url":"https://github.com/ZhuiyiTechnology.png","language":null,"funding_links":[],"categories":["Pretrained Language Model"],"sub_categories":["Repository"],"readme":"# 开源预训练语言模型合集\n\n这是由追一科技有限公司推出的一个预训练模型合集，主要发布自研的预训练语言模型，推动自然语言处理技术的进步。预训练语言模型通过在大规模文本上进行预训练，可以作为下游自然语言处理任务的模型参数或者模型输入以提高模型的整体性能。\n\n## 模型概览\n\n以下是我们目前公开发布的模型概览：\n\n| 名称           | 数据来源     | 训练数据大小 | 词表大小 | 模型大小 | 下载地址 |\n| :----------:  | :---------: | :---------:| :------: | :------: | :------: |\n| RoBERTa Tiny  | 百科,新闻 等  |     35G    | 21128    | 27MB | [下载链接](https://open.zhuiyi.ai/releases/nlp/models/zhuiyi/chinese_roberta_L-4_H-312_A-12.zip) |\n| RoBERTa Small | 百科,新闻 等  |     35G    | 21128  | 48MB  | [下载链接](https://open.zhuiyi.ai/releases/nlp/models/zhuiyi/chinese_roberta_L-6_H-384_A-12.zip) |\n| SimBERT Tiny  | [百度知道](http://zhidao.baidu.com/) | 2200万相似句组 | 13685  | 26MB  | [下载链接](https://open.zhuiyi.ai/releases/nlp/models/zhuiyi/chinese_simbert_L-4_H-312_A-12.zip) |\n| SimBERT Small  | [百度知道](http://zhidao.baidu.com/) | 2200万相似句组 | 13685  | 49MB  | [下载链接](https://open.zhuiyi.ai/releases/nlp/models/zhuiyi/chinese_simbert_L-6_H-384_A-12.zip) |\n| SimBERT Base  | [百度知道](http://zhidao.baidu.com/) | 2200万相似句组 | 13685  | 344MB  | [下载链接](https://open.zhuiyi.ai/releases/nlp/models/zhuiyi/chinese_simbert_L-12_H-768_A-12.zip) |\n| RoBERTa\u003csup\u003e+\u003c/sup\u003e Tiny  | 百科,新闻 等  |     35G    | 21128    | 35MB | [下载链接](https://open.zhuiyi.ai/releases/nlp/models/zhuiyi/chinese_roberta_L-4_H-312_A-12_K-104.zip) |\n| RoBERTa\u003csup\u003e+\u003c/sup\u003e Small | 百科,新闻 等  |     35G    | 21128  | 67MB  | [下载链接](https://open.zhuiyi.ai/releases/nlp/models/zhuiyi/chinese_roberta_L-6_H-384_A-12_K-128.zip) |\n| WoBERT | 百科,新闻 等  |     35G    | 33586/50000  | 400M  | [WoBERT项目](https://github.com/ZhuiyiTechnology/WoBERT) |\n| T5 PEGASUS | 百科,新闻 等  |     35G    | 50000  | 971M  | [T5 PEGASUS项目](https://github.com/ZhuiyiTechnology/t5-pegasus) |\n\n## 评估结果\n\n这里给出部分数据集上模型的评测结果。\n\n(注：以下实验结果均为重复跑三次后的平均值。预测阶段，两个small模型速度完全一致，两个tiny模型速度也完全一致。)\n\n### 文本情感分类\n\n任务来源：https://github.com/bojone/bert4keras/blob/master/examples/datasets/sentiment.zip\n\n评测脚本：\u003ca href=\"https://github.com/ZhuiyiTechnology/pretrained-models/blob/master/examples/task_sentiment.py\"\u003etask_sentiment.py\u003c/a\u003e\n\n评测指标：accuracy\n\n| 模型           | 验证集（valid） | 训练速度    | 第一个epoch结束时的指标值 |  测试集（test） |\n| :----------:  | :------------: | :---------:| :--------------------: | :------------: |\n| RoBERTa Small |     94.89%     |  38s/epoch |         90.48%         |     94.81%     |\n| ALBERT Small  |     94.57%     |  33s/epoch |         91.02%         |     94.52%     |\n| RoBERTa Tiny  |     94.46%     |  23s/epoch |         90.83%         |     94.00%     |\n| ALBERT Tiny   |     94.14%     |  20s/epoch |         90.18%         |     93.78%     |\n\n### IFLYTEK' 长文本分类\n\n任务来源：https://github.com/CLUEbenchmark/CLUE\n\n评测脚本：\u003ca href=\"https://github.com/ZhuiyiTechnology/pretrained-models/blob/master/examples/task_iflytek.py\"\u003etask_iflytek.py\u003c/a\u003e\n\n评测指标：accuracy\n\n| 模型           | 验证集（dev） | 训练速度    | 第一个epoch结束时的指标值 |\n| :----------:  | :---------: | :---------:| :--------------------: |\n| RoBERTa\u003csup\u003e+\u003c/sup\u003e Small |   57.93%    |  32s/epoch |         54.44%         |\n| RoBERTa Small |   57.66%    |  27s/epoch |         52.60%         |\n| ALBERT Small  |   57.14%    |  24s/epoch |         48.21%         |\n| RoBERTa\u003csup\u003e+\u003c/sup\u003e Tiny  |   57.51%    |  19s/epoch |         51.00%         |\n| RoBERTa Tiny  |   57.43%    |  16s/epoch |         49.76%         |\n| ALBERT Tiny   |   56.42%    |  14s/epoch |         43.84%         |\n\n### LIC2019-IE 信息抽取任务\n\n任务来源：http://lic2019.ccf.org.cn/kg\n\n评测脚本：\u003ca href=\"https://github.com/ZhuiyiTechnology/pretrained-models/blob/master/examples/task_lic2019_ie.py\"\u003etask_lic2019_ie.py\u003c/a\u003e\n\n评测指标：F1\n\n| 模型           | 验证集（dev） | 训练速度    | 第一个epoch结束时的指标值 |\n| :----------:  | :---------: | :---------:| :--------------------: |\n| RoBERTa\u003csup\u003e+\u003c/sup\u003e Small |   78.35%    |  446s/epoch |         61.92%        |\n| RoBERTa Small |   78.09%    |  375s/epoch |         63.85%        |\n| ALBERT Small  |   77.69%    |  335s/epoch |         46.58%        |\n| RoBERTa\u003csup\u003e+\u003c/sup\u003e Tiny  |   76.93%    |  264s/epoch |         45.80%        |\n| RoBERTa Tiny  |   76.65%    |  235s/epoch |         46.12%        |\n| ALBERT Tiny   |   75.94%    |  215s/epoch |         31.66%        |\n\n### CIPS-SogouQA 阅读理解式问答\n\n任务来源：http://task.www.sogou.com/cips-sogou_qa/\n\n评测脚本：\u003ca href=\"https://github.com/ZhuiyiTechnology/pretrained-models/blob/master/examples/task_cips_sogou_qa.py\"\u003etask_cips_sogou_qa.py\u003c/a\u003e\n\n评测指标：(EM + F1) / 2\n\n| 模型           | 验证集（dev） | 训练速度    | 第一个epoch结束时的指标值 |\n| :----------:  | :---------: | :---------:| :--------------------: |\n| RoBERTa\u003csup\u003e+\u003c/sup\u003e Small |   71.27%    |  673s/epoch |         56.09%        |\n| RoBERTa Small |   70.35%    |  607s/epoch |         61.07%        |\n| ALBERT Small  |   66.66%    |  582s/epoch |         50.93%        |\n| RoBERTa\u003csup\u003e+\u003c/sup\u003e Tiny  |   68.37%    |  481s/epoch |         52.59%        |\n| RoBERTa Tiny  |   67.85%    |  455s/epoch |         49.78%        |\n| ALBERT Tiny   |   63.41%    |  443s/epoch |         37.47%        |\n\n（注：此处是直接使用UniLM的Seq2Seq方案来做阅读理解，主要测试模型用做文本生成时的能力。但要说明的是，Seq2Seq并非做阅读理解的标准方案。）\n\n## 模型详情\n\n此处对每个模型进行较为详细的介绍\n\n### RoBERTa Tiny\n\n- \u003cstrong\u003e【配置】\u003c/strong\u003e 4层模型，hidden size为312，对Embedding层做了低秩分解(312-\u003e128-\u003e312)，可以用\u003ca href=\"https://github.com/bojone/bert4keras/tree/master/examples\"\u003ebert4keras\u003c/a\u003e加载使用。\n\n- \u003cstrong\u003e【训练】\u003c/strong\u003e 使用\u003ca href=\"https://github.com/bojone/bert4keras/tree/master/pretraining\"\u003ebert4keras\u003c/a\u003e在TPU v3-8上训练，使用带梯度累积的LAMB优化器，批大小为800，累积4步更新，相当于以批大小3200训练了125k步（前3125步为warmup）。\n\n- \u003cstrong\u003e【备注】\u003c/strong\u003e 速度跟\u003ca href=\"https://github.com/brightmart/albert_zh\"\u003ealbert tiny\u003c/a\u003e一致，普通分类性能也基本一致，但由于roberta模型并没有参数共享这个约束，所以在生成式任务等复杂任务上效果优于albert tiny。\n\n### RoBERTa Small\n\n- \u003cstrong\u003e【配置】\u003c/strong\u003e 6层模型，hidden size为384，对Embedding层做了低秩分解(384-\u003e128-\u003e384)，可以用\u003ca href=\"https://github.com/bojone/bert4keras/tree/master/examples\"\u003ebert4keras\u003c/a\u003e加载使用。\n\n- \u003cstrong\u003e【训练】\u003c/strong\u003e 使用\u003ca href=\"https://github.com/bojone/bert4keras/tree/master/pretraining\"\u003ebert4keras\u003c/a\u003e在TPU v3-8上训练，使用带梯度累积的LAMB优化器，批大小为800，累积4步更新，相当于以批大小3200训练了125k步（前3125步为warmup）。\n\n- \u003cstrong\u003e【备注】\u003c/strong\u003e 速度跟\u003ca href=\"https://github.com/brightmart/albert_zh\"\u003ealbert small\u003c/a\u003e一致，普通分类性能也基本一致，但由于roberta模型并没有参数共享这个约束，所以在生成式任务等复杂任务上效果优于albert small。\n\n### SimBERT Base\n\n- \u003cstrong\u003e【配置】\u003c/strong\u003e 跟bert base一致，12层模型，hidden size为768。\n\n- \u003cstrong\u003e【训练】\u003c/strong\u003e 使用\u003ca href=\"https://github.com/bojone/bert4keras/tree/master/pretraining\"\u003ebert4keras\u003c/a\u003e基于\u003ca href=\"https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip\"\u003echinese_L-12_H-768_A-12\u003c/a\u003e进行继续训练，训练任务为“度量学习+UniLM”，以Adam优化器（学习率2e-6，批大小128）的Adam优化器在单个TITAN RTX上训练了117万步。\n\n- \u003cstrong\u003e【备注】\u003c/strong\u003e 简单来说，这就是一个finetune过的bert base模型，但是[CLS]对应的输出具有句向量的意义，可以用于检索任务，理论上短文本效果会更好，在跟\u003ca href=\"https://kexue.fm/archives/5743\"\u003e这里\u003c/a\u003e一样的验证集上得到了0.96的top1准确率；此外还具有一对多生成能力。详见例子\u003ca href=\"examples/simbert_base.py\"\u003esimbert_base.py\u003c/a\u003e。\n\n### RoBERTa\u003csup\u003e+\u003c/sup\u003e Tiny\n\n- \u003cstrong\u003e【配置】\u003c/strong\u003e 4层模型，hidden size为312，key_size增大为104（原来的4倍），对Embedding层做了低秩分解(312-\u003e128-\u003e312)，可以用\u003ca href=\"https://github.com/bojone/bert4keras/tree/master/examples\"\u003ebert4keras\u003c/a\u003e加载使用。\n\n- \u003cstrong\u003e【训练】\u003c/strong\u003e 使用\u003ca href=\"https://github.com/bojone/bert4keras/tree/master/pretraining\"\u003ebert4keras\u003c/a\u003e在TPU v3-8上训练，使用带梯度累积的LAMB优化器，批大小为800，累积4步更新，相当于以批大小3200训练了125k步（前3125步为warmup）。\n\n- \u003cstrong\u003e【备注】\u003c/strong\u003e 速度比RoBERTa Tiny略慢，但整体效果更优。\n\n### RoBERTa\u003csup\u003e+\u003c/sup\u003e Small\n\n- \u003cstrong\u003e【配置】\u003c/strong\u003e 6层模型，hidden size为384，key_size增大为128（原来的4倍），对Embedding层做了低秩分解(384-\u003e128-\u003e384)，可以用\u003ca href=\"https://github.com/bojone/bert4keras/tree/master/examples\"\u003ebert4keras\u003c/a\u003e加载使用。\n\n- \u003cstrong\u003e【训练】\u003c/strong\u003e 使用\u003ca href=\"https://github.com/bojone/bert4keras/tree/master/pretraining\"\u003ebert4keras\u003c/a\u003e在TPU v3-8上训练，使用带梯度累积的LAMB优化器，批大小为800，累积4步更新，相当于以批大小3200训练了125k步（前3125步为warmup）。\n\n- \u003cstrong\u003e【备注】\u003c/strong\u003e 速度比RoBERTa Small略慢，但整体效果更优。\n\n## 如何引用\n\nBibtex：\n\n```tex\n@techreport{zhuiyipretrainedmodels,\n  title={Open Language Pre-trained Model Zoo - ZhuiyiAI},\n  author={Jianlin Su},\n  year={2020},\n  url=\"https://github.com/ZhuiyiTechnology/pretrained-models\",\n}\n```\n\n## 致谢信息\n本项目部分受到[**谷歌TensorFlow Research Cloud**](https://www.tensorflow.org/tfrc)计划资助，在此特别致谢。\n\n## 联系我们\n\n邮箱：ai@wezhuiyi.com\n\n## 相关链接\n\n追一科技：https://zhuiyi.ai\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FZhuiyiTechnology%2Fpretrained-models","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FZhuiyiTechnology%2Fpretrained-models","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FZhuiyiTechnology%2Fpretrained-models/lists"}