{"id":19119441,"url":"https://github.com/cluebenchmark/electra","last_synced_at":"2026-03-01T10:32:44.163Z","repository":{"id":110258592,"uuid":"234102290","full_name":"CLUEbenchmark/ELECTRA","owner":"CLUEbenchmark","description":"中文 预训练 ELECTRA 模型: 基于对抗学习 pretrain Chinese Model","archived":false,"fork":false,"pushed_at":"2020-03-22T01:01:43.000Z","size":978,"stargazers_count":139,"open_issues_count":4,"forks_count":11,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-02-22T12:29:53.796Z","etag":null,"topics":["adversarial-networks","albert","bert","electra","gan","language-model","pretrained-models"],"latest_commit_sha":null,"homepage":"https://openreview.net/forum?id=r1xMH1BtvB","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CLUEbenchmark.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-01-15T14:48:25.000Z","updated_at":"2025-01-17T12:51:28.000Z","dependencies_parsed_at":"2023-06-07T19:01:06.644Z","dependency_job_id":null,"html_url":"https://github.com/CLUEbenchmark/ELECTRA","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/CLUEbenchmark/ELECTRA","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CLUEbenchmark%2FELECTRA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CLUEbenchmark%2FELECTRA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CLUEbenchmark%2FELECTRA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CLUEbenchmark%2FELECTRA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CLUEbenchmark","download_url":"https://codeload.github.com/CLUEbenchmark/ELECTRA/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CLUEbenchmark%2FELECTRA/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29966826,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-01T09:33:09.965Z","status":"ssl_error","status_checked_at":"2026-03-01T09:25:48.915Z","response_time":124,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adversarial-networks","albert","bert","electra","gan","language-model","pretrained-models"],"created_at":"2024-11-09T05:09:41.339Z","updated_at":"2026-03-01T10:32:44.148Z","avatar_url":"https://github.com/CLUEbenchmark.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# ELECTRA\n\n中文 预训练 ELECTREA 模型: 基于对抗学习 pretrain Chinese Model\n\n\ncode Repost from google official code: https://github.com/google-research/electra\n\n具体使用说明：参考 官方链接\n\n# Electra Chinese tiny模型路径\n## google drive\nelectra-tiny \u003ca href='https://drive.google.com/file/d/1UP4byt4-kgenwST0KvyMYNbln6FfaSLp/view?usp=sharing'\u003egoogle-drive\u003c/a\u003e\n## baidu drive\nelectra-tiny \u003ca href='https://pan.baidu.com/s/14b-IiCkjRg-6XIYPXnezZA'\u003ebaidu-pan\u003c/a\u003e\ncode:rs99\n\n\n## 模型说明\n1. 与 tinyBERT 的 配置相同\n2. generator 为 discriminator的 1/4\n\n# How to use official code\n## Steps\n1. 修改 configure_pretraining.py 里面的 数据路径、tpu、gpu 配置\n2. 修改 model_size：可在 code/util/training_utils.py 里面 自行定义模型大小\n3. 数据输入格式：原始的input_ids, input_mask, segment_ids，训练过程中会在线 做 uniform mask sampling（不需要离线 生成 masked input ids）\n\n# Performance\n## gen+disc:\nelectra-tiny\n| metric | value | \n| --- | --- | \n| disc_accuracy | 0.95093095 | \n| disc_auc | 0.9762006 |\n| disc_loss | 0.14071295 |\n| disc_precision | 0.8018275 |\n| disc_recall | 0.6088053 |\n| loss | 9.516352 |\n| masked_lm_accuracy | 0.46732807 |\n| masked_lm_loss | 2.8209455 |\n| sampled_masked_lm_accuracy | 0.3504382 |\n\nThe model are trained on CLUE 10G Chinese Corpus with 1M-steps\n\n## Downstream finetuning on \u003ca href='https://www.cluebenchmarks.com/small_model_classification.html'\u003eCLUE benchmark\u003c/a\u003e:\n注：only use pretrained electra-tiny with layer-wise learning rate decay without any distilaltion、data-augmentation. learning rate is set to 1e-4 for each task and run 10-epochs. (According to official results, the results may have large variance)\n\n|     | AFQMC | TNEWS | IFLYTEK | CMNLI  | WSC  | CSL |\n| --- | ---   | ---   | ---     | ---    | ---  |---  |\n| Metrics | Acc | Acc | Acc | Acc  | Acc  | Acc |\n| ELECTRA-tiny | 70.319 | 54.280 | 53.538 |  73.745 | 64.336  | 78.700 | \n| Roberta-tiny | 69.904 | 54.150 | 56.808 |  74.037 | 64.336  | 74.133 |\n\n注：\n1. electra 在 多分类问题上面 可能会有 performance 下降\n2. gen、disc的规模 配比 比较hacky，与 mask的方法 等相关\n\n\n\u003ca href='https://www.cluebenchmarks.com/NLPCC.html'\u003e报名NLPCC-高性能小模型测评\u003c/a\u003e\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcluebenchmark%2Felectra","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcluebenchmark%2Felectra","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcluebenchmark%2Felectra/lists"}