{"id":13595154,"url":"https://github.com/autoliuweijie/FastBERT","last_synced_at":"2025-04-09T10:33:02.681Z","repository":{"id":40628292,"uuid":"252982952","full_name":"autoliuweijie/FastBERT","owner":"autoliuweijie","description":"The score code of FastBERT (ACL2020)","archived":false,"fork":false,"pushed_at":"2021-10-29T08:01:40.000Z","size":2416,"stargazers_count":604,"open_issues_count":31,"forks_count":90,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-04-07T15:07:14.521Z","etag":null,"topics":["acl2020","bert","fastbert"],"latest_commit_sha":null,"homepage":"https://www.aclweb.org/anthology/2020.acl-main.537/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/autoliuweijie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-04T11:49:18.000Z","updated_at":"2025-04-02T15:00:50.000Z","dependencies_parsed_at":"2022-08-21T10:40:22.431Z","dependency_job_id":null,"html_url":"https://github.com/autoliuweijie/FastBERT","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autoliuweijie%2FFastBERT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autoliuweijie%2FFastBERT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autoliuweijie%2FFastBERT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autoliuweijie%2FFastBERT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/autoliuweijie","download_url":"https://codeload.github.com/autoliuweijie/FastBERT/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248020593,"owners_count":21034459,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acl2020","bert","fastbert"],"created_at":"2024-08-01T16:01:44.919Z","updated_at":"2025-04-09T10:32:57.670Z","avatar_url":"https://github.com/autoliuweijie.png","language":"Python","funding_links":[],"categories":["Python","BERT优化"],"sub_categories":["大语言对话模型及数据"],"readme":"# FastBERT\n![](https://img.shields.io/badge/license-MIT-000000.svg)\n\nSource code for [\"FastBERT: a Self-distilling BERT with Adaptive Inference Time\"](https://www.aclweb.org/anthology/2020.acl-main.537/).\n\n\n## Good News\n\n**2021/10/29 - Code**: Code of FastPLM is released on both [Pypi](https://pypi.org/project/fastplm/) and [Github](https://github.com/autoliuweijie/FastPLM).\n\n**2021/09/08 - Paper**: Journal version of FastBERT (FastPLM) is accepted by IEEE TNNLS. [\"An Empirical Study on Adaptive Inference for Pretrained Language Model\"](https://ieeexplore.ieee.org/document/9585316).\n\n**2020/07/05 - Update**: Pypi version of FastBERT has been launched. Please see [fastbert-pypi](https://pypi.org/project/fastbert/).\n\nInstall ``fastbert`` with ``pip``\n```sh\n$ pip install fastbert\n```\n\n\n## Requirements\n\n``python \u003e= 3.4.0``, Install all the requirements with ``pip``.\n``` \n$ pip install -r requirements.txt\n```\n\n\n## Quick start on the Chinese Book review dataset\n\nDownload the pre-trained Chinese BERT parameters from [here](https://share.weiyun.com/gHkb3N6L), and save it to the ``models`` directory with the name of \"Chinese_base_model.bin\". \n\nRun the following command to validate our FastBERT with ``Speed=0.5`` on the Book review datasets.\n```sh\n$ CUDA_VISIBLE_DEVICES=\"0\" python3 -u run_fastbert.py \\\n        --pretrained_model_path ./models/Chinese_base_model.bin \\\n        --vocab_path ./models/google_zh_vocab.txt \\\n        --train_path ./datasets/douban_book_review/train.tsv \\\n        --dev_path ./datasets/douban_book_review/dev.tsv \\\n        --test_path ./datasets/douban_book_review/test.tsv \\\n        --epochs_num 3 --batch_size 32 --distill_epochs_num 5 \\\n        --encoder bert --fast_mode --speed 0.5 \\\n        --output_model_path  ./models/douban_fastbert.bin\n```\n\nMeaning of each option.\n```\nusage: --pretrained_model_path Path to initialize model parameters.\n       --vocab_path Path to the vocabulary.\n       --train_path Path to the training dataset.\n       --dev_path Path to the validating dataset.\n       --test_path Path to the testing dataset.\n       --epochs_num The epoch numbers of fine-tuning.\n       --batch_size Batch size.\n       --distill_epochs_num The epoch numbers of the self-distillation.\n       --encoder The type of encoder.\n       --fast_mode Whether to enable the fast mode of FastBERT.\n       --speed The Speed value in the paper.\n       --output_model_path Path to the output model parameters.\n```\n\nTest results on the Book review dataset.\n```\nTest results at fine-tuning epoch 3 (Baseline): Acc.=0.8688;  FLOPs=21785247744;\nTest results at self-distillation epoch 1     : Acc.=0.8698;  FLOPs=6300902177;\nTest results at self-distillation epoch 2     : Acc.=0.8691;  FLOPs=5844839008;\nTest results at self-distillation epoch 3     : Acc.=0.8664;  FLOPs=5170940850;\nTest results at self-distillation epoch 4     : Acc.=0.8664;  FLOPs=5170940327;\nTest results at self-distillation epoch 5     : Acc.=0.8664;  FLOPs=5170940327;\n```\n\n\n## Quick start on the English Ag.news dataset\n\nDownload the pre-trained English BERT parameters from [here](https://share.weiyun.com/gHkb3N6L), and save it to the ``models`` directory with the name of \"English_uncased_base_model.bin\". \n\nDownload the ``ag_news.zip`` from [here](https://share.weiyun.com/ZctQJP8h), and then unzip it to the ``datasets`` directory. \n\nRun the following command to validate our FastBERT with ``Speed=0.5`` on the Ag.news datasets.\n```sh\n$ CUDA_VISIBLE_DEVICES=\"0\" python3 -u run_fastbert.py \\\n        --pretrained_model_path ./models/English_uncased_base_model.bin \\\n        --vocab_path ./models/google_uncased_en_vocab.txt \\\n        --train_path ./datasets/ag_news/train.tsv \\\n        --dev_path ./datasets/ag_news/test.tsv \\\n        --test_path ./datasets/ag_news/test.tsv \\\n        --epochs_num 3 --batch_size 32 --distill_epochs_num 5 \\\n        --encoder bert --fast_mode --speed 0.5 \\\n        --output_model_path  ./models/ag_news_fastbert.bin\n```\n\nTest results on the Ag.news dataset.\n```\nTest results at fine-tuning epoch 3 (Baseline): Acc.=0.9447;  FLOPs=21785247744;\nTest results at self-distillation epoch 1     : Acc.=0.9308;  FLOPs=2172009009;\nTest results at self-distillation epoch 2     : Acc.=0.9311;  FLOPs=2163471246;\nTest results at self-distillation epoch 3     : Acc.=0.9314;  FLOPs=2108341649;\nTest results at self-distillation epoch 4     : Acc.=0.9314;  FLOPs=2108341649;\nTest results at self-distillation epoch 5     : Acc.=0.9314;  FLOPs=2108341649;\n```\n\n\n## Datasets\n\nMore datasets can be downloaded from [here](https://share.weiyun.com/ZctQJP8h).\n\n\n## Other implementations\n\nThere are some other excellent implementations of FastBERT.\n\n* BitVoyage/FastBERT (Pytorch): https://github.com/BitVoyage/FastBERT\n\n\n## Acknowledgement\n\nThis work is funded by 2019 Tencent Rhino-Bird Elite Training Program. Work done while this author was an intern at Tencent.\n\nIf you use this code, please cite this paper:\n```\n@inproceedings{weijie2020fastbert,\n  title={{FastBERT}: a Self-distilling BERT with Adaptive Inference Time},\n  author={Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, Qi Ju},\n  booktitle={Proceedings of ACL 2020},\n  year={2020}\n}\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautoliuweijie%2FFastBERT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fautoliuweijie%2FFastBERT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautoliuweijie%2FFastBERT/lists"}