{"id":13679000,"url":"https://github.com/IntelLabs/academic-budget-bert","last_synced_at":"2025-04-29T16:30:39.721Z","repository":{"id":38195922,"uuid":"357911256","full_name":"IntelLabs/academic-budget-bert","owner":"IntelLabs","description":"Repository containing code for \"How to Train BERT with an Academic Budget\" paper","archived":true,"fork":false,"pushed_at":"2023-09-18T20:15:51.000Z","size":100,"stargazers_count":309,"open_issues_count":11,"forks_count":47,"subscribers_count":16,"default_branch":"main","last_synced_at":"2024-11-11T21:38:09.558Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IntelLabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-14T13:15:32.000Z","updated_at":"2024-09-30T07:59:41.000Z","dependencies_parsed_at":"2024-08-02T13:17:38.883Z","dependency_job_id":"6f317449-34d4-4848-889b-e7b8385531e2","html_url":"https://github.com/IntelLabs/academic-budget-bert","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IntelLabs%2Facademic-budget-bert","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IntelLabs%2Facademic-budget-bert/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IntelLabs%2Facademic-budget-bert/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IntelLabs%2Facademic-budget-bert/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IntelLabs","download_url":"https://codeload.github.com/IntelLabs/academic-budget-bert/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251540029,"owners_count":21605833,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T13:01:00.684Z","updated_at":"2025-04-29T16:30:39.274Z","avatar_url":"https://github.com/IntelLabs.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# DISCONTINUATION OF PROJECT #  \nThis project will no longer be maintained by Intel.  \nThis project has been identified as having known security escapes.  \nIntel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.  \nIntel no longer accepts patches to this project.  \n\n\n# Training BERT with Compute/Time (Academic) Budget\n\nThis repository contains scripts for pre-training and finetuning BERT-like models with limited time and compute budget.\nThe code is based on the work presented in the following paper:\n\nPeter Izsak, Moshe Berchansky, Omer Levy, [How to Train BERT with an Academic Budget](https://aclanthology.org/2021.emnlp-main.831.pdf) (EMNLP 2021).\n\n## Installation\n\nThe pre-training and finetuning scripts are based on [Deepspeed](https://github.com/microsoft/DeepSpeed) and HuggingFace [Transformers](https://github.com/huggingface/transformers) libraries.\n\n### Preliminary Installation\n\nWe recommend creating a virtual environment with python 3.6+, PyTorch and [`apex`](https://github.com/NVIDIA/apex).\n\n### Installation Requirements\n```bash\npip install -r requirements.txt\n```\n\nWe suggest running Deepspeed's utility `ds_report` and verify Deepspeed components can be compiled (JIT).\n\n## Dataset\n\nThe [`dataset`](dataset/) directory includes scripts to pre-process the datasets we used in our experiments (Wikipedia, Bookcorpus). See dedicated [README](dataset/README.md) for full details.\n\n## Pretraining\n\nPretraining script: `run_pretraining.py`\n\nFor all possible pretraining arguments see: `python run_pretraining.py -h`\n\nWe highly suggest reviewing the various [training features](#time-based-training) we provide within the library.\n\n##### Example for training with the best configuration presented in our paper (24-layers/1024H/time-based learning rate schedule/fp16):\n\n```bash\ndeepspeed run_pretraining.py \\\n  --model_type bert-mlm --tokenizer_name bert-large-uncased \\\n  --hidden_act gelu \\\n  --hidden_size 1024 \\\n  --num_hidden_layers 24 \\\n  --num_attention_heads 16 \\\n  --intermediate_size 4096 \\\n  --hidden_dropout_prob 0.1 \\\n  --attention_probs_dropout_prob 0.1 \\\n  --encoder_ln_mode pre-ln \\\n  --lr 1e-3 \\\n  --train_batch_size 4096 \\\n  --train_micro_batch_size_per_gpu 32 \\\n  --lr_schedule time \\\n  --curve linear \\\n  --warmup_proportion 0.06 \\\n  --gradient_clipping 0.0 \\\n  --optimizer_type adamw \\\n  --weight_decay 0.01 \\\n  --adam_beta1 0.9 \\\n  --adam_beta2 0.98 \\\n  --adam_eps 1e-6 \\\n  --total_training_time 24.0 \\\n  --early_exit_time_marker 24.0 \\\n  --dataset_path \u003cdataset path\u003e \\\n  --output_dir /tmp/training-out \\\n  --print_steps 100 \\\n  --num_epochs_between_checkpoints 10000 \\\n  --job_name pretraining_experiment \\\n  --project_name budget-bert-pretraining \\\n  --validation_epochs 3 \\\n  --validation_epochs_begin 1 \\\n  --validation_epochs_end 1 \\\n  --validation_begin_proportion 0.05 \\\n  --validation_end_proportion 0.01 \\\n  --validation_micro_batch 16 \\\n  --deepspeed \\\n  --data_loader_type dist \\\n  --do_validation \\\n  --use_early_stopping \\\n  --early_stop_time 180 \\\n  --early_stop_eval_loss 6 \\\n  --seed 42 \\\n  --fp16\n```\n\n### Time-based Training\n\nPretraining can be limited to a time-based value by defining `--total_training_time=24.0` (24 hours for example).\n\n### Time-based Learning Rate Scheduling\n\nThe learning rate can be scheduled to change according to the configured total training time. The argument `--total_training_time` controls the total time assigned for the trainer to run, and must be specified in order to use time-based learning rate scheduling.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/lr_schedule.png\" alt=\"Time-based Learning rate schedule\" width=\"450\"/\u003e\n\u003c/p\u003e\n\nTo select time-based learning rate scheduling, define `--lr_schedule time`, and define a shape for for the annealing curve (`--curve=linear` for example, as seen in the figure). The warmup phase of the learning rate is define by specifying a proportion (`--warmup_proportion`) which accounts for the time-budget proportion available in the training session (as defined by `--total_training_time`). For example, for a 24 hour training session, `warmup_proportion=0.1` would account for 10% of 24 hours, that is, 2.4 hours (or 144 minutes) to reach peak learning rate. The learning rate will then be scheduled to reach 0 at the end of the time budget. We refer to the provided figure for an example.\n\n### Checkpoints and Finetune Checkpoints\n\nThere are 2 types of checkpoints that can be enabled:\n\n- Training checkpoint - saves model weights, optimizer state and training args. Defined by `--num_epochs_between_checkpoints`.\n- Finetuning checkpoint - saves model weights and configuration to be used for finetuning later on. Defined by `--finetune_time_markers`.\n\n`finetune_time_markers` can be assigned multiple points in the training time-budget by providing a list of time markers of the overall training progress. For example `--finetune_time_markers=0.5` will save a finetuning checkpoint when reaching 50% of training time budget. For multiple finetuning checkpoints, use commas without space `0.5,0.6,0.9`.\n\n### Validation Scheduling\n\nEnable validation while pre-training with `--do_validation`\n\nControl the number of epochs between validation runs with `--validation_epochs=\u003cnum\u003e`\n\nTo control the amount of validation runs in the beginning and end (running more that `validation_epochs`) use `validation_begin_proportion` and `validation_end_proportion` to specify the proportion of time and, `validation_epochs_begin` and `validation_epochs_end` to control the custom values accordingly. \n\n### Mixed Precision Training\n\nMixed precision is supported by adding `--fp16`. Use `--fp16_backend=ds` to use Deepspeed's mixed precision backend and `--fp16_backend=apex` for `apex` (`--fp16_opt` controls optimization level).\n\n## Finetuning\n\nUse `run_glue.py` to run finetuning for a saved checkpoint on GLUE tasks. \n\nThe finetuning script is identical to the one provided by Huggingface with the addition of our model.\n\nFor all possible pretraining arguments see: `python run_glue.py -h`\n\n##### Example for finetuning on MRPC:\n\n```bash\npython run_glue.py \\\n  --model_name_or_path \u003cpath to model\u003e \\\n  --task_name MRPC \\\n  --max_seq_length 128 \\\n  --output_dir /tmp/finetuning \\\n  --overwrite_output_dir \\\n  --do_train --do_eval \\\n  --evaluation_strategy steps \\\n  --per_device_train_batch_size 32 --gradient_accumulation_steps 1 \\\n  --per_device_eval_batch_size 32 \\\n  --learning_rate 5e-5 \\\n  --weight_decay 0.01 \\\n  --eval_steps 50 --evaluation_strategy steps \\\n  --max_grad_norm 1.0 \\\n  --num_train_epochs 5 \\\n  --lr_scheduler_type polynomial \\\n  --warmup_steps 50\n```\n\n\n## Generating Pretraining Commands\n\nWe provide a useful script for generating multiple (or single) pretraining commands by using `python generate_training_commands.py`.\n\n```bash\npython generate_training_commands.py -h\n\n\t--param_file PARAM_FILE Hyperparameter and configuration yaml\n  \t--job_name JOB_NAME   job name\n \t--init_cmd INIT_CMD   initialization command (deepspeed or python directly)\n```\n\nA parameter yaml must be defined with 2 main keys: `hyperparameters` with argument values defined as a list of possible values, and `default_parameters` as default values. Each generated command will be a possible combination of the various arguments specified in the `hyperparameters` section.\n\nExample:\n\n```yaml\nhyperparameters:\n  param1: [val1, val2]\n  param2: [val1, val2]\n\ndefault_parameters:\n  param3: 0.0\n```\n\nwill result in:\n\n```bash\ndeepspeed run_pretraining.py --param1=val1 --param2=val1 --param3=0.0\ndeepspeed run_pretraining.py --param1=val1 --param2=val2 --param3=0.0\ndeepspeed run_pretraining.py --param1=val2 --param2=val1 --param3=0.0\ndeepspeed run_pretraining.py --param1=val2 --param2=val2 --param3=0.0\n```\n\n## Citation\n\nIf you find this paper or this code useful, please cite this paper:\n\n```\n@inproceedings{izsak-etal-2021-train,\n    title = \"How to Train {BERT} with an Academic Budget\",\n    author = \"Izsak, Peter  and\n      Berchansky, Moshe  and\n      Levy, Omer\",\n    booktitle = \"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing\",\n    month = nov,\n    year = \"2021\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2021.emnlp-main.831\",\n}\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FIntelLabs%2Facademic-budget-bert","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FIntelLabs%2Facademic-budget-bert","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FIntelLabs%2Facademic-budget-bert/lists"}