{"id":17141054,"url":"https://github.com/zphang/bert_on_stilts","last_synced_at":"2025-04-13T09:32:14.545Z","repository":{"id":116145664,"uuid":"174241814","full_name":"zphang/bert_on_stilts","owner":"zphang","description":"Fork of huggingface/pytorch-pretrained-BERT for BERT on STILTs","archived":false,"fork":false,"pushed_at":"2022-11-19T08:53:45.000Z","size":1147,"stargazers_count":106,"open_issues_count":5,"forks_count":30,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-27T01:46:09.381Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zphang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-03-07T00:25:11.000Z","updated_at":"2024-05-07T13:21:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"896ad70e-8313-4013-b416-a5a45663b3aa","html_url":"https://github.com/zphang/bert_on_stilts","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zphang%2Fbert_on_stilts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zphang%2Fbert_on_stilts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zphang%2Fbert_on_stilts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zphang%2Fbert_on_stilts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zphang","download_url":"https://codeload.github.com/zphang/bert_on_stilts/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248690874,"owners_count":21146221,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T20:24:13.942Z","updated_at":"2025-04-13T09:32:13.894Z","avatar_url":"https://github.com/zphang.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BERT on STILTs\n\n*STILTs = **S**upplementary **T**raining on **I**ntermediate **L**abeled-data **T**asks*\n\nThis repository contains code for [BERT on STILTs](https://arxiv.org/abs/1811.01088v2). It is a fork of the [Hugging Face implementation of BERT](https://github.com/huggingface/pytorch-pretrained-BERT).\n\nSTILTs is a method for supplementary training on an intermediate task before fine-tuning for a downstream target task. We show in [our paper](https://arxiv.org/abs/1811.01088v2) that STILTs can improve the performance and stability of the final model on the target task.\n\n**BERT on STILTs** achieves a [GLUE score](https://gluebenchmark.com/leaderboard) of 82.0, compared to 80.5 of BERT without STILTs.\n\n## Trained Models\n\n| Base Model | Intermediate Task | Target Task | Download | Val Score | Test Score |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| BERT-Large   | -        | **CoLA**   | [Link](https://drive.google.com/file/d/1bYuvIrnYjI-22xd6koYdDlkgLMtN6Uey/view?usp=sharing) | 65.3 | 61.2 |\n| BERT-Large   | **MNLI** | **SST**    | [Link](https://drive.google.com/file/d/1M0ubTzGO4oNC7szc6bRxMIf81iTWgAPL/view?usp=sharing) | 93.9 | 95.1 |\n| BERT-Large   | **MNLI** | **MRPC**   | [Link](https://drive.google.com/file/d/1b0FdK-95yLk_P2ro009opSRX6GgwegGB/view?usp=sharing) | 90.4 | 88.6 |\n| BERT-Large   | **MNLI** | **STS-B**  | [Link](https://drive.google.com/file/d/1VWZbqFvM2myLoE2-uVh-KtAUmhgS9anb/view?usp=sharing) | 90.7 | 89.0 |\n| BERT-Large   | -        | **QQP**    | [Link](https://drive.google.com/file/d/1d5KMckz2txwYtE_wGL6g8591nGFw9Vid/view?usp=sharing) | 90.0 | 81.2 |\n| BERT-Large   | -        | **MNLI**   | [Link](https://drive.google.com/file/d/1na4cULKs5qe9odhF0qA-x4H2ZZKXNl7N/view?usp=sharing) | 86.7 | 86.2 | \n| BERT-Large   | **MNLI** | **QNLI**   | [Link](https://drive.google.com/file/d/1cHehR1PXxQ38UrKBdzwUCykZcKoIUeCv/view?usp=sharing) | 92.3 | 92.8 |\n| BERT-Large   | **MNLI** | **RTE**    | [Link](https://drive.google.com/file/d/1YIYiqcBTXRCMh8gvKnGCO0mXuhR6PnKF/view?usp=sharing) | 84.1 | 79.0 |\n| BERT-Large   | -        | WNLI*     | N/A | 56.3 | 65.1 |\n\nOverall GLUE Score: **82.0**\n \n*Models differ slightly from published results because they were retrained.*\n\n## Example usage\n\n#### Preparation\n\nYou will need to download the GLUE data to run our tasks. See [here](https://github.com/jsalt18-sentence-repl/jiant#downloading-data).\n\nYou will also need to set the two following environment variables:\n\n* `GLUE_DIR`: This should point to the location of the GLUE data downloaded from `jiant`.\n* `BERT_ALL_DIR`: Set `BERT_ALL_DIR=/PATH_TO_THIS_REPO/cache/bert_metadata` \n    * For mor general use: `BERT_ALL_DIR` should point to the location of BERT downloaded from [here](https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-24_H-1024_A-16.zip). Importantly, the `BERT_ALL_DIR` needs to contain the files `uncased_L-24_H-1024_A-16/bert_config.json` and `uncased_L-24_H-1024_A-16/vocab.txt`.\n\n##### Example 1: Generating Predictions\n\nTo generate validation/test predictions, as well as validation metrics, run something like the following:\n\n```bash\nexport TASK=rte\nexport BERT_LOAD_PATH=path/to/mnli__rte.p\nexport OUTPUT_PATH=rte_output\n\npython glue/train.py \\\n    --task_name $TASK \\\n    --do_val --do_test \\\n    --do_lower_case \\\n    --bert_model bert-large-uncased \\\n    --bert_load_mode full_model_only \\\n    --bert_load_path $BERT_LOAD_PATH \\\n    --eval_batch_size 64 \\\n    --output_dir $OUTPUT_PATH\n``` \n\n##### Example 2: Fine-tuning from vanilla BERT\n\nWe recommend training with a batch size of 16/24/32.\n\n```bash\nexport TASK=mnli\nexport OUTPUT_PATH=mnli_output\n\npython glue/train.py \\\n    --task_name $TASK \\\n    --do_train --do_val --do_test --do_val_history \\\n    --do_save \\\n    --do_lower_case \\\n    --bert_model bert-large-uncased \\\n    --bert_load_mode from_pretrained \\\n    --bert_save_mode model_all \\\n    --train_batch_size 24 \\\n    --learning_rate 2e-5 \\\n    --output_dir $OUTPUT_PATH\n``` \n\n\n##### Example 3: Fine-tuning from MNLI model\n\n```bash\nexport PRETRAINED_MODEL_PATH=/path/to/mnli.p\nexport TASK=rte\nexport OUTPUT_PATH=rte_output\n\npython glue/train.py \\\n    --task_name $TASK \\\n    --do_train --do_val --do_test --do_val_history \\\n    --do_save \\\n    --do_lower_case \\\n    --bert_model bert-large-uncased \\\n    --bert_load_path $PRETRAINED_MODEL_PATH \\\n    --bert_load_mode model_only \\\n    --bert_save_mode model_all \\\n    --train_batch_size 24 \\\n    --learning_rate 2e-5 \\\n    --output_dir $OUTPUT_PATH\n``` \n\n\n##### Example 4: STILTs MNLI \u0026rarr; RTE \n\n```bash\nexport TASK_A=mnli\nexport TASK_B=rte\nexport OUTPUT_PATH_A=mnli\nexport OUTPUT_PATH_B=mnli__rte\n\n# MNLI\npython glue/train.py \\\n    --task_name $TASK_A \\\n    --do_train --do_val \\\n    --do_save \\\n    --do_lower_case \\\n    --bert_model bert-large-uncased \\\n    --bert_load_mode from_pretrained \\\n    --bert_save_mode model_all \\\n    --train_batch_size 24 \\\n    --learning_rate 2e-5 \\\n    --output_dir $OUTPUT_PATH_A\n    \n# MNLI -\u003e RTE\npython glue/train.py \\\n    --task_name $TASK_B \\\n    --do_train --do_val --do_test --do_val_history \\\n    --do_save \\\n    --do_lower_case \\\n    --bert_model bert-large-uncased \\\n    --bert_load_path $OUTPUT_PATH_A/all_state.p \\\n    --bert_load_mode state_model_only \\\n    --bert_save_mode model_all \\\n    --train_batch_size 24 \\\n    --learning_rate 2e-5 \\\n    --output_dir $OUTPUT_PATH_B\n``` \n\n\n## Submission to GLUE leaderboard\n\nWe have included helper scripts for exporting submissions to the GLUE leaderboard. To prepare for submission, copy the template from `cache/submission_template` to a given new output folder:\n\n```bash\ncp -R cache/submission_template /path/to/new_submission\n```\n\nAfter running a fine-tuned/pretrained model on a task with the `--do_test` argument, a folder (e.g. `rte_output`) will be created containing `test_preds.csv` among other files. Run the following command to convert `test_preds.csv` to the submission format in the output folder.\n\n```bash\npython glue/format_for_glue.py \n    --task-name rte \\\n    --input-base-path /path/to/rte_output \\\n    --output-base-path /path/to/new_submission\n```\n\nOnce you have exported submission predictions for each task, you should have 11 `.tsv` files in total. If you run `wc -l *.tsv`, you should see something like the following:\n\n```\n   1105 AX.tsv\n   1064 CoLA.tsv\n   9848 MNLI-mm.tsv\n   9797 MNLI-m.tsv\n   1726 MRPC.tsv\n   5464 QNLI.tsv\n 390966 QQP.tsv\n   3001 RTE.tsv\n   1822 SST-2.tsv\n   1380 STS-B.tsv\n    147 WNLI.tsv\n 426597 total \n```\n\nNext run `zip -j -D submission.zip *.tsv` in the folder to generate the submission zip file. Upload the zip file to [https://gluebenchmark.com/submit](https://gluebenchmark.com/submit) to submit to the leaderboard.\n\n## Extras\n\nThis repository also supports the use of [Adapter layers](https://arxiv.org/abs/1902.00751) for BERT.\n\n## FAQ\n\n\u003e Q: What does STILTs stand for?\n\nSTILTs stand for ***S**upplementary **T**raining on **I**ntermediate **L**abeled-data **T**asks*.\n\n\u003e Q: That's it? You finetune on one task, then finetune on another?\n\nYes—in some sense, this is the simplest possible approach to pretrain/multi-task-train on another task. We do not perform multi-task training or balancing of multi-task losses, any additional hyperparameter search, early stopping or any other modification to the training procedure. We simply apply the standard BERT training procedure *twice*.\n\nEven so, this simple method of supplementary training is still competitive with more complex multi-task methods such as MT-DNN and ALICE.\n\nSo far, we have observed three main benefits of STILTs:\n\n1. STILTs tends to improve performance on tasks with little data (\u003c10k training examples)\n2. STILTs tends to stabilize training on tasks with little data\n3. In cases where the intermediate and target tasks are closely related (e.g. MNLI/RTE), we observe a significant improvement in performance. \n\n\u003e Q: The paper/abstract mentions a GLUE score of 81.8, while the leaderboard shows 82.0. What's going on?\n\nThe GLUE benchmark underwent an update wherein QNLI(v1) was replaced by a QNLIv2, which has a different train/val/test split. The GLUE leaderboard currently reports QNLIv2 scores. On the other hand, all experiments in the paper were run on QNLIv1, so we chose to report the GLUE score based on QNLIv1 in the paper.\n\nIn short, with QNLIv1, we get a GLUE score of 81.8. With QNLIv2, we got a score of 82.0.\n\n\u003e Q: When I run finetuning on CoLA/MRPC/STS-B/RTE, I get terrible results. Why?\n\nFinetuning BERT-Large on tasks with little training data (\u003c10k) tends to be unstable. This is referenced in the original BERT paper.\n\n\u003e Q: Where are the other models (GPT, ELMo) referenced in the paper?\n\nThose results were obtained using the [jiant](https://github.com/jsalt18-sentence-repl/jiant) framework. We currently have no plans to publish the trained models for those experiments.  \n\n## Citation\n\n```\n@article{phang2018stilts,\n  title={Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks},\n  author={Phang, Jason and F\\'evry,, Thibault and Bowman, Samuel R.},\n  journal={arXiv preprint arXiv:1811.01088v2},\n  year={2018}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzphang%2Fbert_on_stilts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzphang%2Fbert_on_stilts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzphang%2Fbert_on_stilts/lists"}