{"id":18816106,"url":"https://github.com/orionw/mtlvsift","last_synced_at":"2025-09-02T19:30:55.091Z","repository":{"id":98822110,"uuid":"503494227","full_name":"orionw/MTLvsIFT","owner":"orionw","description":"Code for the paper \"When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning\"","archived":false,"fork":false,"pushed_at":"2022-06-14T19:40:51.000Z","size":187,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-11-07T23:53:05.709Z","etag":null,"topics":["acl2022","machine-learning","multi-task","natural-language-processing","transfer-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/orionw.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-14T19:28:56.000Z","updated_at":"2024-06-30T05:38:19.000Z","dependencies_parsed_at":"2023-05-25T15:15:15.369Z","dependency_job_id":null,"html_url":"https://github.com/orionw/MTLvsIFT","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orionw%2FMTLvsIFT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orionw%2FMTLvsIFT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orionw%2FMTLvsIFT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orionw%2FMTLvsIFT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/orionw","download_url":"https://codeload.github.com/orionw/MTLvsIFT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231804139,"owners_count":18429028,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acl2022","machine-learning","multi-task","natural-language-processing","transfer-learning"],"created_at":"2024-11-07T23:52:30.653Z","updated_at":"2024-12-30T01:36:50.887Z","avatar_url":"https://github.com/orionw.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning](https://aclanthology.org/2022.acl-short.30/)\n========\nA repository for understanding when to use different transfer learning method for NLP, by Orion Weller, Kevin Seppi, and Matt Gardner\n\nTo reproduce the results, follow these instructions. Note that this repository was started in 2020 and contains many out of date versions of NLP packages, such as a very old version of Huggingface Transformers. Future work should likely move to more up-to-date transfer learning frameworks in order to include newer models and for ease of use.\n\n\nSetup\n-----\n0. Create the version of python you will use (e.g. `python3 -m venv env` and `source env/bin/activate`)\n1. Run the install: `./bin/install`\n2. Verify your installation with `./bin/verify`\n3. Gather the GLUE data by running `python ./bin/get_glue.py --data_dir \u003cPATH\u003e`.  This will download (most of) the GLUE data, due to legal requirements.\n\n\nTo Reproduce\n-------------------\n0. Generate the models and scores: \n- Use `./bin/glue/single_glue` to generate the single task GLUE models and scores (used for MTL dynamic sampling and for STILTs)\n- Run `./bin/batch_glue_pairs_by_seed.sh` to generate the scores for MTL, replacing the path for the single task scores.\n- Use `./bin/batch_glue_pairs_by_seed.sh _all` for MTL-ALL.\n- Use `./bin/glue/transfer_glue` after adding the single-task model checkpoints to train on the target task for STILTs.\n1. Gather the scores needed after running the models\n- For single-task scores use `./bin/create_matrices --dir_path \u003cPATH_TO_SINGLE\u003e --output_dir \u003cOUTPUT_DIR\u003e --single --use_seed`\n- For MTL and STILTs, use `./bin/create_matrices --dir_path \u003cPATH_TO_SCORES\u003e --output_dir \u003cOUTPUT_PATH\u003e --use_seed --single_data_path \u003cPATH_TO_SINGLE\u003e` to generate the matrices for these over the different random seeds\n- For MTL-ALL use `python3 ./bin/summarize_mtl_all_results.py --data_dir \u003cDATA_DIR\u003e --output_dir \u003cOUTPUT_DIR\u003e`\n2. Compare the models to make Figure 1 in the paper: `./bin/compare_matrices --mtl_matrix MTL_score_matrix.csv --inter_matrix STILTS_score_matrix.csv --output_dir \u003cOUTPUT_DIR\u003e`\n\nFor the validation size experiment, see files in `./bin/glue/*validation*` which follow the same pattern as above, but with \"validation\" in the name.\n\n\nLimitations\n-----------\nSince this code base was written in 2020, it is possible that these packages and dependencies may have problems in the future. Unfortunately, this repository will not be updated to current versions. However, please reach out or file an issue if you have any problems with reproducing this work using the given dependencies.\n\n\nCitation\n---------\nIf you found this paper or code helpful, please consider citing:\n```\n@inproceedings{weller-etal-2022-use,\n    title = \"When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning\",\n    author = \"Weller, Orion  and\n      Seppi, Kevin  and\n      Gardner, Matt\",\n    booktitle = \"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)\",\n    month = may,\n    year = \"2022\",\n    address = \"Dublin, Ireland\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2022.acl-short.30\",\n    doi = \"10.18653/v1/2022.acl-short.30\",\n    pages = \"272--282\",\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forionw%2Fmtlvsift","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Forionw%2Fmtlvsift","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forionw%2Fmtlvsift/lists"}