{"id":3879,"url":"https://github.com/thuml/awesome-multi-task-learning","name":"awesome-multi-task-learning","description":"A curated list of DATASETS, CODEBASES and PAPERS on Multi-Task Learning (MTL),  from Machine Learning perspective.","projects_count":233,"last_synced_at":"2026-07-02T19:00:42.619Z","repository":{"id":41066688,"uuid":"344677220","full_name":"thuml/awesome-multi-task-learning","owner":"thuml","description":"A curated list of DATASETS, CODEBASES and PAPERS on Multi-Task Learning (MTL),  from Machine Learning perspective.","archived":false,"fork":false,"pushed_at":"2026-03-03T01:07:34.000Z","size":187,"stargazers_count":837,"open_issues_count":1,"forks_count":65,"subscribers_count":19,"default_branch":"main","last_synced_at":"2026-05-29T16:03:02.090Z","etag":null,"topics":["adapter","awesome-list","computer-vision","deep-learning","deep-neural-networks","loss-strategy","machine-learning","multi-domain-learning","multi-task-architecture","multi-task-learning","multi-task-optimization","neural-language-processing","transfer-learning"],"latest_commit_sha":null,"homepage":"https://github.com/thuml/awesome-multi-task-learning","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thuml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-03-05T03:01:25.000Z","updated_at":"2026-05-28T10:17:39.000Z","dependencies_parsed_at":"2025-05-06T21:34:11.428Z","dependency_job_id":"5ded34a8-b30f-4c7c-b22e-f7bc04e17b3d","html_url":"https://github.com/thuml/awesome-multi-task-learning","commit_stats":null,"previous_names":["thuml/awesome-multi-task-learning","manchery/awesome-multi-task-learning"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/thuml/awesome-multi-task-learning","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thuml%2Fawesome-multi-task-learning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thuml%2Fawesome-multi-task-learning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thuml%2Fawesome-multi-task-learning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thuml%2Fawesome-multi-task-learning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thuml","download_url":"https://codeload.github.com/thuml/awesome-multi-task-learning/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thuml%2Fawesome-multi-task-learning/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34349926,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-15T02:00:07.085Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2024-01-06T15:11:11.754Z","updated_at":"2026-07-02T19:00:42.620Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":["Optimization","Benchmark \u0026 Dataset","Task Relationship Learning: Grouping, Tree (Hierarchy) \u0026 Cascading","Survey","Architecture","Misc","Theory","Codebase"],"sub_categories":["Distillation","Computer Vision","Recommendation","Consistency","Loss \u0026 Gradient Strategy","Modulation \u0026 Adapters","Hard Parameter Sharing","NLP","RL \u0026 Robotics","Graph","Soft Parameter Sharing","Decoder-focused Model","Task Sampling","Task Interference","Modularity, MoE, Routing \u0026 NAS","Others","Adversarial Training","Pareto"],"readme":"# Awesome Multi-Task Learning\n\nA curated list of datasets, codebases, and papers on Multi-Task Learning (MTL), from a Machine Learning perspective. \n\nThis project greatly appreciates the surveys below, which have been incredibly helpful.\n\nWe welcome your contributions! If you find any mistakes or omissions, please let us know.\n\n**Contact**: [Jialong Wu](https://manchery.github.io/)\n\n## Table of Contents\n\n\u003cdetails\u003e\n  \u003csummary\u003eAwesome Multi-Task Learning\u003c/summary\u003e\n\n- [Survey](#survey)\n- [Benchmark \u0026 Dataset](#benchmark--dataset)\n  - [Computer Vision](#computer-vision)\n  - [NLP](#nlp)\n  - [RL \u0026 Robotics](#rl--robotics)\n  - [Graph](#graph)\n  - [Recommendation](#recommendation)\n- [Codebase](#codebase)\n- [Architecture](#architecture)\n  - [Hard Parameter Sharing](#hard-parameter-sharing)\n  - [Soft Parameter Sharing](#soft-parameter-sharing)\n  - [Decoder-focused Model](#decoder-focused-model)\n  - [Modulation \u0026 Adapters](#modulation--adapters)\n  - [Modularity, MoE, Routing \u0026 NAS](#modularity-moe-routing--nas)\n  - [Task Representation](#task-representation)\n  - [Others](#others)\n- [Optimization](#optimization)\n  - [Loss \u0026 Gradient Strategy](#loss--gradient-strategy)\n  - [Task Interference](#task-interference)\n  - [Task Sampling](#task-sampling)\n  - [Adversarial Training](#adversarial-training)\n  - [Pareto](#pareto)\n  - [Distillation](#distillation)\n  - [Consistency](#consistency)\n- [Task Relationship Learning: Grouping, Tree (Hierarchy) \u0026 Cascading](#task-relationship-learning-grouping-tree-hierarchy--cascading)\n- [Theory](#theory)\n- [Misc](#misc)\n\u003c/details\u003e\n\n\n## Survey\n\n- ✨ Yu, J., Dai, Y., Liu, X., Huang, J., Shen, Y., Zhang, K., ... \u0026 Chen, Y. [Unleashing the Power of Multi-Task Learning: A Comprehensive Survey Spanning Traditional, Deep, and Pretrained Foundation Model Eras](https://arxiv.org/abs/2404.18961). ArXiv, 2024.\n- ✨ Vandenhende, S., Georgoulis, S., Proesmans, M., Dai, D., \u0026 Van Gool, L.  [Multi-Task Learning for Dense Prediction Tasks: A Survey](https://arxiv.org/abs/2004.13379). TPAMI, 2021.\n- Crawshaw, M.  [Multi-Task Learning with Deep Neural Networks: A Survey](http://arxiv.org/abs/2009.09796). ArXiv, 2020. \n- Worsham, J., \u0026 Kalita, J.  [Multi-task learning for natural language processing in the 2020s: Where are we going?](https://doi.org/10.1016/j.patrec.2020.05.031) *Pattern Recognition Letters*, 2020.\n- Gong, T., Lee, T., Stephenson, C., Renduchintala, V., Padhy, S., Ndirango, A., Keskin, G., \u0026 Elibol, O. H.  [A Comparison of Loss Weighting Strategies for Multi task Learning in Deep Neural Networks](https://ieeexplore.ieee.org/document/8848395). IEEE Access, 2019.\n- Li, J., Liu, X., Yin, W., Yang, M., Ma, L., \u0026 Jin, Y.  [Empirical Evaluation of Multi-task Learning in Deep Neural Networks for Natural Language Processing](https://link.springer.com/article/10.1007/s00521-020-05268-w). Neural Computing and Applications, 2021.\n- ✨ Ruder, S.  [An Overview of Multi-Task Learning in Deep Neural Networks](http://arxiv.org/abs/1706.05098). ArXiv, 2017. \n- ✨ Zhang, Y., \u0026 Yang, Q.  [A Survey on Multi-Task Learning](https://ieeexplore.ieee.org/abstract/document/9392366). IEEE TKDE, 2021.\n\n\n\n## Benchmark \u0026 Dataset\n\n### Computer Vision\n\n- MultiMNIST / MultiFashionMNIST\n  - a multitask variant of the MNIST / FashionMNIST dataset\n  - ⚠️ *Toy datasets*\n  - See: [MGDA](http://arxiv.org/abs/1810.04650), [Pareto MTL](http://papers.nips.cc/paper/9374-pareto-multi-task-learning.pdf), [IT-MTL](https://arxiv.org/abs/2010.15413), *etc*.\n- ✨ NYUv2 [[URL](https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html)]\n  - 3 Tasks: Semantic Segmentation, Depth Estimation, Surface Normal Estimation\n  - Silberman, N., Hoiem, D., Kohli, P., \u0026 Fergus, R. (2012). [Indoor Segmentation and Support Inference from RGBD Images](https://cs.nyu.edu/~silberman/papers/indoor_seg_support.pdf). ECCV, 2012.\n- ✨ CityScapes [[URL](https://www.cityscapes-dataset.com/)]\n  - 3 Tasks: Semantic Segmentation, Instance Segmentation, Depth Estimation\n- ✨ PASCAL Context [[URL](https://cs.stanford.edu/~roozbeh/pascal-context/)]\n  - Tasks: Semantic Segmentation, Human Part Segmentation, Semantic Edge Detection, Surface Normals Prediction, Saliency Detection.\n- ✨ CelebA [[URL](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html)]\n  - Tasks: 40 human face Attributes.\n- ✨ Taskonomy [[URL](http://taskonomy.stanford.edu/)]\n  - 26 Tasks: Scene Categorization, Semantic Segmentation, Edge Detection, Monocular Depth Estimation, Keypoint Detection, *etc*.\n- Visual Domain Decathlon [[URL](https://www.robots.ox.ac.uk/~vgg/decathlon/)]\n  - 10 Datasets: ImageNet, Aircraft, CIFAR100, *etc*. \n  - Multi-domain multi-task learning\n  - Rebuffi, S.-A., Bilen, H., \u0026 Vedaldi, A.  [Learning multiple visual domains with residual adapters](https://arxiv.org/abs/1705.08045). NeurIPS, 2017.\n- BDD100K [[URL](https://bdd-data.berkeley.edu/)]\n  - 10-task Driving Dataset\n  - Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., \u0026 Darrell, T.  [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](http://arxiv.org/abs/1805.04687). CVPR, 2020.\n- MS COCO\n  - Object detection, pose estimation, semantic segmentation.\n  - See: [MultiTask-CenterNet (MCN): Efficient and Diverse Multitask Learning using an Anchor Free Approach](http://arxiv.org/abs/2108.05060).\n- Omnidata [[URL](https://omnidata.vision/)]\n  - A pipeline to resample comprehensive 3D scans from the real-world into static multi-task vision datasets\n  - Eftekhar, A., Sax, A., Bachmann, R., Malik, J., \u0026 Zamir, A.  [Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans](http://arxiv.org/abs/2110.04994). ICCV, 2021. \n\n\n\n### NLP\n\n- ✨ GLUE \\- General Language Understanding Evaluation [[URL](https://gluebenchmark.com/)]\n- ✨ decaNLP - The Natural Language Decathlon: A Multitask Challenge for NLP [[URL](https://github.com/salesforce/decaNLP)]\n- WMT Multilingual Machine Translation\n- `tasksource` - 500+ MultipleChoice/Classification/TokenClassification tasks from HuggingFace Datasets Hub [[URL](https://github.com/sileod/tasksource)]\n  - Sileo, D.  [tasksource: Structured Dataset Preprocessing Annotations for Frictionless Extreme Multi-Task Learning and Evaluation](https://arxiv.org/abs/2301.05948). ArXiv, 2023.\n\n\n### RL \u0026 Robotics\n\n- ✨ MetaWorld [[URL](https://meta-world.github.io/)]\n- MTEnv [[URL](https://github.com/facebookresearch/mtenv)]\n\n### Graph\n\n- QM9 [[URL](https://www.nature.com/articles/sdata201422)]\n  - 11 properties of molecules; multi-task regression\n  - See: [Multi-Task Learning as a Bargaining Game](http://arxiv.org/abs/2202.01017).\n\n### Recommendation\n\n- AliExpress [[URL](https://tianchi.aliyun.com/dataset/74690)]\n  - 2 Tasks: CTR and CTCVR from 5 countries\n  - Li, P., Li, R., Da, Q., Zeng, A. X., \u0026 Zhang, L. [Improving Multi-Scenario Learning to Rank in E-commerce by Exploiting Task Relationships in the Label Space](https://cs.nju.edu.cn/_upload/tpl/01/0c/268/template268/pdf/CIKM-2020-Li.pdf). CIKM, 2020.\n  - See: [MTReclib](https://github.com/easezyc/Multitask-Recommendation-Library#datasets)\n- MovieLens [[URL](https://www.tensorflow.org/datasets/catalog/movielens)]\n  - 2 Tasks: binary classification (whether the user will watch) \u0026 regression (user’s rating)\n  - See: [DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning](http://arxiv.org/abs/2106.03760)\n\n\n\n## Codebase\n\n- **General**\n  - ✨ [LibMTL](https://github.com/median-research-group/libmtl): LibMTL: A PyTorch Library for Multi-Task Learning\n  - ✨ [TorchJD](https://github.com/TorchJD/torchjd): Library for Jacobian descent with PyTorch. It enables the optimization of neural networks with multiple losses (e.g., multi-task learning).\n  - [MALSAR](https://github.com/jiayuzhou/MALSAR): Multi-task learning via Structural Regularization (⚠️ Non-deep Learning)\n- **Computer Vision**\n  - ✨ [Multi-Task-Learning-PyTorch](https://github.com/SimonVandenhende/Multi-Task-Learning-PyTorch): PyTorch implementation of multi-task learning architectures\n  - ✨ [mtan](https://github.com/lorenmt/mtan): The implementation of \"End-to-End Multi-Task Learning with Attention\"\n  - ✨ [auto-lambda](https://github.com/lorenmt/auto-lambda): The Implementation of \"Auto-Lambda: Disentangling Dynamic Task Relationships\"\n  - [astmt](https://github.com/facebookresearch/astmt): Attentive Single-tasking of Multiple Tasks\n- **NLP**\n  - ✨ [mt-dnn](https://github.com/namisan/mt-dnn): Multi-Task Deep Neural Networks for Natural Language Understanding\n- **Recommendation System**\n  - ✨ [MTReclib](https://github.com/easezyc/Multitask-Recommendation-Library): MTReclib provides a PyTorch implementation of multi-task recommendation models and common datasets.\n- **RL**\n  - [mtrl](https://github.com/facebookresearch/mtrl): Multi Task RL Baselines\n\n\n\n## Architecture\n\n### Hard Parameter Sharing\n\n- Zhao, Z., Ziser, Y., \u0026 Cohen, S. B.  [Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models](https://aclanthology.org/2024.emnlp-main.847.pdf). EMNLP, 2024.\n- Heuer, F., Mantowsky, S., Bukhari, S. S., \u0026 Schneider, G.  [MultiTask-CenterNet (MCN): Efficient and Diverse Multitask Learning using an Anchor Free Approach](http://arxiv.org/abs/2108.05060). ICCV, 2021.\n- Hu, R., \u0026 Singh, A.  [UniT: Multimodal Multitask Learning with a Unified Transformer](http://arxiv.org/abs/2102.10772). ICCV, 2021.\n- ✨ Liu, X., He, P., Chen, W., \u0026 Gao, J.  [Multi-Task Deep Neural Networks for Natural Language Understanding](https://arxiv.org/pdf/1901.11504.pdf). ACL, 2019.\n- ✨ Kokkinos, I.  [UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory](https://arxiv.org/abs/1609.02132). CVPR, 2017.\n- Teichmann, M., Weber, M., Zoellner, M., Cipolla, R., \u0026 Urtasun, R.  [MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving](http://arxiv.org/abs/1612.07695). ArXiv, 2016.\n- Caruana, R. [Multitask Learning](https://link.springer.com/content/pdf/10.1023/A:1007379606734.pdf). 1997.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"imgs/hard-parameter-sharing.png\" alt=\"client-demo\" width=\"600px\" /\u003e\n\u003c/p\u003e\n\n### Soft Parameter Sharing\n\n- Ruder, S., Bingel, J., Augenstein, I., \u0026 Søgaard, A.  [Latent Multi-task Architecture Learning](https://arxiv.org/abs/1705.08142). AAAI, 2019.\n- Gao, Y., Ma, J., Zhao, M., Liu, W., \u0026 Yuille, A. L.  [NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reduction](https://arxiv.org/abs/1801.08297). CVPR, 2019.\n- Long, M., Cao, Z., Wang, J., \u0026 Yu, P. S.  [Learning Multiple Tasks with Multilinear Relationship Networks](https://proceedings.neurips.cc/paper/2017/file/03e0704b5690a2dee1861dc3ad3316c9-Paper.pdf). NeurIPS, 2017.\n- ✨ Misra, I., Shrivastava, A., Gupta, A., \u0026 Hebert, M.  [Cross-Stitch Networks for Multi-task Learning](https://arxiv.org/abs/1604.03539). CVPR, 2016.\n- ✨ Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., \u0026 Hadsell, R.  [Progressive Neural Networks](https://arxiv.org/abs/1606.04671). ArXiv, 2016.\n- ✨ Yang, Y., \u0026 Hospedales, T. [Deep Multi-task Representation Learning: A Tensor Factorisation Approach](https://arxiv.org/abs/1605.06391). ICLR, 2017.\n- Yang, Y., \u0026 Hospedales, T. M. [Trace Norm Regularised Deep Multi-Task Learning](http://arxiv.org/abs/1606.04038). ICLR Workshop, 2017.\n\n### Decoder-focused Model\n\n- Ye, H., \u0026 Xu, D.  [TaskPrompter: Spatial-Channel Multi-Task Prompting for Dense Scene Understanding](https://openreview.net/forum?id=-CwPopPJda). ICLR, 2023.\n- Ye, H., \u0026 Xu, D.  [Inverted Pyramid Multi-task Transformer for Dense Scene Understanding](https://arxiv.org/abs/2203.07997). ECCV, 2022.\n- Bruggemann, D., Kanakis, M., Obukhov, A., Georgoulis, S., \u0026 Van Gool, L.  [Exploring Relational Context for Multi-Task Dense Prediction](http://arxiv.org/abs/2104.13874). ICCV, 2021. \n- Vandenhende, S., Georgoulis, S., \u0026 Van Gool, L.  [MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning](http://arxiv.org/abs/2001.06902). ECCV, 2020.\n- Zhang, Z., Cui, Z., Xu, C., Yan, Y., Sebe, N., \u0026 Yang, J.  [Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation](https://openaccess.thecvf.com/content_CVPR_2019/papers/Zhang_Pattern-Affinitive_Propagation_Across_Depth_Surface_Normal_and_Semantic_Segmentation_CVPR_2019_paper.pdf). CVPR, 2019. \n- Xu, D., Ouyang, W., Wang, X., \u0026 Sebe, N.  [PAD-Net: Multi-tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing](https://arxiv.org/abs/1805.04409). CVPR, 2018.\n\n### Modulation \u0026 Adapters\n\n- Schmied, T., Hofmarcher, M., Paischer, F., Pascanu, R., \u0026 Hochreiter, S.  [Learning to Modulate pre-trained Models in RL](https://doi.org/10.48550/arXiv.2306.14884). NeurIPS, 2023.\n- Sharma, M., Fantacci, C., Zhou, Y., Koppula, S., Heess, N., Scholz, J., \u0026 Aytar, Y.  [Lossless Adaptation of Pretrained Vision Models For Robotic Manipulation](https://doi.org/10.48550/arXiv.2304.06600). ICLR, 2023. \n- ✨ He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., \u0026 Neubig, G. [Towards a Unified View of Parameter-Efficient Transfer Learning](http://arxiv.org/abs/2110.04366). ICLR, 2022.\n- Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., \u0026 Raffel, C.  [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://openreview.net/forum?id=rBCvMG-JsPd). NeurIPS, 2022.\n- Zhang, L., Yang, Q., Liu, X., \u0026 Guan, H.  [Rethinking Hard-Parameter Sharing in Multi-Domain Learning](http://arxiv.org/abs/2107.11359). ICME, 2022.\n- Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., \u0026 Pfister, T.  [Learning to Prompt for Continual Learning](http://arxiv.org/abs/2112.08654). CVPR, 2022. \n- ✨ Lester, B., Al-Rfou, R., \u0026 Constant, N.  [The Power of Scale for Parameter-Efficient Prompt Tuning](https://arxiv.org/abs/2104.08691). EMNLP, 2021.\n- ✨ Li, X. L., \u0026 Liang, P.  [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://arxiv.org/abs/2101.00190).  ACL, 2021.\n- Zhu, Y., Feng, J., Zhao, C., Wang, M., \u0026 Li, L.  [Counter-Interference Adapter for Multilingual Machine Translation](https://aclanthology.org/2021.findings-emnlp.240). Findings of EMNLP, 2021. \n- ✨ Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., \u0026 Chen, W.  [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685). Arxiv, 2021.\n- Pilault, J., Elhattami, A., \u0026 Pal, C. J. [Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters \u0026 Less Data](https://openreview.net/forum?id=de11dbHzAMF). ICLR, 2021.\n- Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., \u0026 Gurevych, I.  [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](http://arxiv.org/abs/2005.00247). EACL, 2021.\n- Kanakis, M., Bruggemann, D., Saha, S., Georgoulis, S., Obukhov, A., \u0026 Van Gool, L.  [Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference](http://arxiv.org/abs/2007.12540). ECCV, 2020.\n- Pham, M. Q., Crego, J. M., Yvon, F., \u0026 Senellart, J.  [A Study of Residual Adapters for Multi-Domain Neural Machine Translation](https://www.aclweb.org/anthology/2020.wmt-1.72/). WMT, 2020.\n- ✨ Pfeiffer, J., Rücklé, A., Poth, C., Kamath, A., Vulić, I., Ruder, S., Cho, K., \u0026 Gurevych, I.  [AdapterHub: A Framework for Adapting Transformers](http://arxiv.org/abs/2007.07779). EMNLP 2020: Systems Demonstrations.\n- Pfeiffer, J., Vulić, I., Gurevych, I., \u0026 Ruder, S.  [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://doi.org/10.18653/v1/2020.emnlp-main.617). EMNLP, 2020.\n- Zhao, M., Lin, T., Mi, F., Jaggi, M., \u0026 Schütze, H.  [Masking as an Efficient Alternative to Finetuning for Pretrained Language Models](http://arxiv.org/abs/2004.12406). EMNLP, 2020.\n- ✨ **[MTAN]** Liu, S., Johns, E., \u0026 Davison, A. J.  [End-to-End Multi-Task Learning with Attention](http://arxiv.org/abs/1803.10704). CVPR, 2019. \n- Strezoski, G., Noord, N., \u0026 Worring, M.  [Many Task Learning With Task Routing](https://arxiv.org/abs/1903.12117). ICCV, 2019.\n- Maninis, K.-K., Radosavovic, I., \u0026 Kokkinos, I.  [Attentive Single-Tasking of Multiple Tasks](http://arxiv.org/abs/1904.08918). CVPR, 2019.\n- ✨ Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., \u0026 Gelly, S.  [Parameter-Efficient Transfer Learning for NLP](http://arxiv.org/abs/1902.00751). ICML, 2019.\n- Stickland, A. C., \u0026 Murray, I.  [BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning](http://arxiv.org/abs/1902.02671). ICML, 2019.\n- Zhao, X., Li, H., Shen, X., Liang, X., \u0026 Wu, Y.  [A Modulation Module for Multi-task Learning with Applications in Image Retrieval](https://arxiv.org/abs/1807.06708). ECCV, 2018.\n- ✨ Rebuffi, S.-A., Vedaldi, A., \u0026 Bilen, H.  [Efficient Parametrization of Multi-domain Deep Neural Networks](https://arxiv.org/abs/1803.10082). CVPR, 2018.\n- ✨ Rebuffi, S.-A., Bilen, H., \u0026 Vedaldi, A.  [Learning multiple visual domains with residual adapters](https://arxiv.org/abs/1705.08045). NeurIPS, 2017.\n\n### Modularity, MoE, Routing \u0026 NAS\n\n- Chen, Z., Shen, Y., Ding, M., Chen, Z., Zhao, H., Learned-Miller, E., \u0026 Gan, C.  [Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners](https://arxiv.org/abs/2212.08066). CVPR, 2023.\n- ✨ Yang, X., Ye, J., \u0026 Wang, X.  [Factorizing Knowledge in Neural Networks](http://arxiv.org/abs/2207.03337). ECCV, 2022. \n- ✨ Liang, H., Fan, Z., Sarkar, R., Jiang, Z., Chen, T., Zou, K., ... \u0026 Wang, Z.  [M$^ 3$ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design](https://arxiv.org/abs/2210.14793). NeurIPS, 2022.\n- Zhang, L., Liu, X., \u0026 Guan, H.  [AutoMTL: A Programming Framework for Automated Multi-Task Learning](http://arxiv.org/abs/2110.13076). NeurIPS, 2022.\n- Gesmundo, A., \u0026 Dean, J.  [An Evolutionary Approach to Dynamic Introduction of Tasks in Large-scale Multitask Learning Systems](http://arxiv.org/abs/2205.12755). ArXiv, 2022. \n- Tang, D., Zhang, F., Dai, Y., Zhou, C., Wu, S., \u0026 Shi, S.  [SkillNet-NLU: A Sparsely Activated Model for General-Purpose Natural Language Understanding](https://arxiv.org/abs/2203.03312). ArXiv, 2022.\n- Ponti, E. M., Sordoni, A., Bengio, Y., \u0026 Reddy, S.  [Combining Modular Skills in Multitask Learning](https://arxiv.org/abs/2202.13914). ArXiv, 2022.\n- Hazimeh, H., Zhao, Z., Chowdhery, A., Sathiamoorthy, M., Chen, Y., Mazumder, R., Hong, L., \u0026 Chi, E. H.  [DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning](http://arxiv.org/abs/2106.03760). NeurIPS, 2021.\n- ✨ **[Pathways]** *Introducing Pathways: A next-generation AI architecture*. Oct 28, 2021. Retrieved March 9, 2022, from https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/\n- ✨ Yang, R., Xu, H., Wu, Y., \u0026 Wang, X.  [Multi-Task Reinforcement Learning with Soft Modularization](http://arxiv.org/abs/2003.13661). NeurIPS, 2020. \n- Sun, X., Panda, R., \u0026 Feris, R.  [AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning](http://arxiv.org/abs/1911.12423). NeurIPS, 2020. \n- Bruggemann, D., Kanakis, M., Georgoulis, S., \u0026 Van Gool, L.  [Automated Search for Resource-Efficient Branched Multi-Task Networks](http://arxiv.org/abs/2008.10292). BMVC, 2020. \n- Gao, Y., Bai, H., Jie, Z., Ma, J., Jia, K., \u0026 Liu, W. [MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning](https://arxiv.org/abs/2003.14058). CVPR, 2020.\n- ✨ **[PLE]** Tang, H., Liu, J., Zhao, M., \u0026 Gong, X.  [Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations](https://doi.org/10.1145/3383313.3412236). RecSys, 2020 (Best Paper).\n- Bragman, F., Tanno, R., Ourselin, S., Alexander, D., \u0026 Cardoso, J.  [Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernels](https://arxiv.org/abs/1908.09597). ICCV, 2019.\n- Ahn, C., Kim, E., \u0026 Oh, S.  [Deep Elastic Networks with Model Selection for Multi-Task Learning](http://arxiv.org/abs/1909.04860). ICCV, 2019. \n- Ma, J., Zhao, Z., Chen, J., Li, A., Hong, L., \u0026 Chi, E. H.  [SNR: Sub-Network Routing for Flexible Parameter Sharing in Multi-Task Learning](https://ojs.aaai.org/index.php/AAAI/article/view/3788/3666). AAAI, 2019.\n- Maziarz, K., Kokiopoulou, E., Gesmundo, A., Sbaiz, L., Bartok, G., \u0026 Berent, J.  [Flexible Multi-task Networks by Learning Parameter Allocation](http://arxiv.org/abs/1910.04915). ArXiv,  2019.\n- Newell, A., Jiang, L., Wang, C., Li, L.-J., \u0026 Deng, J.  [Feature Partitioning for Efficient Multi-Task Architectures](https://arxiv.org/abs/1908.04339). ArXiv, 2019.\n- ✨ **[MMoE]** Ma, J., Zhao, Z., Yi, X., Chen, J., Hong, L., \u0026 Chi, E. H.  [Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts](https://dl.acm.org/doi/pdf/10.1145/3219819.3220007). KDD, 2018.\n- Rosenbaum, C., Klinger, T., \u0026 Riemer, M.  [Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning](http://arxiv.org/abs/1711.01239). ICLR, 2018.\n- Meyerson, E., \u0026 Miikkulainen, R.  [Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering](http://arxiv.org/abs/1711.00108). ICLR, 2018.\n- Liang, J., Meyerson, E., \u0026 Miikkulainen, R.  [Evolutionary architecture search for deep multitask networks](https://arxiv.org/abs/1803.03745). *Proceedings of the Genetic and Evolutionary Computation Conference*, 2018.\n- Kim, E., Ahn, C., \u0026 Oh, S.  [NestedNet: Learning Nested Sparse Structures in Deep Neural Networks](https://openaccess.thecvf.com/content_cvpr_2018/papers/Kim_NestedNet_Learning_Nested_CVPR_2018_paper.pdf). CVPR, 2018.\n- Andreas, J., Klein, D., \u0026 Levine, S.  [Modular Multitask Reinforcement Learning with Policy Sketches](http://arxiv.org/abs/1611.01796). ICML, 2017.\n- Devin, C., Gupta, A., Darrell, T., Abbeel, P., \u0026 Levine, S.  [Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer](http://arxiv.org/abs/1609.07088). ICRA, 2017\n- ✨ Fernando, C., Banarse, D., Blundell, C., Zwols, Y., Ha, D., Rusu, A. A., Pritzel, A., \u0026 Wierstra, D.  [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](http://arxiv.org/abs/1701.08734). ArXiv, 2017. \n\n### Task Representation\n\n- Sodhani, S., Zhang, A., \u0026 Pineau, J. [Multi-Task Reinforcement Learning with Context-based Representations](http://arxiv.org/abs/2102.06177). ICML, 2021. \n\n### Others\n\n- Sun, T., Shao, Y., Li, X., Liu, P., Yan, H., Qiu, X., \u0026 Huang, X.  [Learning Sparse Sharing Architectures for Multiple Tasks](http://arxiv.org/abs/1911.05034). AAAI, 2020.\n- Lee, H. B., Yang, E., \u0026 Hwang, S. J.  [Deep Asymmetric Multi-task Feature Learning](http://proceedings.mlr.press/v80/lee18d/lee18d.pdf). ICML, 2018.\n- Zhang, Y., Wei, Y., \u0026 Yang, Q.  [Learning to Multitask](https://papers.nips.cc/paper/2018/file/aeefb050911334869a7a5d9e4d0e1689-Paper.pdf). NeurIPS, 2018.\n- ✨ Mallya, A., Davis, D., \u0026 Lazebnik, S.  [Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights](https://arxiv.org/abs/1801.06519). ECCV 2018.\n- ✨ Mallya, A., \u0026 Lazebnik, S.  [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://arxiv.org/abs/1711.05769). CVPR, 2018.\n- Lee, G., Yang, E., \u0026 Hwang, S. J.  [Asymmetric Multi-task Learning based on Task Relatedness and Confidence](http://proceedings.mlr.press/v48/leeb16.pdf). ICML, 2016.\n\n\n\n## Optimization\n\n### Loss \u0026 Gradient Strategy\n- **[FS-MTL]** Phan, H., Tran, L., Tran, N. N., Ho, N., Phung, D., \u0026 Le, T. [Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspective](https://arxiv.org/abs/2211.13723). ICCV Highlight, 2025.\n- **[UPGrad]** Quinton, P. \u0026 Rey, V. [Jacobian Descent for Multi-Objective Optimization](https://arxiv.org/pdf/2406.16232). ArXiv, 2024.\n- **[ConFIG]** Liu, Q., Chu, M., \u0026 Thuerey, N.  [ConFIG: Towards conflict-free training of physics informed neural networks](https://arxiv.org/pdf/2408.11104). ArXiv, 2024.\n- **[FairGrad]** Ban, H., \u0026 Ji, K. [Fair Resource Allocation in Multi-Task Learning](https://arxiv.org/abs/2402.15638). ICML, 2024.\n- **[SDMGrad]** Xiao, P., Ban, H., \u0026 Ji, K. [Direction-oriented multi-objective learning: Simple and provable stochastic algorithms](https://openreview.net/forum?id=4Ks8RPcXd9). NeurIPS, 2023.\n- **[Population-Based Training]** Royer, A., Blankevoort, T., \u0026 Bejnordi, B. E.  [Scalarization for Multi-Task and Multi-Domain Learning at Scale](https://arxiv.org/abs/2310.08910). NeurIPS, 2023.\n- **[IGB]** Dai, Y., Fei, N., \u0026 Lu, Z. [Improvable Gap Balancing for Multi-Task Learning](https://arxiv.org/abs/2307.15429). UAI, 2023.\n- **[Aligned-MTL]** Senushkin, D., Patakin, N., Kuznetsov, A., \u0026 Konushin, A.  [Independent Component Alignment for Multi-Task Learning](https://openaccess.thecvf.com/content/CVPR2023/papers/Senushkin_Independent_Component_Alignment_for_Multi-Task_Learning_CVPR_2023_paper.pdf). CVPR, 2023.\n- **[MoCo]** Fernando, H. D., Shen, H., Liu, M., Chaudhury, S., Murugesan, K., \u0026 Chen, T.  [Mitigating Gradient Bias in Multi-objective Learning: A Provably Convergent Approach](https://openreview.net/forum?id=dLAYGdKTi2). ICLR, 2023.\n- **[FAMO]** Liu, B., Feng, Y., Stone, P., \u0026 Liu, Q.  [FAMO: Fast Adaptive Multitask Optimization](http://arxiv.org/abs/2306.03792). ArXiv, 2023. \n- ✨ **[ForkMerge]** Jiang, J., Chen, B., Pan, J., Wang, X., Dapeng, L., Jiang, J., \u0026 Long, M.  [ForkMerge: Overcoming Negative Transfer in Multi-Task Learning](http://arxiv.org/abs/2301.12618). ArXiv, 2023. \n- **[AuxiNash]** Shamsian, A., Navon, A., Glazer, N., Kawaguchi, K., Chechik, G., \u0026 Fetaya, E.  [Auxiliary Learning as an Asymmetric Bargaining Game](https://arxiv.org/abs/2301.13501). ArXiv, 2023.\n- ✨ Xin, Derrick, Behrooz Ghorbani, Justin Gilmer, Ankush Garg, and Orhan Firat. **[Do Current Multi-Task Optimization Methods in Deep Learning Even Help?](https://openreview.net/forum?id=A2Ya5aLtyuG)** NeurIPS, 2022.\n- **[Unitary Scalarization]** Kurin, V., De Palma, A., Kostrikov, I., Whiteson, S., \u0026 Kumar, M. P.  [In Defense of the Unitary Scalarization for Deep Multi-Task Learning](http://arxiv.org/abs/2201.04122). NeurIPS, 2022. \n  - Minimize the multi-task training objective with a standard gradient-based algorithm.\n- **[Auto-λ]** Liu, S., James, S., Davison, A. J., \u0026 Johns, E.  [Auto-Lambda: Disentangling Dynamic Task Relationships](http://arxiv.org/abs/2202.03091). TMLR, 2022. \n- **[Nash-MTL]** Navon, A., Shamsian, A., Achituve, I., Maron, H., Kawaguchi, K., Chechik, G., \u0026 Fetaya, E.  [Multi-Task Learning as a Bargaining Game](http://arxiv.org/abs/2202.01017). ICML, 2022.\n  - Also resurrects important **Scale-invariant (SI)** baseline which minimizes $\\sum_k \\log \\ell_k$.\n- **[Rotograd]** Javaloy, A., \u0026 Valera, I.  [RotoGrad: Gradient Homogenization in Multitask Learning](http://arxiv.org/abs/2103.02631). ICLR, 2022. \n- **[RLW / RGW]** Lin, B., Ye, F., \u0026 Zhang, Y.  [Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learning](http://arxiv.org/abs/2111.10603). TMLR, 2022.\n- [PINNsNTK] Wang, S., Yu, X., \u0026 Perdikaris, P.  [When and why PINNs fail to train: A neural tangent kernel perspective](https://arxiv.org/abs/2007.14527). *Journal of Computational Physics*, 2022.\n- [Inverse-Dirichlet PINNs] Maddu, S., Sturm, D., Müller, C. L., \u0026 Sbalzarini, I. F.  [Inverse Dirichlet weighting enables reliable training of physics informed neural networks](https://arxiv.org/abs/2107.00940). *Machine Learning: Science and Technology*, 2022.\n- **[CAGrad]** Liu, B., Liu, X., Jin, X., Stone, P., \u0026 Liu, Q.  [Conflict-Averse Gradient Descent for Multi-task Learning](https://arxiv.org/abs/2110.14048). NeurIPS, 2021.\n- ✨ **[Gradient Vaccine]** Wang, Z., Tsvetkov, Y., Firat, O., \u0026 Cao, Y.  [Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models](https://openreview.net/forum?id=F1vEjWK-lH_). ICLR, 2021.\n- **[IMTL]** Liu, L., Li, Y., Kuang, Z., Xue, J.-H., Chen, Y., Yang, W., Liao, Q., \u0026 Zhang, W.  [Towards Impartial Multi-task Learning](https://openreview.net/forum?id=IMPnRXEWpvr). ICLR, 2021.\n- [GradientPathologiesPINNs] Wang, S., Teng, Y., \u0026 Perdikaris, P.  [Understanding and mitigating gradient flow pathologies in physics-informed neural networks](https://arxiv.org/abs/2001.04536). *SIAM Journal on Scientific Computing*, 2021.\n- **[IT-MTL]** Fifty, C., Amid, E., Zhao, Z., Yu, T., Anil, R., \u0026 Finn, C.  [Measuring and Harnessing Transference in Multi-Task Learning](https://arxiv.org/abs/2010.15413). ArXiv, 2020.\n- **[GradDrop]** Chen, Z., Ngiam, J., Huang, Y., Luong, T., Kretzschmar, H., Chai, Y., \u0026 Anguelov, D.  [Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout](https://proceedings.neurips.cc//paper/2020/file/16002f7a455a94aa4e91cc34ebdb9f2d-Paper.pdf). NeurIPS, 2020.\n- ✨ **[PCGrad]** Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., \u0026 Finn, C.  [Gradient Surgery for Multi-Task Learning](http://arxiv.org/abs/2001.06782). NeurIPS, 2020.\n- **[Dynamic Stop-and-Go (DSG)]** Lu, J., Goswami, V., Rohrbach, M., Parikh, D., \u0026 Lee, S. [12-in-1: Multi-Task Vision and Language Representation Learning](https://openaccess.thecvf.com/content_CVPR_2020/papers/Lu_12-in-1_Multi-Task_Vision_and_Language_Representation_Learning_CVPR_2020_paper.pdf). CVPR, 2020.\n- **[Online Learning for Auxiliary losses (OL-AUX)]** Lin, X., Baweja, H., Kantor, G., \u0026 Held, D.  [Adaptive Auxiliary Task Weighting for Reinforcement Learning](https://papers.nips.cc/paper/2019/hash/0e900ad84f63618452210ab8baae0218-Abstract.html). NeurIPS, 2019.\n- **[PopArt]** Hessel, M., Soyer, H., Espeholt, L., Czarnecki, W., Schmitt, S., \u0026 Van Hasselt, H. (2019). [Multi-Task Deep Reinforcement Learning with PopArt](https://doi.org/10.1609/aaai.v33i01.33013796). AAAI, 2019.\n  - PopArt: [Learning values across many orders of magnitude](https://arxiv.org/abs/1602.07714). NeurIPS, 2016.\n- **[Dynamic Weight Average (DWA)]** Liu, S., Johns, E., \u0026 Davison, A. J.  [End-to-End Multi-Task Learning with Attention](http://arxiv.org/abs/1803.10704). CVPR, 2019. \n- **[Geometric Loss Strategy (GLS)]** Chennupati, S., Sistu, G., Yogamani, S., \u0026 Rawashdeh, S. A.  [MultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learning](http://arxiv.org/abs/1904.08492). CVPR 2019 Workshop on Autonomous Driving (WAD).\n- **[Orthogonal]** Suteu, M., \u0026 Guo, Y.  [Regularizing Deep Multi-Task Networks using Orthogonal Gradients](http://arxiv.org/abs/1912.06844). ArXiv, 2019. \n  - Enforcing near orthogonal gradients\n- **[LBTW]** Liu, S., Liang, Y., \u0026 Gitter, A. [Loss-Balanced Task Weighting to Reduce Negative Transfer in Multi-Task Learning](https://ojs.aaai.org/index.php/AAAI/article/view/5125). AAAI, 2019. \n- ✨ **[Gradient Cosine Similarity]** Du, Y., Czarnecki, W. M., Jayakumar, S. M., Farajtabar, M., Pascanu, R., \u0026 Lakshminarayanan, B.  [Adapting Auxiliary Losses Using Gradient Similarity](http://arxiv.org/abs/1812.02224). ArXiv, 2018.\n  - Uses a thresholded cosine similarity to determine whether to use each auxiliary task.\n  - Extension: **OL-AUX**\n- **[Revised Uncertainty]** Liebel, L., \u0026 Körner, M.  [Auxiliary Tasks in Multi-task Learning](http://arxiv.org/abs/1805.06334). ArXiv, 2018.\n- ✨ **[GradNorm]** Chen, Z., Badrinarayanan, V., Lee, C.-Y., \u0026 Rabinovich, A.  [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](http://arxiv.org/abs/1711.02257). ICML, 2018.\n- ✨ **[Dynamic Task Prioritization]** Guo, M., Haque, A., Huang, D.-A., Yeung, S., \u0026 Fei-Fei, L.  [Dynamic Task Prioritization for Multitask Learning](https://openaccess.thecvf.com/content_ECCV_2018/papers/Michelle_Guo_Focus_on_the_ECCV_2018_paper.pdf). ECCV, 2018.\n- ✨ **[Uncertainty]** Kendall, A., Gal, Y., \u0026 Cipolla, R.  [Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://arxiv.org/abs/1705.07115). CVPR, 2018.\n- ✨ **[MGDA]** Sener, O., \u0026 Koltun, V.  [Multi-Task Learning as Multi-Objective Optimization](http://arxiv.org/abs/1810.04650). NeurIPS, 2018.\n- **[AdaLoss]** Hu, H., Dey, D., Hebert, M., \u0026 Bagnell, J. A.  [Learning Anytime Predictions in Neural Networks via Adaptive Loss Balancing](http://arxiv.org/abs/1708.06832). ArXiv, 2017.\n  - The weights are inversely proportional to average of each loss.\n- **[Task-wise Early Stopping]** Zhang, Z., Luo, P., Loy, C. C., \u0026 Tang, X. [Facial Landmark Detection by Deep Multi-task Learning](https://personal.ie.cuhk.edu.hk/~ccloy/files/eccv_2014_deepfacealign.pdf). ECCV, 2014.\n\n*Note*: \n\n- We find that **AdaLoss**, **IMTL-l**, and **Uncertainty** are quite similiar in form.\n\n### Task Interference\n\n- Porrello, A., Buzzega, P., Dangel, F., Sommariva, T., Salami, R., Bonicelli, L., Calderara, S. [Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature](https://openreview.net/pdf?id=32mrjmaeMP). ICLR, 2026.\n  - Mitigates cross-task interference during task combination using a KFAC regularization approach.\n- Jiang, J., Chen, B., Pan, J., Wang, X., Dapeng, L., Jiang, J., \u0026 Long, M.  [ForkMerge: Overcoming Negative Transfer in Multi-Task Learning](http://arxiv.org/abs/2301.12618). ArXiv, 2023.\n- Wang, Z., Lipton, Z. C., \u0026 Tsvetkov, Y.  [On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment](http://arxiv.org/abs/2010.03017). EMNLP, 2020. \n- Schaul, T., Borsa, D., Modayil, J., \u0026 Pascanu, R.  [Ray Interference: A Source of Plateaus in Deep Reinforcement Learning](http://arxiv.org/abs/1904.11455). Arxiv, 2019. \n- Zhao, X., Li, H., Shen, X., Liang, X., \u0026 Wu, Y.  [A Modulation Module for Multi-task Learning with Applications in Image Retrieval](https://arxiv.org/abs/1807.06708). ECCV, 2018.\n  - Uses Update Compliance Ratio (UCR) to identify the *destructive interference*\n\n### Task Sampling\n\n- **[Scheduled Multi-Task Training]** Cho, M., Park, J., Lee, S., \u0026 Sung, Y. [Hard Tasks First: Multi-Task Reinforcement Learning Through Task Scheduling](https://proceedings.mlr.press/v235/cho24d.html). ICML, 2024.\n- **[MT-Uncertainty Sampling]** Pilault, J., Elhattami, A., \u0026 Pal, C. J. [Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters \u0026 Less Data](https://openreview.net/forum?id=de11dbHzAMF). ICLR, 2021.\n- **[Uniform, Task size, Counterfactual]** Glover, J., \u0026 Hokamp, C.  [Task Selection Policies for Multitask Learning](http://arxiv.org/abs/1907.06214). ArXiv, 2019. \n\n### Adversarial Training\n\n- ✨ Maninis, K.-K., Radosavovic, I., \u0026 Kokkinos, I.  [Attentive Single-Tasking of Multiple Tasks](http://arxiv.org/abs/1904.08918). CVPR, 2019.\n- Sinha, A., Chen, Z., Badrinarayanan, V., \u0026 Rabinovich, A.  [Gradient Adversarial Training of Neural Networks](http://arxiv.org/abs/1806.08028). ArXiv, 2018.\n- Liu, P., Qiu, X., \u0026 Huang, X.  [Adversarial Multi-task Learning for Text Classification](http://arxiv.org/abs/1704.05742). ACL, 2017.\n\n### Pareto\n\n- Phan, H., Tran, N., Le, T., Tran, T., Ho, N., \u0026 Phung, D.  [Stochastic Multiple Target Sampling Gradient Descent](https://arxiv.org/abs/2206.01934). NeurIPS, 2022.\n- Ma, P., Du, T., \u0026 Matusik, W. [Effcient Continuous Pareto Exploration in Multi-Task Learning](https://arxiv.org/abs/2006.16434). ICML, 2020.\n- Lin, X., Zhen, H.-L., Li, Z., Zhang, Q.-F., \u0026 Kwong, S.  [Pareto Multi-Task Learning](http://papers.nips.cc/paper/9374-pareto-multi-task-learning.pdf). NeurIPS, 2019.\n\n### Distillation\n\n- ✨ Yang, X., Ye, J., \u0026 Wang, X.  [Factorizing Knowledge in Neural Networks](http://arxiv.org/abs/2207.03337). ECCV, 2022. \n- Li, W.-H., Liu, X., \u0026 Bilen, H.  [Universal Representations: A Unified Look at Multiple Task and Domain Learning](https://arxiv.org/abs/2204.02744). ArXiv, 2022.\n- Ghiasi, G., Zoph, B., Cubuk, E. D., Le, Q. V., \u0026 Lin, T.-Y.  [Multi-Task Self-Training for Learning General Representations](http://arxiv.org/abs/2108.11353). ICCV, 2021.\n- Li, W. H., \u0026 Bilen, H. [Knowledge Distillation for Multi-task Learning](https://arxiv.org/pdf/2007.06889.pdf), ECCV-Workshop, 2020.\n- ✨ Teh, Yee Whye, Victor Bapst, Wojciech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. [Distral: Robust Multitask Reinforcement Learning](https://arxiv.org/abs/1707.04175). NeurIPS, 2017.\n- ✨ Parisotto, Emilio, Jimmy Lei Ba, and Ruslan Salakhutdinov.  [Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning](https://arxiv.org/abs/1511.06342). ICLR, 2016.\n- ✨ Rusu, Andrei A., Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. [Policy Distillation](https://arxiv.org/abs/1511.06295). ICLR, 2016.\n\n### Consistency\n\n- ✨ Zamir, A., Sax, A., Yeo, T., Kar, O., Cheerla, N., Suri, R., Cao, Z., Malik, J., \u0026 Guibas, L.  [Robust Learning Through Cross-Task Consistency](http://arxiv.org/abs/2006.04096). CVPR, 2020.\n\n\n\n## Task Relationship Learning: Grouping, Tree (Hierarchy) \u0026 Cascading\n\n- Zhao, Z., Ziser, Y., \u0026 Cohen, S. B.  [Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models](https://aclanthology.org/2024.emnlp-main.847.pdf). EMNLP, 2024.\n- ✨ Hu, Y., Yang, J., Chen, L., Li, K., Sima, C., Zhu, X., ... \u0026 Li, H.  [Planning-oriented autonomous driving](https://arxiv.org/abs/2212.10156). CVPR, 2023 (Best Paper).\n- ✨ Ilharco, Gabriel, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. [Editing Models with Task Arithmetic](https://arxiv.org/abs/2212.04089). ICLR, 2023. \n- Song, Xiaozhuang, Shun Zheng, Wei Cao, James Yu, and Jiang Bian. [Efficient and Effective Multi-Task Grouping via Meta Learning on Task Combinations](https://openreview.net/forum?id=Rqe-fJQtExY). NeurIPS, 2022.\n- Zhang, L., Liu, X., \u0026 Guan, H.  [A Tree-Structured Multi-Task Model Recommender](http://arxiv.org/abs/2203.05092). AutoML-Conf, 2022.\n- ✨ Fifty, C., Amid, E., Zhao, Z., Yu, T., Anil, R., \u0026 Finn, C.  [Efficiently Identifying Task Groupings for Multi-Task Learning](http://arxiv.org/abs/2109.04617). NeurIPS, 2021.\n- ✨ Vandenhende, S., Georgoulis, S., De Brabandere, B., \u0026 Van Gool, L.  [Branched Multi-Task Networks: Deciding What Layers To Share](http://arxiv.org/abs/1904.02920). BMVC, 2020.\n- Bruggemann, D., Kanakis, M., Georgoulis, S., \u0026 Van Gool, L.  [Automated Search for Resource-Efficient Branched Multi-Task Networks](http://arxiv.org/abs/2008.10292). BMVC, 2020. \n- ✨ Standley, T., Zamir, A. R., Chen, D., Guibas, L., Malik, J., \u0026 Savarese, S.  [Which Tasks Should Be Learned Together in Multi-task Learning?](http://arxiv.org/abs/1905.07553) ICML, 2020.\n- Guo, P., Lee, C.-Y., \u0026 Ulbricht, D.  [Learning to Branch for Multi-Task Learning](https://arxiv.org/abs/2006.01895). ICML, 2020.\n- Achille, A., Lam, M., Tewari, R., Ravichandran, A., Maji, S., Fowlkes, C., Soatto, S., \u0026 Perona, P.  [Task2Vec: Task Embedding for Meta-Learning](https://doi.org/10.1109/ICCV.2019.00653). ICCV, 2019.\n- Dwivedi, K., \u0026 Roig, G.  [Representation Similarity Analysis for Efficient Task Taxonomy \u0026 Transfer Learning](https://arxiv.org/abs/1904.11740). CVPR, 2019.\n- Guo, H., Pasunuru, R., \u0026 Bansal, M. [AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning](https://arxiv.org/abs/1904.04153). NAACL, 2019.\n- ✨ Sanh, V., Wolf, T., \u0026 Ruder, S.  [A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks](http://arxiv.org/abs/1811.06031). AAAI, 2019. \n- ✨ Zamir, A. R., Sax, A., Shen, W., Guibas, L. J., Malik, J., \u0026 Savarese, S.  [Taskonomy: Disentangling Task Transfer Learning](https://openaccess.thecvf.com/content_cvpr_2018/papers/Zamir_Taskonomy_Disentangling_Task_CVPR_2018_paper.pdf). CVPR, 2018.\n- Kim, J., Park, Y., Kim, G., \u0026 Hwang, S. J.  [SplitNet: Learning to Semantically Split Deep Networks for Parameter Reduction and Model Parallelization](http://proceedings.mlr.press/v70/kim17b.html). ICML, 2017.\n- Alonso, H. M., \u0026 Plank, B.  [When is multitask learning effective? Semantic sequence prediction under varying data conditions](http://arxiv.org/abs/1612.02251). EACL, 2017. \n- ✨ Bingel, J., \u0026 Søgaard, A.  [Identifying beneficial task relations for multi-task learning in deep neural networks](http://arxiv.org/abs/1702.08303). EACL, 2017.\n- Hand, E. M., \u0026 Chellappa, R.  [Attributes for Improved Attributes: A Multi-Task Network Utilizing Implicit and Explicit Relationships for Facial Attribute Classification](https://www.aaai.org/ocs/index.php/AAAI/AAAI17/paper/viewFile/14749/14282). AAAI, 2017.\n- ✨ Lu, Y., Kumar, A., Zhai, S., Cheng, Y., Javidi, T., \u0026 Feris, R.  [Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification](http://arxiv.org/abs/1611.05377). CVPR, 2017.\n- Hashimoto, K., xiong,  caiming, Tsuruoka, Y., \u0026 Socher, R.  [A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks](https://arxiv.org/abs/1611.01587). EMNLP, 2017.\n- Søgaard, A., \u0026 Goldberg, Y.  [Deep multi-task learning with low level tasks supervised at lower layers](https://www.aclweb.org/anthology/P16-2038.pdf). ACL, 2016.\n- Kumar, A., \u0026 Daume III, H. [Learning Task Grouping and Overlap in Multi-task Learning](http://arxiv.org/abs/1206.6417). ICML, 2012.\n- Kang, Z., Grauman, K., \u0026 Sha, F.  [Learning with Whom to Share in Multi-task Feature Learning](http://www.cs.utexas.edu/~grauman/papers/icml2011.pdf). ICML, 2011.\n- Zhang, Y., \u0026 Yeung, D.-Y.  [A Convex Formulation for Learning Task Relationships in Multi-Task Learning](http://arxiv.org/abs/1203.3536). UAI, 2010.\n\n\n\n## Theory\n\n- Wang, H., Zhao, H., \u0026 Li, B.  [Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation](http://arxiv.org/abs/2106.09017). ICML, 2021.\n- Tiomoko, M., Ali, H. T., \u0026 Couillet, R. [Deciphering and Optimizing Multi-Task Learning: A Random Matrix Approach](https://openreview.net/forum?id=Cri3xz59ga). ICLR, 2021.\n- ✨ Tripuraneni, N., Jordan, M. I., \u0026 Jin, C.  [On the Theory of Transfer Learning: The Importance of Task Diversity](https://proceedings.neurips.cc//paper/2020/file/59587bffec1c7846f3e34230141556ae-Paper.pdf). NeurIPS, 2020.\n- Wu, S., Zhang, H. R., \u0026 Re, C.  [Understanding and Improving Information Transfer in Multi-Task Learning](https://arxiv.org/abs/2005.00944). ICLR, 2020.\n\n\n\n## Misc\n\n- ✨ Bachmann, R., Mizrahi, D., Atanov, A., \u0026 Zamir, A. [MultiMAE: Multi-modal Multi-task Masked Autoencoders](http://arxiv.org/abs/2204.01678). ECCV, 2022.\n- Deng, W., Gould, S., \u0026 Zheng, L.  [What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?](http://arxiv.org/abs/2106.05961). ICML, 2021. \n- Lu, J., Goswami, V., Rohrbach, M., Parikh, D., \u0026 Lee, S. [12-in-1: Multi-Task Vision and Language Representation Learning](https://openaccess.thecvf.com/content_CVPR_2020/papers/Lu_12-in-1_Multi-Task_Vision_and_Language_Representation_Learning_CVPR_2020_paper.pdf). CVPR, 2020.\n- Mao, C., Gupta, A., Nitin, V., Ray, B., Song, S., Yang, J., \u0026 Vondrick, C. [Multitask Learning Strengthens Adversarial Robustness](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123470154.pdf). ECCV, 2020.\n- Guo, P., Xu, Y., Lin, B., \u0026 Zhang, Y.  [Multi-Task Adversarial Attack](http://arxiv.org/abs/2011.09824). ArXiv, 2020.\n- Clark, K., Luong, M.-T., Khandelwal, U., Manning, C. D., \u0026 Le, Q. V.  [BAM! Born-Again Multi-Task Networks for Natural Language Understanding](https://www.aclweb.org/anthology/P19-1595/). ACL, 2019.\n- Pramanik, S., Agrawal, P., \u0026 Hussain, A.  [OmniNet: A unified architecture for multi-modal multi-task learning](http://arxiv.org/abs/1907.07804). ArXiv, 2019.\n- Zimin, A., \u0026 Lampert, C. H. [Tasks Without Borders: A New Approach to Online Multi-Task Learning](https://openreview.net/pdf?id=HkllV5Bs24). AMTL Workshop at ICML 2019.\n- Meyerson, E., \u0026 Miikkulainen, R.  [Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domains](http://arxiv.org/abs/1906.00097).  NeurIPS, 2019.\n- Meyerson, E., \u0026 Miikkulainen, R.  [Pseudo-task Augmentation: From Deep Multitask Learning to Intratask Sharing---and Back](http://arxiv.org/abs/1803.04062). ICML, 2018. \n- Chou, Y.-M., Chan, Y.-M., Lee, J.-H., Chiu, C.-Y., \u0026 Chen, C.-S.  [Unifying and Merging Well-trained Deep Neural Networks for Inference Stage](http://arxiv.org/abs/1805.04980). IJCAI-ECAI, 2018.\n- Doersch, C., \u0026 Zisserman, A.  [Multi-task Self-Supervised Visual Learning](http://arxiv.org/abs/1708.07860). ICCV, 2017.\n- Smith, V., Chiang, C.-K., Sanjabi, M., \u0026 Talwalkar, A. S.  [Federated Multi-Task Learning](https://proceedings.neurips.cc/paper/2017/file/6211080fa89981f66b1a0c9d55c61d0f-Paper.pdf). NeurIPS, 2017.\n- Kaiser, L., Gomez, A. N., Shazeer, N., Vaswani, A., Parmar, N., Jones, L., \u0026 Uszkoreit, J.  [One Model To Learn Them All](http://arxiv.org/abs/1706.05137). ArXiv, 2017. \n- Yang, Y., \u0026 Hospedales, T. M.  [Unifying Multi-Domain Multi-Task Learning: Tensor and Neural Network Perspectives](http://arxiv.org/abs/1611.09345). ArXiv,  2016.\n- Yang, Y., \u0026 Hospedales, T. M. [A Unified Perspective on Multi-Domain and Multi-Task Learning](http://arxiv.org/abs/1412.7489). ICLR, 2015.\n\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/thuml%2Fawesome-multi-task-learning/projects"}