Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.
Awesome Lists | Featured Topics | Projects
https://github.com/SimonVandenhende/Awesome-Multi-Task-Learning

A list of multi-task learning papers and projects.
https://github.com/SimonVandenhende/Awesome-Multi-Task-Learning
List: Awesome-Multi-Task-Learning
computer-vision machine-learning multitask-learning research
Last synced: about 2 months ago
JSON representation
A list of multi-task learning papers and projects.
Host: GitHub
URL: https://github.com/SimonVandenhende/Awesome-Multi-Task-Learning
Owner: SimonVandenhende
Created: 2020-08-24T07:41:31.000Z (about 4 years ago)
Default Branch: master
Last Pushed: 2022-01-31T17:47:22.000Z (over 2 years ago)
Last Synced: 2024-05-19T20:13:34.433Z (4 months ago)
Topics: computer-vision, machine-learning, multitask-learning, research
Homepage:
Size: 270 KB
Stars: 352
Watchers: 16
Forks: 51
Open Issues: 1
Metadata Files:
- Readme: README.md
Awesome Lists containing this project

awesome-interesting-topics-in-nlp - CV
awesome-machine-learning-resources - **[List - Multi-Task-Learning?style=social) (Table of Contents)
ultimate-awesome - Awesome-Multi-Task-Learning - A list of multi-task learning papers and projects. . (Other Lists / PowerShell Lists)
README

        # Awesome Multi-Task Learning

This page contains a list of papers on multi-task learning for computer vision. 

Please create a pull request if you wish to add anything. 

If you are interested, consider reading our recent survey paper.

> [**Multi-Task Learning for Dense Prediction Tasks: A Survey**](https://arxiv.org/abs/2004.13379)

>

> [Simon Vandenhende](https://twitter.com/svandenh1), [Stamatios Georgoulis](https://twitter.com/stam_g), Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai and Luc Van Gool.

## Workshop

:loudspeaker: :loudspeaker: :loudspeaker: We organized a **workshop** on multi-task learning at ICCV 2021 ([Link](https://sites.google.com/view/deepmtlworkshop/home)).

- Jan 13: The recordings of our invited talks are now available on [Youtube](https://youtube.com/playlist?list=PLJwr5SeuN6XMXbWxnzwD1e7n-aOOJpWJn).

## Table of Contents:

- [Survey papers](#survey) 

- [Datasets](#datasets)

- [Architectures](#architectures)

  - [Encoder-based](#encoder)

  - [Decoder-based](#decoder)

  - [Other](#otherarchitectures)

- [Neural Architecture Search](#nas)

- [Optimization strategies](#optimization)

- [Transfer learning](#transfer)



## Survey papers

-  Vandenhende, S., Georgoulis, S., Van Gansbeke, W., Proesmans, M., Dai, D., & Van Gool, L. 

*[Multi-Task Learning for Dense Prediction Tasks: A Survey](https://ieeexplore.ieee.org/abstract/document/9336293)*,

T-PAMI, 2020. [[PyTorch](https://github.com/SimonVandenhende/Multi-Task-Learning-PyTorch)]

-  Ruder, S. 

*[An overview of multi-task learning in deep neural networks](https://arxiv.org/abs/1706.05098)*,

ArXiv, 2017. 

-  Zhang, Y.

*[A survey on multi-task learning](https://arxiv.org/abs/1707.08114)*, 

ArXiv, 2017.

-  Gong, T., Lee, T., Stephenson, C., Renduchintala, V., Padhy, S., Ndirango, A., ... & Elibol, O. H. 

*[A comparison of loss weighting strategies for multi task learning in deep neural networks](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8848395)*,

IEEE Access, 2019. 



## Datasets

The following datasets have been regularly used in the context of multi-task learning:

- [NYUDv2](https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html)

- [Cityscapes](https://www.cityscapes-dataset.com/)

- [PASCAL](https://github.com/facebookresearch/astmt)

- [Taskonomy](https://github.com/StanfordVL/taskonomy)

- [KITTI](http://www.cvlibs.net/datasets/kitti/)

- [SUN RGB-D](https://rgbd.cs.princeton.edu/)

- [BDD100K](https://arxiv.org/pdf/1805.04687.pdf)



## Architectures



### Encoder-based architectures

-  Misra, I., Shrivastava, A., Gupta, A., & Hebert, M.

*[Cross-stitch networks for multi-task learning](https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/Misra_Cross-Stitch_Networks_for_CVPR_2016_paper.html)*,

CVPR, 2016. [[PyTorch](https://github.com/SimonVandenhende/Multi-Task-Learning-PyTorch)]

-  Gao, Y., Ma, J., Zhao, M., Liu, W., & Yuille, A. L. 

*[Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction](https://openaccess.thecvf.com/content_CVPR_2019/html/Gao_NDDR-CNN_Layerwise_Feature_Fusing_in_Multi-Task_CNNs_by_Neural_Discriminative_CVPR_2019_paper.html)*,

CVPR, 2019. [[Tensorflow](https://github.com/ethanygao/NDDR-CNN)] [[PyTorch](https://github.com/SimonVandenhende/Multi-Task-Learning-PyTorch)]

-  Liu, S., Johns, E., & Davison, A. J. 

*[End-to-end multi-task learning with attention](https://arxiv.org/abs/1803.10704)*,

CVPR, 2019. [[PyTorch](https://github.com/lorenmt/mtan)]



### Decoder-based architectures

-  Bilen, H., Vedaldi, A.

*[Integrated perception with recurrent multi-task neural networks](https://proceedings.neurips.cc/paper/2016/file/06409663226af2f3114485aa4e0a23b4-Paper.pdf)*,

NeurIPS, 2016.

-  Xu, D., Ouyang, W., Wang, X., & Sebe, N.

*[Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing](https://openaccess.thecvf.com/content_cvpr_2018/html/Xu_PAD-Net_Multi-Tasks_Guided_CVPR_2018_paper.html)*,

CVPR, 2018.  

-  Zhang, Z., Cui, Z., Xu, C., Jie, Z., Li, X., & Yang, J.

*[Joint task-recursive learning for semantic segmentation and depth estimation](https://openaccess.thecvf.com/content_ECCV_2018/html/Zhenyu_Zhang_Joint_Task-Recursive_Learning_ECCV_2018_paper.html)*,

ECCV, 2018.

-  Ruder, S., Bingel, J., Augenstein, I., & Søgaard, A. 

*[Latent multi-task architecture learning](https://www.aaai.org/ojs/index.php/AAAI/article/view/4410)*,

AAAI, 2019.

-  Zhang, Z., Cui, Z., Xu, C., Yan, Y., Sebe, N., & Yang, J. 

*[Pattern-affinitive propagation across depth, surface normal and semantic segmentation](https://openaccess.thecvf.com/content_CVPR_2019/html/Zhang_Pattern-Affinitive_Propagation_Across_Depth_Surface_Normal_and_Semantic_Segmentation_CVPR_2019_paper.html)*,

CVPR, 2019.

-  Zhou, L., Cui, Z., Xu, C., Zhang, Z., Wang, C., Zhang, T., & Yang, J.

*[Pattern-Structure Diffusion for Multi-Task Learning](https://openaccess.thecvf.com/content_CVPR_2020/html/Zhou_Pattern-Structure_Diffusion_for_Multi-Task_Learning_CVPR_2020_paper.html)*,

CVPR, 2020.

-  Vandenhende, S., Georgoulis, S., & Van Gool, L. 

*[MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning](https://arxiv.org/abs/2001.06902)*,

ECCV, 2020. [[PyTorch](https://github.com/SimonVandenhende/Multi-Task-Learning-PyTorch)]

  



### Other

-  Yang, Y., & Hospedales, T. 

*[Deep multi-task representation learning: A tensor factorisation approach](https://arxiv.org/abs/1605.06391)*,

ICLR, 2017.

-  Kokkinos, Iasonas.

*[Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory](https://openaccess.thecvf.com/content_cvpr_2017/html/Kokkinos_Ubernet_Training_a_CVPR_2017_paper.html)*,

CVPR, 2017.

-  Rebuffi, S. A., Bilen, H., & Vedaldi, A. 

*[Learning multiple visual domains with residual adapters](https://arxiv.org/abs/1705.08045)*,

NIPS, 2017.

-  Long, M., Cao, Z., Wang, J., & Philip, S. Y. 

*[Learning multiple tasks with multilinear relationship networks](http://papers.nips.cc/paper/6757-learning-multiple-tasks-with-deep-relationship-networks)*,

NIPS, 2017.

-  Meyerson, E., & Miikkulainen, R. 

*[Beyond shared hierarchies: Deep multitask learning through soft layer ordering](https://arxiv.org/abs/1711.00108)*,

ICLR, 2018.

-  Rosenbaum, C., Klinger, T., & Riemer, M.

*[Routing networks: Adaptive selection of non-linear functions for multi-task learning](https://arxiv.org/abs/1711.01239)*,

ICLR, 2018.

-  Mallya, A., Davis, D., & Lazebnik, S.

*[Piggyback: Adapting a single network to multiple tasks by learning to mask weights](https://openaccess.thecvf.com/content_ECCV_2018/html/Arun_Mallya_Piggyback_Adapting_a_ECCV_2018_paper.html)*,

ECCV, 2018.

-  Rebuffi, S. A., Bilen, H., & Vedaldi, A.

*[Efficient parametrization of multi-domain deep neural networks](https://arxiv.org/abs/1803.10082)*,

CVPR, 2018.

-  Maninis, K. K., Radosavovic, I., & Kokkinos, I. 

*[Attentive single-tasking of multiple tasks](https://arxiv.org/abs/1904.08918)*,

CVPR, 2019. [[PyTorch](https://github.com/facebookresearch/astmt)]

-  Kanakis, M., Bruggemann, D., Saha, S., Georgoulis, S., Obukhov, A., & Van Gool, L.

*[Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference](https://arxiv.org/abs/2007.12540)*,

ECCV, 2020.

-  Wang, Q., Ke, J., Greaves, J., Chu, G., Bender, G., Sbaiz, L., Go, A., Howard, A., Yang, F., Yang, M.H. & Gilbert, J.

*[Multi-path Neural Networks for On-device Multi-domain Visual Classification](https://arxiv.org/pdf/2010.04904.pdf)*,

WACV, 2021.

-  Bruggemann, D., Kanakis, M., Obukhov, A., Georgoulis, S., & Van Gool, L. *[Exploring Relational Context for Multi-Task Dense Prediction](https://arxiv.org/abs/2104.13874)*, ArXiv, 2021.

-  Li, W. H., Liu, X., & Bilen, H. *[Universal Representation Learning from Multiple Domains for Few-shot Classification](https://openaccess.thecvf.com/content/ICCV2021/papers/Li_Universal_Representation_Learning_From_Multiple_Domains_for_Few-Shot_Classification_ICCV_2021_paper.pdf)*, ICCV, 2021.

-  Li, W. H., Liu, X., & Bilen, H. *[Learning Multiple Dense Prediction Tasks from Partially Annotated Data](https://arxiv.org/pdf/2111.14893.pdf)*, ArXiv, 2021.



## Neural Architecture Search

-  Lu, Y., Kumar, A., Zhai, S., Cheng, Y., Javidi, T., & Feris, R.

*[Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification](https://openaccess.thecvf.com/content_cvpr_2017/html/Lu_Fully-Adaptive_Feature_Sharing_CVPR_2017_paper.html)*,

CVPR, 2017. 

-  Bragman, F. J., Tanno, R., Ourselin, S., Alexander, D. C., & Cardoso, J.

*[Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels](https://openaccess.thecvf.com/content_ICCV_2019/html/Bragman_Stochastic_Filter_Groups_for_Multi-Task_CNNs_Learning_Specialist_and_Generalist_ICCV_2019_paper.html)*,

ICCV, 2019.

-  Newell, A., Jiang, L., Wang, C., Li, L. J., & Deng, J. 

*[Feature partitioning for efficient multi-task architectures](https://arxiv.org/abs/1908.04339)*,

 ArXiv, 2019.

-  Guo, P., Lee, C. Y., & Ulbricht, D. 

*[Learning to Branch for Multi-Task Learning](https://proceedings.icml.cc/static/paper_files/icml/2020/5057-Paper.pdf)*, 

ICML, 2020. 

-  Standley, T., Zamir, A. R., Chen, D., Guibas, L., Malik, J., & Savarese, S. 

*[Which Tasks Should Be Learned Together in Multi-task Learning?](https://arxiv.org/pdf/1905.07553.pdf)*,

ICML, 2020.

-  Vandenhende, S., Georgoulis, S., De Brabandere, B., & Van Gool, L. 

*[Branched multi-task networks: deciding what layers to share](https://arxiv.org/abs/1904.02920)*, 

BMVC, 2020. 

-  Bruggemann, D., Kanakis, M., Georgoulis, S., & Van Gool, L.

*[Automated Search for Resource-Efficient Branched Multi-Task Networks](https://arxiv.org/abs/2008.10292)*,

BMVC, 2020.

-  Sun, X., Panda, R., & Feris, R. 

*[AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning](https://arxiv.org/abs/1911.12423)*,

NIPS, 2020.



## Optimization strategies

-  Kendall, A., Gal, Y., & Cipolla, R. 

*[Multi-task learning using uncertainty to weigh losses for scene geometry and semantics](https://openaccess.thecvf.com/content_cvpr_2018/html/Kendall_Multi-Task_Learning_Using_CVPR_2018_paper.html)*,

CVPR, 2018. 

-  Zhao, X., Li, H., Shen, X., Liang, X., & Wu, Y. 

*[A modulation module for multi-task learning with applications in image retrieval](https://openaccess.thecvf.com/content_ECCV_2018/html/Xiangyun_Zhao_A_Modulation_Module_ECCV_2018_paper.html)*,

ECCV, 2018.

-  Chen, Z., Badrinarayanan, V., Lee, C. Y., & Rabinovich, A. 

*[Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks](http://proceedings.mlr.press/v80/chen18a.html)*,

ICML, 2018.

-  Sener, O., & Koltun, V. 

*[Multi-task learning as multi-objective optimization](http://papers.nips.cc/paper/7334-multi-task-learning-as-multi-objective-optimization)*,

NIPS, 2018. [[PyTorch](https://github.com/intel-isl/MultiObjectiveOptimization)]

-  Liu, P., Qiu, X., & Huang, X.

*[Adversarial multi-task learning for text classification](https://www.aclweb.org/anthology/P17-1001.pdf)*,

ACL, 2018.

-  Guo, M., Haque, A., Huang, D. A., Yeung, S., & Fei-Fei, L.

*[Dynamic task prioritization for multitask learning](https://openaccess.thecvf.com/content_ECCV_2018/html/Michelle_Guo_Focus_on_the_ECCV_2018_paper.html)*,

ECCV, 2018. 

-  Lin, X., Zhen, H. L., Li, Z., Zhang, Q. F., & Kwong, S.

*[Pareto multi-task learning](https://papers.nips.cc/paper/9374-pareto-multi-task-learning)*,

NIPS, 2019.

-  Suteu, M., & Guo, Y. 

*[Regularizing Deep Multi-Task Networks using Orthogonal Gradients](https://arxiv.org/abs/1912.06844)*,

ArXiv, 2019. 

-  Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., & Finn, C. 

*[Gradient surgery for multi-task learning](https://arxiv.org/abs/2001.06782)*,

NIPS, 2020. [[Tensorflow](https://github.com/tianheyu927/PCGrad)]

-  Chen, Z., Ngiam, J., Huang, Y., Luong, T., Kretzschmar, H., Chai, Y., & Anguelov, D. 

*[Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout](https://arxiv.org/pdf/2010.06808.pdf)*,

NIPS, 2020.

-  Li, W. H., & Bilen, H. 

*[Knowledge Distillation for Multi-task Learning](https://arxiv.org/pdf/2007.06889.pdf)*,

ECCV-Workshop, 2020. [[PyTorch](https://github.com/WeiHongLee/KD-MTL)]

-  Borse, S., Wang, Y., Zhang, Y., & Porikli, F. *[InverseForm: A Loss Function for Structured Boundary-Aware Segmentation](https://openaccess.thecvf.com/content/CVPR2021/papers/Borse_InverseForm_A_Loss_Function_for_Structured_Boundary-Aware_Segmentation_CVPR_2021_paper.pdf)*, CVPR 2021.

-  Vasu P., Saxena S., Tuzel O. *[Instance-Level Task Parameters: A Robust Multi-task Weighting Framework](https://arxiv.org/pdf/2106.06129.pdf)*, ArXiv, 2021.    

     

     



## Transfer learning & Domain Adaptation

-  Cui, Y., Song, Y., Sun, C., Howard, A., & Belongie, S.

*[Large scale fine-grained categorization and domain-specific transfer learning](https://openaccess.thecvf.com/content_cvpr_2018/html/Cui_Large_Scale_Fine-Grained_CVPR_2018_paper.html)*,

CVPR, 2018.

-  Zamir, A. R., Sax, A., Shen, W., Guibas, L. J., Malik, J., & Savarese, S.

*[Taskonomy: Disentangling task transfer learning](https://openaccess.thecvf.com/content_cvpr_2018/html/Zamir_Taskonomy_Disentangling_Task_CVPR_2018_paper.html)*,

CVPR, 2018. [[PyTorch](https://github.com/StanfordVL/taskonomy)]

-  Achille, A., Lam, M., Tewari, R., Ravichandran, A., Maji, S., Fowlkes, C. C., ... & Perona, P.

*[Task2vec: Task embedding for meta-learning](https://openaccess.thecvf.com/content_ICCV_2019/html/Achille_Task2Vec_Task_Embedding_for_Meta-Learning_ICCV_2019_paper.html)*,

ICCV, 2019. [[PyTorch](https://github.com/awslabs/aws-cv-task2vec)]

-  Dwivedi, K., & Roig, G.

*[Representation similarity analysis for efficient task taxonomy & transfer learning](https://openaccess.thecvf.com/content_CVPR_2019/html/Dwivedi_Representation_Similarity_Analysis_for_Efficient_Task_Taxonomy__Transfer_Learning_CVPR_2019_paper.html)*,

CVPR, 2019. [[PyTorch](https://github.com/kshitijd20/RSA-CVPR19-release)]

-  Saha, S., Obukhov, A., Paudel, D. P., Kanakis, M., Chen, Y., Georgoulis, S., & Van Gool, L. *[Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation](https://openaccess.thecvf.com/content/CVPR2021/papers/Saha_Learning_To_Relate_Depth_and_Semantics_for_Unsupervised_Domain_Adaptation_CVPR_2021_paper.pdf)*, CVPR, 2021.



## Robustness

-  Mao, C., Gupta, A., Nitin, V., Ray, B., Song, S., Yang, J., & Vondrick, C.

*[Multitask Learning Strengthens Adversarial Robustness](http://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123470154.pdf)*,

ECCV, 2020. [[PyTorch](https://github.com/columbia/MTRobust)]

-  Zamir, A. R., Sax, A., Cheerla, N., Suri, R., Cao, Z., Malik, J., & Guibas, L. J. 

*[Robust Learning Through Cross-Task Consistency](https://openaccess.thecvf.com/content_CVPR_2020/html/Zamir_Robust_Learning_Through_Cross-Task_Consistency_CVPR_2020_paper.html)*,

CVPR, 2020. 

-  Georgescu, M. I., Barbalau, A., Ionescu, R. T., Khan, F. S., Popescu, M., & Shah, M. *[Anomaly Detection in Video via Self-Supervised and Multi-Task Learning](https://openaccess.thecvf.com/content/CVPR2021/papers/Georgescu_Anomaly_Detection_in_Video_via_Self-Supervised_and_Multi-Task_Learning_CVPR_2021_paper.pdf)*, CVPR, 2021.



## Other

-  Eftekhar, A.,Sax, A., Malik, J., Zamir, A. *[Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets From 3D Scans](https://openaccess.thecvf.com/content/ICCV2021/html/Eftekhar_Omnidata_A_Scalable_Pipeline_for_Making_Multi-Task_Mid-Level_Vision_Datasets_ICCV_2021_paper.html)*, ICCV, 2021.