{"id":13737906,"url":"https://github.com/lorenmt/auto-lambda","last_synced_at":"2025-05-08T15:32:03.442Z","repository":{"id":41554921,"uuid":"431073265","full_name":"lorenmt/auto-lambda","owner":"lorenmt","description":"The Implementation of \"Auto-Lambda: Disentangling Dynamic Task Relationships\" [TMLR 2022].","archived":false,"fork":false,"pushed_at":"2023-02-05T01:48:53.000Z","size":70,"stargazers_count":122,"open_issues_count":0,"forks_count":16,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-05-22T11:33:46.343Z","etag":null,"topics":["auxiliary-learning","meta-learning","multi-task-learning"],"latest_commit_sha":null,"homepage":"https://shikun.io/projects/auto-lambda","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lorenmt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-11-23T11:25:19.000Z","updated_at":"2024-05-21T13:13:24.000Z","dependencies_parsed_at":"2024-01-13T00:38:51.615Z","dependency_job_id":"2db3779a-4e9a-4261-b881-309edee07c88","html_url":"https://github.com/lorenmt/auto-lambda","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorenmt%2Fauto-lambda","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorenmt%2Fauto-lambda/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorenmt%2Fauto-lambda/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorenmt%2Fauto-lambda/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lorenmt","download_url":"https://codeload.github.com/lorenmt/auto-lambda/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253096249,"owners_count":21853566,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["auxiliary-learning","meta-learning","multi-task-learning"],"created_at":"2024-08-03T03:02:05.363Z","updated_at":"2025-05-08T15:32:03.168Z","avatar_url":"https://github.com/lorenmt.png","language":"Python","funding_links":[],"categories":["Python","Codebase","Benchmarks \u0026 Code"],"sub_categories":["Recommendation","Image Classification"],"readme":"# Auto-Lambda\nThis repository contains the source code of Auto-Lambda and baselines from the paper, [Auto-Lambda: Disentangling Dynamic Task Relationships](https://arxiv.org/abs/2202.03091). \n\nWe encourage readers to check out our [project page](https://shikun.io/projects/auto-lambda), including more interesting discussions and insights which are not covered in our technical paper.\n\n## Multi-task Methods\nWe implemented all weighting and gradient-based baselines presented in the paper for computer vision tasks: Dense Prediction Tasks (for NYUv2 and CityScapes) and Multi-domain Classification Tasks (for CIFAR-100). \n\nSpecifically, we have covered the implementation of these following multi-task optimisation methods:\n\n### Weighting-based:\n- **Equal** - All task weightings are 1. `--weight equal`\n- **Uncertainty** - [https://arxiv.org/abs/1705.07115](https://arxiv.org/abs/1705.07115) `--weight uncert`\n- **Dynamic Weight Average** - [https://arxiv.org/abs/1803.10704](https://arxiv.org/abs/1803.10704) `--weight dwa`\n- **Auto-Lambda** - Our approach. `--weight autol`\n\n### Gradient-based:\n- **GradDrop** -  [https://arxiv.org/abs/2010.06808](https://arxiv.org/abs/2010.06808) `--grad_method graddrop`\n- **PCGrad** - [https://arxiv.org/abs/2001.06782](https://arxiv.org/abs/2001.06782) `--grad_method pcgrad`\n- **CAGrad** - [https://arxiv.org/abs/2110.14048](https://arxiv.org/abs/2110.14048) `--grad_method cagrad`\n\n*Note: Applying a combination of both weighting and gradient-based methods can further improve performance.*\n\n## Datasets\nWe applied the same data pre-processing following our previous project: [MTAN](https://github.com/lorenmt/mtan) which experimented on:\n\n- [**NYUv2 [3 Tasks]**](https://www.dropbox.com/sh/86nssgwm6hm3vkb/AACrnUQ4GxpdrBbLjb6n-mWNa?dl=0)  - 13 Class Segmentation + Depth Estimation + Surface Normal. [288 x 384] Resolution.\n- [**CityScapes [3 Tasks]**](https://www.dropbox.com/sh/qk3cr18d55d08gj/AAA5OCTPNFDEDk5fZsmCfmrAa?dl=0) - 19 Class Segmentation + 10 Class Part Segmentation + Disparity (Inverse Depth) Estimation. [256 x 512] Resolution.\n\n*Note: We have included a new task: [Part Segmentation](https://github.com/pmeletis/panoptic_parts) for CityScapes dataset. Please install the `pip install panoptic_parts` for CityScapes experiments.  The pre-processing file for CityScapes has also been included in the `dataset` folder.*\n\n\n## Experiments\nAll experiments were written in `PyTorch 1.7` and can be trained with different flags (hyper-parameters) when running each training script. We briefly introduce some important flags below. \n\n| Flag Name     | Usage                                                                                                                                    | Comments                                                                            |\n|---------------|------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------|\n| `network`     | choose multi-task network: `split, mtan`                                                                                                 | both architectures are based on ResNet-50; only available in dense prediction tasks |\n| `dataset`     | choose dataset: `nyuv2, cityscapes`                                                                                                      | only available in dense prediction tasks                                            |\n| `weight`      | choose weighting-based method: `equal, uncert, dwa, autol`                                                                               | only `autol` will behave differently when set to different primary tasks            |\n| `grad_method` | choose gradient-based method: `graddrop, pcgrad, cagrad`                                                                                 | `weight` and `grad_method` can be applied together                                  |\n| `task`        | choose primary tasks: `seg, depth, normal` for NYUv2, `seg, part_seg, disp` for CityScapes, `all`: a combination of all standard 3 tasks | only available in dense prediction tasks                                            |\n| `with_noise`  | toggle on to add noise prediction task for training (to evaluate robustness in auxiliary learning setting)                               | only available in dense prediction tasks                                            |\n| `subset_id`   | choose domain ID for CIFAR-100, choose `-1` for the multi-task learning setting                                                          | only available in CIFAR-100 tasks                                                   |\n| `autol_init`  | initialisation of Auto-Lambda, default `0.1`                                                                                             | only available when applying Auto-Lambda                        |\n| `autol_lr`    | learning rate of Auto-Lambda, default `1e-4`  for NYUv2 and `3e-5` for CityScapes                                                        | only available when applying Auto-Lambda                       |\n\nTraining Auto-Lambda in Multi-task / Auxiliary Learning Mode:\n```\npython trainer_dense.py --dataset [nyuv2, cityscapes] --task [PRIMARY_TASK] --weight autol --gpu 0   # for NYUv2 or CityScapes dataset\npython trainer_cifar.py --subset_id [PRIMARY_DOMAIN_ID] --weight autol --gpu 0   # for CIFAR-100 dataset\n```\n\nTraining in Single-task Learning Mode:\n```\npython trainer_dense_single.py --dataset [nyuv2, cityscapes] --task [PRIMARY_TASK]  --gpu 0   # for NYUv2 or CityScapes dataset\npython trainer_cifar_single.py --subset_id [PRIMARY_DOMAIN_ID] --gpu 0   # for CIFAR-100 dataset\n```\n\n*Note: All experiments in the original paper were trained from scratch without pre-training.*\n\n## Benchmark\nFor standard 3 tasks in NYUv2 (without noise prediction task) in the multi-task learning setting with Split architecture, please follow the results below.\n\n| Method               | Type | Sem. Seg. (mIOU) | Depth (aErr.) | Normal (mDist.) | Delta MTL |\n|----------------------|-------|-----------|---------------|-----------------|-----------|\n|Single | - | 43.37\t| 52.24\t        |22.40| - |\n| Equal\t               | W |44.64\t           | 43.32\t        | 24.48\t          | +3.57%    |\n| DWA\t                 | W |45.14            | \t43.06        | \t24.17          | \t+4.58%   |\n| GradDrop             | G |45.39            | 43.23         | 24.18           | +4.65%    |\n| PCGrad               | G | 45.15            | 42.38         | 24.13           | +5.09%    |\n| Uncertainty          | W |\t45.98           | \t41.26        | \t24.09          | \t+6.50%   |\n| CAGrad               | G |46.14            | 41.91         | 23.52           | +7.05%    |\n| Auto-Lambda          | W |\t47.17           | \t40.97\t       | 23.68           | \t+8.21%   |\n| Auto-Lambda + CAGrad | W + G|\t48.26           | \t39.82\t       | 22.81           | \t+11.07%  |\n\n*Note: The results were averaged across three random seeds. You should expect the error range less than +/-1%.*\n\n## Citation\nIf you found this code/work to be useful in your own research, please considering citing the following:\n\n```\n@article{liu2022auto_lambda,\n    title={Auto-Lambda: Disentangling Dynamic Task Relationships},\n    author={Liu, Shikun and James, Stephen and Davison, Andrew J and Johns, Edward},\n    journal={Transactions on Machine Learning Research},\n    year={2022}\n}\n```\n\n## Acknowledgement\nWe would like to thank [@Cranial-XIX](https://github.com/Cranial-XIX) for his clean implementation for gradient-based optimisation methods.\n\n## Contact\nIf you have any questions, please contact `sk.lorenmt@gmail.com`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Florenmt%2Fauto-lambda","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Florenmt%2Fauto-lambda","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Florenmt%2Fauto-lambda/lists"}