{"id":37762013,"url":"https://github.com/nik-dim/pamal","last_synced_at":"2026-01-16T14:38:02.350Z","repository":{"id":175590893,"uuid":"653555294","full_name":"nik-dim/pamal","owner":"nik-dim","description":"Official repository of \"Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task models\" [ICML 2023]","archived":false,"fork":false,"pushed_at":"2025-01-10T20:13:41.000Z","size":314,"stargazers_count":15,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-10T21:23:19.340Z","etag":null,"topics":["icml-2023","mtl","multi-task-learning","multitask-learning","pareto-front","weight-ensemble"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2210.09759","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nik-dim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-14T09:22:40.000Z","updated_at":"2025-01-10T20:13:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"caf070fa-d38f-4b63-a6a6-11a633316bf7","html_url":"https://github.com/nik-dim/pamal","commit_stats":null,"previous_names":["nik-dim/pamal"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nik-dim/pamal","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nik-dim%2Fpamal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nik-dim%2Fpamal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nik-dim%2Fpamal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nik-dim%2Fpamal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nik-dim","download_url":"https://codeload.github.com/nik-dim/pamal/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nik-dim%2Fpamal/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28479399,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T11:59:17.896Z","status":"ssl_error","status_checked_at":"2026-01-16T11:55:55.838Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["icml-2023","mtl","multi-task-learning","multitask-learning","pareto-front","weight-ensemble"],"created_at":"2026-01-16T14:38:01.738Z","updated_at":"2026-01-16T14:38:02.344Z","avatar_url":"https://github.com/nik-dim.png","language":"Python","readme":"# PaMaL\nOfficial implementation of [Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task models](https://arxiv.org/abs/2210.09759). \n\n![PaMaL](docs/media/img.png)\n## Installation\n\n```bash\ngit clone https://github.com/nik-dim/pamal.git \u0026\u0026 cd pamal\n\n# Method 1: Conda\nconda create -n pamal python=3.10\nconda activate pamal\npip install -r requirements.txt\n\n# Method 2: Dockerfile\ndocker build -t pamal .\n```\n\n\n\n## Running Experiments\n\nThe experiments are configured using [Hydra](https://hydra.cc/). Hydra uses a hierarchical structure to define experiments. See [here](https://hydra.cc/docs/intro/) for more details. \n\nThe configuration files are in the [`configs/`](configs/) directory and are further split into two directories. The first, [`configs/general/`](configs/general/), houses the definitions of methods, optimizers, datasets etc, while the second one,  [`configs/experiment/`](configs/experiment/) makes uses of these definitions to configure the experiments for each dataset. The supported methods are outlined [here](#supported-methods) and the supported datasets [here](#supported-datasets).\n\n\n\nFor example, we can run experiments on the MultiMNIST dataset as follows:\n\n```bash\n# run experiment on MultiMNIST using the proposed method (PaMaL) \npython multimnist.py method=pamal data.root=/path/to/data\n\n# ...modify the `inner_method` parameter specific to pamal\npython multimnist.py method=pamal data.root=/path/to/data method.inner_method=ls # or gradnorm or rwa\n\n# run the same experiment for a baseline\npython multimnist.py data.root=/path/to/data method=ls # or mgda or phn etc\n```\n\n\nTo launch multiple experiments at the same time (e.g. ablation studies), we can use the `--multirun` (`-m`) flag. Note that this flag is sometimes problematic in conjunction with Weights\u0026Biases logging. For example, to run the ablation study for the proposed method, we can run:\n\n```bash\n# launch the previous experiment for multiple seeds\npython multimnist.py -m method=pamal seed=0,1,2\n```\nAlternatively, check the [run.sh](run.sh) script for launching multiple experiments.\n\n## Supported Methods\n\nThis repository supports the methods outlined in the table below. \nApart from the proposed method `pamal`, we support Pareto Front Approximation methods `phn` and `cosmos`, as *single solution* methods, such as `rlw`, `graddrop`, `autol`, `rotograd`, `cagrad`, `nashmtl`, `pcgrad`, `dwa`, `si`, `mgda`, `uw`, `ls`, `stl`.\n\nIn order to select one baseline set `method=XXX` where `XXX` is the ticker presented below. For example, to run the `ls` baseline, run `python census.py method=ls`.\n\n\n| Ticker   | Paper                                                                                                                      |\n|----------|----------------------------------------------------------------------------------------------------------------------------|\n| stlX     | Single-Task Learning for task X, X=0,1,2...                                                                                |\n| ls       | Fixed Weighting (e.g. equal)                                                                                               |\n| uw       | [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://arxiv.org/abs/1705.07115) |\n| mgda     | [Multi-Task Learning as Multi-Objective Optimization](https://arxiv.org/abs/1810.04650)                                    |\n| si       | See [Multi-Task Learning as a Bargaining Game](https://arxiv.org/abs/2202.01017) for details                               |\n| dwa      | [End-to-End Multi-Task Learning with Attention](https://arxiv.org/abs/1803.10704)                                          |\n| pcgrad   | [Gradient Surgery for Multi-Task Learning](https://arxiv.org/abs/2001.06782)                                               |\n| imtl     | [Towards Impartial Multi-task Learning](https://openreview.net/forum?id=IMPnRXEWpvr)                                       |\n| cagrad   | [Conflict-Averse Gradient Descent for Multi-task Learning](https://arxiv.org/pdf/2110.14048.pdf)                           |\n| nashmtl  | [Multi-Task Learning as a Bargaining Game](https://arxiv.org/abs/2202.01017)                                               |\n| rlw      | [A Closer Look at Loss Weighting in Multi-Task Learning](https://arxiv.org/pdf/2111.10603.pdf)                             |\n| graddrop | [Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout](https://arxiv.org/abs/2010.06808)          |\n| autol    | [Auto-Lambda: Disentangling Dynamic Task Relationships](https://arxiv.org/abs/2202.03091)                                  |\n| rotograd | [RotoGrad: Gradient Homogenization in Multitask Learning](https://arxiv.org/abs/2103.02631)                                |\n| phn      | [Learning the Pareto Front with Hypernetworks](https://arxiv.org/abs/2010.04104)                                           |\n| cosmos   | [Scalable Pareto Front Approximation for Deep Multi-Objective Learning](https://arxiv.org/abs/2103.13392)                  |\n| pamal    | [Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task models](https://arxiv.org/abs/2210.09759)  |\n\n\n\n\n\n## Supported Datasets\nThe repository supports the datasets in the table below. By default, the datasets will be downloaded to the `~/benchmarks/data` directory. The root path can be modified in [`src/datasets/base_data_module.py`](src/datasets/base_data_module.py).\n\n| Dataset     | Notes                  |\n|-------------|------------------------|\n| MultiMNIST  | needs to be created    |\n| Census      | needs to be downloaded |\n| MultiMNIST3 | needs to be created    |\n| UtkFace     | needs to be downloaded |\n| Cityscapes  | needs to be downloaded |\n| CelebA      | needs to be downloaded |\n\n\n## Extending the codebase\nThe code is structured similar to PyTorch Lightning. We have used various codebase sources for baselines. For this reason, the API is not consistent for all baselines. Most baselines can be found in [`src/methods/weight_methods.py`](src/methods/weight_methods.py). However, `graddrop`, `autol`, `rotograd`, `phn` and `cosmos` are implemented in their own specific way. In order to add a new `single solution` method, check [`src/methods/weight_methods.py`](src/methods/weight_methods.p) where multiple loss and gradient balancing methods can be found. For `multi solution` methods check [`src/trainer/multi_solution_trainer.py`](src/trainer/multi_solution_trainer.py).\n\nIf you have any questions, please contact me. We plan to merge and unify the baselines, with method-specific callbacks. For the moment, this is the case only for Auto-$\\lambda$ in [`src/utils/callbacks/auto_lambda_callback.py`](src/utils/callbacks/auto_lambda_callback.py).\n\n## References\nThe reporsitory makes heavy use of previously developped codebases. Specifically, we use:\n1. https://github.com/lorenmt/mtan\n2. https://github.com/AvivNavon/nash-mtl\n3. https://github.com/AvivNavon/pareto-hypernetworks\n4. https://github.com/ruchtem/cosmos\n5. https://github.com/adrianjav/rotograd\n6. https://github.com/intel-isl/MultiObjectiveOptimization\n\n\n## Citation\nIf you find our work useful, please consider citing our paper:\n```bibtex\n@inproceedings{\n    dimitriadis2023pareto,\n    title={Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task models},\n    author={Dimitriadis, Nikolaos and Frossard, Pascal and Fleuret, Fran{\\c{c}}ois},\n    booktitle={International Conference on Machine Learning},\n    year={2023},\n    url={https://arxiv.org/abs/2210.09759}\n}\n```\n\n## Contact\nIf you have any questions, please contact nikolaos.dimitriadis@epfl.ch. ","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnik-dim%2Fpamal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnik-dim%2Fpamal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnik-dim%2Fpamal/lists"}