{"id":13771421,"url":"https://github.com/facebookresearch/astmt","last_synced_at":"2025-05-11T04:30:42.040Z","repository":{"id":66082555,"uuid":"180285548","full_name":"facebookresearch/astmt","owner":"facebookresearch","description":"Attentive Single-tasking of Multiple Tasks","archived":true,"fork":false,"pushed_at":"2019-10-27T07:04:11.000Z","size":1813,"stargazers_count":83,"open_issues_count":1,"forks_count":15,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-02-23T00:19:49.527Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/facebookresearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-04-09T04:35:36.000Z","updated_at":"2024-12-16T21:51:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"3b05c360-c56e-473c-a0f5-e7ef1c92492b","html_url":"https://github.com/facebookresearch/astmt","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fastmt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fastmt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fastmt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fastmt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/facebookresearch","download_url":"https://codeload.github.com/facebookresearch/astmt/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253518941,"owners_count":21921074,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T17:00:51.315Z","updated_at":"2025-05-11T04:30:41.090Z","avatar_url":"https://github.com/facebookresearch.png","language":"Python","funding_links":[],"categories":["Codebase","Datasets","Benchmarks \u0026 Code"],"sub_categories":["Recommendation","Image Classification"],"readme":"# Attentive Single-Tasking of Multiple Tasks\nVisit our [project page](http://www.vision.ee.ethz.ch/~kmaninis/astmt) for accessing the paper, and the pre-computed results.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"doc/astmt.png\" width=\"500\"\u003e\n\u003c/p\u003e\n\nThis is the implementation (in PyTorch) of the following paper:\n \n*[Kevis-Kokitsi Maninis](http://www.vision.ee.ethz.ch/~kmaninis/), [Ilija Radosavovic](http://www.ilijar.com/), and [Iasonas Kokkinos](http://www0.cs.ucl.ac.uk/staff/I.Kokkinos/index.html).\n[\"Attentive Single-Tasking of Multiple Tasks\"](https://arxiv.org/abs/1904.08918), in CVPR 2019.*\n\n```\n@InProceedings{MRK19,\n  Author    = {Kevis-Kokitsi Maninis and Ilija Radosavovic and Iasonas Kokkinos},\n  Title     = {Attentive Single-Tasking of Multiple Tasks},\n  Booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},\n  Year      = {2019}\n}\n```\n\n### Abstract\nIn this work we address task interference in universal networks by considering that a network is trained on multiple tasks, but performs one task at a time, an approach we refer to as \"single-tasking multiple tasks\". The network thus modifies its behaviour through task-dependent feature adaptation, or task attention. This gives the network the ability to accentuate the features that are adapted to a task, while shunning irrelevant ones. We further reduce task interference by forcing the task gradients to be statistically indistinguishable through adversarial training, ensuring that the common backbone architecture serving all tasks is not dominated by any of the task-specific gradients.   \nResults in three multi-task dense labelling problems consistently show: (i)  a large reduction in the number of parameters while preserving, or even improving performance and (ii) a smooth trade-off between computation and multi-task accuracy.\n\n### Results\nExample performance on PASCAL Context, for ResNet-101 backbone (more results in the paper):\n\n|  Experiment   | Edge Detection (F) | Semantic Segmentation (mIoU) | Human Parts (mIoU) | Surface Normals (mErr) | Saliency (mIoU)| Average Drop (%)|\n| ------------  | ------------------ | ---------------------------- | ------------------ | ---------------------- | -------------- | --------------- |\n|  Single Task  |        72.7        |            68.30             |       60.70        |           14.61        |      65.40     |          -      |\n| MTL w/o ASTMT |        69.2        |            63.20             |       55.10        |           16.04        |      63.60     |         6.81    |\n| MTL w/ ASTMT  |        72.4        |            68.00             |       61.12        |           14.68        |      65.71     |         0.04    |\n \n\u003cp align=\"center\"\u003e\n\u003cimg src=\"doc/qual.png\"\u003e\n\u003c/p\u003e\n\n\n### License\n\nThis repository is released under the following [LICENSE](https://github.com/facebookresearch/astmt/blob/master/LICENSE).\n\n###  Installation / Setup:\n\nThis code was tested with Python 3.6, PyTorch 0.4.1/1.0, and CUDA 9.0.\n\n0. Install PyTorch\n    ```\n    conda install pytorch=0.4.1 torchvision cuda90 -c pytorch\n    ```\n\n1. Install additional dependencies.\n    ```\n    conda install scikit-image pillow\n    pip install graphviz opencv-python easydict pycocotools\n    pip install tensorboard tensorboardx tensorflow\n    ```\n    \n2. Clone the multi-task repo.\n    ```\n    git clone https://github.com/facebookresearch/astmt.git\n    ```\n\n3. Add the multi-task package directory, easiest if you initialize it into your ~/.bashrc.\n    ```\n    echo \"export PYTHONPATH=$PYTHONPATH:/path/to/this/repo/\" \u003e\u003e ~/.bashrc\n    ```\n    \n4. Move (copy) `mypath.py` underd `util/`. Complete the `/path/to/something/` paths. \n\n\n\n### Experiments\n\nThe experiments that result in the findings of the paper can be found under `experiments/dense_predict/`.\nFor example, in order to test ASTMT on PASCAL (5 tasks) using modulation (squeeze and excitation, and residual adapters), and adversarial loss, run:\n```\ncd pascal_resnet\npython main.py --resume_epoch 60\n```\n\nTo train the model run:\n```\npython main.py \n```\nThis will train a resnet26 version of ASTMT. To use different setup (deeper network, etc.) please check `config.py` inside each experiment.\n\n#### Use Tensorboard\n- `cd /path/to/experiment`\n- `tensorboard --logdir . --port XXXX` .If you are using port forwarding to your local machine, access through `localhost:XXXX`.\n\n#### Evaluation\nEvaluation scripts are run at the end of each experiment and the results are dumped into the experiment's folder.\nThe evaluation part of boundary detection is disabled by default, since we used the MATLAB-based repo of [seism](https://github.com/jponttuset/seism).\n\nBriefly, the following metrics has been implemented:\n- `edge`: F-measures per dataset (odsF), F-measure per Instance (oisF), and Average Precision (AP).\n- `semseg`: mean Intersection over Union (mIoU) per class.\n- `human_part`: mean Intersection over Union (mIoU) per class.\n- `normals`: Difference in predicted and ground-truth angles (mean, median, RMSE, \u003c 11.25, \u003c 22.5, \u003c 30).\n- `sal`: Maximal mean Intersection over Union (max mIoU) and maximal F (maxF).\n- `albedo`: Mean Squared Error (MSE).\n- `depth`: Mean Squared Error (MSE).\n\nEnjoy :)\n\n\n \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2Fastmt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffacebookresearch%2Fastmt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2Fastmt/lists"}