{"id":16495020,"url":"https://github.com/nachifur/dmtn","last_synced_at":"2025-03-21T08:30:30.636Z","repository":{"id":113289078,"uuid":"539531731","full_name":"nachifur/DMTN","owner":"nachifur","description":"IEEE TMM 2023: A Decoupled Multi-Task Network for Shadow Removal","archived":false,"fork":false,"pushed_at":"2024-05-19T11:48:22.000Z","size":1338,"stargazers_count":19,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-17T22:44:27.916Z","etag":null,"topics":["shadow","shadow-removal"],"latest_commit_sha":null,"homepage":"https://ieeexplore.ieee.org/document/10058544","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nachifur.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-21T14:30:16.000Z","updated_at":"2025-02-28T03:12:11.000Z","dependencies_parsed_at":null,"dependency_job_id":"7eb14145-95fa-4c2a-92f1-ae392672e02d","html_url":"https://github.com/nachifur/DMTN","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nachifur%2FDMTN","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nachifur%2FDMTN/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nachifur%2FDMTN/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nachifur%2FDMTN/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nachifur","download_url":"https://codeload.github.com/nachifur/DMTN/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244764512,"owners_count":20506653,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["shadow","shadow-removal"],"created_at":"2024-10-11T14:27:07.099Z","updated_at":"2025-03-21T08:30:29.992Z","avatar_url":"https://github.com/nachifur.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DMTN\n\n\u003cimg width=\"512\" height=\"512\" src=\"https://github.com/nachifur/DMTN/blob/main/img/fig1.jpg\"/\u003e\n\n# 1. Resources\n[国内资源链接(密码：e2ww)](https://rec.ustc.edu.cn/share/16db9c80-15d5-11ef-9ca4-7b154f4fe8b0)\n\n## 1.1 Dataset\n* SRD ([github](https://github.com/Liangqiong/DeShadowNet) | [paper](https://openaccess.thecvf.com/content_cvpr_2017/papers/Qu_DeshadowNet_A_Multi-Context_CVPR_2017_paper.pdf))\n* ISTD ([github](https://github.com/DeepInsight-PCALab/ST-CGAN) | [paper](http://openaccess.thecvf.com/content_cvpr_2018/papers/Wang_Stacked_Conditional_Generative_CVPR_2018_paper.pdf))\n* ISTD+DA, SRD+DA ([github](https://github.com/vinthony/ghost-free-shadow-removal) | [paper](https://arxiv.org/abs/1911.08718))\n* [SSRD](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/EckYI_84wMdJpVgf5EqkmSABmnOD_-53YZ6v2KIQsiLeXA?e=67Gj47)\n\nThe SSRD dataset does not contain the ground truth of shadow-free images due to the presence of self shadow in images.\n\n## 1.2 Results | Model Weight\n\n**TEST RESULTS ON SRD:**\n* Results on SRD: [DMTN_SRD](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/EQt3ZoJAbZ5Cq_mhHxzrUVYBcsiaPLjnsN-SmhotYz-UOg?e=hS2RkJ) | Weight: [DMTN_SRD.pth](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/EWc4B9PP-rtGp4LxPWGOkfoB6oi6Coh1tu-qG5qxBk-7Cg?e=sdlSiA)\n* Results on SRD: [DMTN+Mask_SRD](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/EU0JfEPuOUNNlSQDc0TfYgQBOAtMpXjK5yRoa3q2H_bcnQ?e=MgsEu5) | Weight: [DMTN+Mask_SRD.pth](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/EU4NJ0CPbwpBrzyXH5FLlMEBzqwKhcXlxe8k4vQiXRrJUw?e=FPKlfF)\n\n**TEST RESULTS ON ISTD:**\n* Results on ISTD: [DMTN_ISTD](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/EbyzaV72N2FElC5nOsp3-ZYBuUoVLiy29rmXBMXVXXY6Lg?e=wKA55D) | Weight: [DMTN_ISTD.pth](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/EROUwnLgz9BGi3OtJa5SIs8BwdgYBZTXeMJ1NLcGfHAwCg?e=v1f51U)\n* Results on ISTD: [DMTN+Mask_ISTD](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/EcRgA4y1UAZIkRIabbm71iIBNhRH-JIugQDbInyWE3rpNQ?e=D8BiGD) | Weight: [DMTN+Mask_ISTD.pth](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/EcnDQNKeoRdBtUYjQdirl34BR73n--qRnFIo6RnxPvk-KQ?e=9V0LIR)\n* Results on ISTD+DA: [DMTN_ISTD_DA](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/ERGASEyFybBDm9rYZv4a3I4B6FwMmrhZMImk_-b7Lo-YeQ?e=MbzrMk) | Weight: [DMTN_ISTD_DA.pth](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/EWwqrUr7Qh9KugvJ2S5KsdMBYz6aiR-ufiX3kn3zB626lg?e=7QJRra)\n\n**TEST RESULTS ON ISTD+:**\n* Results on ISTD+: [DMTN_ISTD+](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/EZnB81g7L3VPuGo2zhVclVEBPhsO6MBJYPtbOnqxmEDHuw?e=MZLmUM) | Weight: [DMTN_ISTD+.pth](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/ETVno1MtDsdLknqDNKq60VwB9Bq-oq8kZ8B8aiwQBZXbQQ?e=B0S37N)\n* Results on ISTD+: [DMTN+Mask_ISTD+](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/EZEQr_hD7XdGgPiesl0L8aABSugt0z5U6V9q2Wv-fEr-VA?e=zq5A7s) | Weight: [DMTN+Mask_ISTD+.pth](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/ETo6UMeCGNhFjJ20o0RedaQBG7XIDcfbqucJ3A-hK6IQAQ?e=uN6sTs)\n\n**TEST RESULTS ON SSRD:** (DHAN and DMTN are pretrained on SRD dataset (size:420x320))\n\n* Results of DMTN on SSRD: [DMTN_SSRD](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/ET7vtW6b-RNFiK7hJe8coXoBjMMUj2vZ4nEj1SitH8wuKA?e=ZDnfYV) | Weight: [DMTN_SRD_420_320.pth](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/EQgZbEFJCLZGiAM8rnbE-ZUBHXw3zyTrhdet7JDSCrYiuA?e=6PcofV)\n* Results of DHAN on SSRD:[DHAN_SSRD](https://mailustceducn-my.sharepoint.com/:u:/g/personal/nachifur_mail_ustc_edu_cn/EROyGJwa2C5JkO1bLDVV_AsBbXRPKoZbBy5EsjAsz6xujg?e=nw53O6)\n\n\n## 1.3 Visual results\n\n\n*Visual comparison results of **penumbra** removal on the SRD dataset - (Powered by [MulimgViewer](https://github.com/nachifur/MulimgViewer))*\n\n\u003cimg src=\"https://github.com/nachifur/DMTN/blob/main/img/fig2.jpg\"/\u003e\n\n*Visual comparison results of **self shadow** removal on the SSRD dataset - (Powered by [MulimgViewer](https://github.com/nachifur/MulimgViewer))*\n\n\u003cimg src=\"https://github.com/nachifur/DMTN/blob/main/img/fig3.jpg\"/\u003e\n\n\n## 1.4 Evaluation Code\nCurrently, MATLAB evaluation codes are used in most state-of-the-art works for shadow removal.\n\n[Our evaluation code](https://github.com/nachifur/DMTN/blob/main/evaluation/evaluate_MAE_PSNR_SSIM.m) (i.e., 1+2)\n1. MAE (i.e., RMSE in paper): https://github.com/tsingqguo/exposure-fusion-shadow-removal\n2. PSNR+SSIM: https://github.com/zhuyr97/AAAI2022_Unfolding_Network_Shadow_Removal/tree/master/codes\n\nNotably, there are slight differences between the different evaluation codes.\n* [wang_cvpr2018](https://github.com/DeepInsight-PCALab/ST-CGAN), [le_iccv2019](https://github.com/cvlab-stonybrook/SID): no imresize;\n* [fu_cvpr2021](https://github.com/tsingqguo/exposure-fusion-shadow-removal): first imresize, then double;\n* [zhu_aaai2022](https://github.com/zhuyr97/AAAI2022_Unfolding_Network_Shadow_Removal): first double, then imresize;\n* Our evaluation code: MAE-\u003efu_cvpr2021, psnr+ssim-\u003ezhu_aaai2022\n\n# 2. Environments\n**ubuntu18.04+cuda10.2+pytorch1.7.1**\n1. create environments\n```\nconda env create -f install.yaml\n```\n2. activate environments\n```\nconda activate DMTN\n```\n\n# 3. Data Processing\nFor example, generate the dataset list of ISTD:\n1. Download:\n   * ISTD and SRD\n   * [USR shadowfree images](https://github.com/xw-hu/Mask-ShadowGAN)\n   * [Syn. Shadow](https://github.com/vinthony/ghost-free-shadow-removal)\n   * [SRD shadow mask](https://github.com/vinthony/ghost-free-shadow-removal)\n   * train_B_ISTD:\n   ```\n   cp -r ISTD_Dataset_arg/train_B ISTD_Dataset_arg/train_B_ISTD\n   cp -r ISTD_Dataset_arg/train_B SRD_Dataset_arg/train_B_ISTD\n   ```\n   * [VGG19](https://download.pytorch.org/models/vgg19-dcbb9e9d.pth)\n   ```\n   cp vgg19-dcbb9e9d.pth ISTD_Dataset_arg/\n   cp vgg19-dcbb9e9d.pth SRD_Dataset_arg/\n   ```\n2. The data folders should be:\n    ```\n    ISTD_Dataset_arg\n        * train\n            - train_A # ISTD shadow image\n            - train_B # ISTD shadow mask\n            - train_C # ISTD shadowfree image\n            - shadow_free # USR shadowfree images\n            - synC # Syn. shadow\n            - train_B_ISTD # ISTD shadow mask\n        * test\n            - test_A # ISTD shadow image\n            - test_B # ISTD shadow mask\n            - test_C # ISTD shadowfree image\n        * vgg19-dcbb9e9d.pth\n\n    SRD_Dataset_arg\n        * train #  renaming the original `Train` folder in `SRD`.\n            - train_A # SRD shadow image, renaming the original `shadow` folder in `SRD`.\n            - train_B # SRD shadow mask\n            - train_C # SRD shadowfree image, renaming the original `shadow_free` folder in `SRD`.\n            - shadow_free # USR shadowfree images\n            - synC # Syn. shadow\n            - train_B_ISTD # ISTD shadow mask\n        * test #  renaming the original `test_data` folder in `SRD`.\n            - train_A # SRD shadow image, renaming the original `shadow` folder in `SRD`.\n            - train_B # SRD shadow mask\n            - train_C # SRD shadowfree image, renaming the original `shadow_free` folder in `SRD`.\n        * vgg19-dcbb9e9d.pth \n    ```\n3. Edit `generate_flist_istd.py`: (Replace path)\n\n```\nISTD_path = \"/Your_data_storage_path/ISTD_Dataset_arg\"\n```\n4. Generate Datasets List. (Already contains ISTD+DA.)\n```\nconda activate DMTN\ncd script/\npython generate_flist_istd.py\n```\n5. Edit `config_ISTD.yml`: (Replace path)\n```\nDATA_ROOT: /Your_data_storage_path/ISTD_Dataset_arg\n```\n\n# 4. Training+Test+Evaluation\n## 4.1 Training+Test+Evaluation\nFor example, training+test+evaluation on ISTD dataset.\n```\ncp config/config_ISTD.yml config.yml \ncp config/run_ISTD.py run.py\nconda activate DMTN\npython run.py\n```\n## 4.2 Only Test and Evaluation\nFor example, test+evaluation on ISTD dataset.\n1. Download weight file(`DMTN_ISTD.pth`) to `pre_train_model/ISTD`\n2. Copy file\n```\ncp config/config_ISTD.yml config.yml \ncp config/run_ISTD.py run.py\nmkdir -p checkpoints/ISTD/\ncp config.yml checkpoints/ISTD/config.yml\ncp pre_train_model/ISTD/DMTN_ISTD.pth  checkpoints/ISTD/ShadowRemoval.pth\n```\n\n3. Edit `run.py`. Comment the training code.\n\n```\n    # # pre_train (no data augmentation)\n    # MODE = 0\n    # print('\\nmode-'+str(MODE)+': start pre_training(data augmentation)...\\n')\n    # for i in range(1):\n    #     skip_train = init_config(checkpoints_path, MODE=MODE,\n    #                             EVAL_INTERVAL_EPOCH=1, EPOCH=[90,i])\n    #     if not skip_train:\n    #         main(MODE, config_path)\n    # src_path = Path('./pre_train_model') / \\\n    #     config[\"SUBJECT_WORD\"]/(config[\"MODEL_NAME\"]+'_pre_da.pth')\n    # copypth(dest_path, src_path)\n\n    # # train\n    # MODE = 2\n    # print('\\nmode-'+str(MODE)+': start training...\\n')\n    # for i in range(1):\n    #     skip_train = init_config(checkpoints_path, MODE=MODE,\n    #                             EVAL_INTERVAL_EPOCH=0.1, EPOCH=[60,i])\n    #     if not skip_train:\n    #         main(MODE, config_path)\n    # src_path = Path('./pre_train_model') / \\\n    #     config[\"SUBJECT_WORD\"]/(config[\"MODEL_NAME\"]+'_final.pth')\n    # copypth(dest_path, src_path)\n```\n4. Run\n\n```\nconda activate DMTN\npython run.py\n```\n## 4.3 Show Results\nAfter evaluation, execute the following code to display the final RMSE.\n```\npython show_eval_result.py\n```\nOutput:\n```\nrunning rmse-shadow: xxx, rmse-non-shadow: xxx, rmse-all: xxx # ISRD\n```\nThis is the evaluation result of python+pytorch, which is only used during training. To get the evaluation results in the paper, you need to run the [matlab code](#1.4).\n\n## 4.4 Test on SSRD\n1. Edit `src/network/network_DMTN.py`. Modify the line (https://github.com/nachifur/DMTN/blob/main/src/network/network_DMTN.py#L339).\n```\nSSRD = True\n```\n2. Test like the section `4.2 Only Test and Evaluation`.\n\n# 5. Acknowledgements\nPart of the code is based upon:\n* https://github.com/nachifur/LLPC\n* https://github.com/vinthony/ghost-free-shadow-removal\n* https://github.com/knazeri/edge-connect\n\n# 6. Citation\n```\n@ARTICLE{liu2023decoupled,\n  author={Liu, Jiawei and Wang, Qiang and Fan, Huijie and Li, Wentao and Qu, Liangqiong and Tang, Yandong},\n  journal={IEEE Transactions on Multimedia}, \n  title={A Decoupled Multi-Task Network for Shadow Removal}, \n  year={2023},\n  volume={},\n  number={},\n  pages={1-14},\n  doi={10.1109/TMM.2023.3252271}}\n```\n# 7. Contact\nPlease contact Jiawei Liu if there is any question (liujiawei18@mails.ucas.ac.cn).\n\n# 8. Revised Errors in the Paper\nSorry! Here are the revised errors:\n1. In Section III-C-2)-`Fig. 7 (or Fig. 5(b)) shows...`, \"we can achieve feature decoupling, i.e., some channels of F represent shadow images (~~`I_m`~~ `I_s`)\".\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnachifur%2Fdmtn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnachifur%2Fdmtn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnachifur%2Fdmtn/lists"}