{"id":21027669,"url":"https://github.com/vinthony/ghost-free-shadow-removal","last_synced_at":"2025-04-06T09:11:01.566Z","repository":{"id":39663041,"uuid":"221343624","full_name":"vinthony/ghost-free-shadow-removal","owner":"vinthony","description":"[AAAI 2020] Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN","archived":false,"fork":false,"pushed_at":"2023-12-12T09:09:39.000Z","size":3981,"stargazers_count":310,"open_issues_count":13,"forks_count":60,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-30T08:10:36.315Z","etag":null,"topics":["data-augmentation","deep-learning","scene-understanding","shadow-removal","tensorflow"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1911.08718","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vinthony.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-13T01:08:44.000Z","updated_at":"2025-03-28T17:27:12.000Z","dependencies_parsed_at":"2024-11-19T11:58:44.121Z","dependency_job_id":"bb3d8df8-8675-482b-bbcd-04dba54e29fc","html_url":"https://github.com/vinthony/ghost-free-shadow-removal","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vinthony%2Fghost-free-shadow-removal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vinthony%2Fghost-free-shadow-removal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vinthony%2Fghost-free-shadow-removal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vinthony%2Fghost-free-shadow-removal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vinthony","download_url":"https://codeload.github.com/vinthony/ghost-free-shadow-removal/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247457803,"owners_count":20941906,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-augmentation","deep-learning","scene-understanding","shadow-removal","tensorflow"],"created_at":"2024-11-19T11:52:03.601Z","updated_at":"2025-04-06T09:11:01.536Z","avatar_url":"https://github.com/vinthony.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Towards Ghost-free Shadow Removal \n\nThis repo contains the code and results of the AAAI 2020 paper:\n\n\u003ci\u003e\u003cb\u003e [Towards Ghost-free Shadow Removal via \u003cbr\u003e Dual Hierarchical Aggregation Network and Shadow Matting GAN ](https://arxiv.org/abs/1911.08718)\u003c/b\u003e\u003c/i\u003e\u003cbr\u003e\n[Xiaodong Cun](http://vinthony.github.io), [Chi-Man Pun\u003csup\u003e*\u003c/sup\u003e](http://www.cis.umac.mo/~cmpun/), Cheng Shi \u003cbr\u003e\n[University of Macau](http://um.edu.mo/)\n\n[Syn. Datasets](#Resources) | [Models](#Resources) | [Results](#Resources) | [Paper](https://arxiv.org/abs/1911.08718) | [Supp.](https://github.com/vinthony/academicpages.github.io/blob/master/images/ghost-free-shadow-removal-supp.pdf) | Poster | [🔥Online Demo!(Google CoLab)](https://colab.research.google.com/drive/1cJ_dsBUXFaFtjoZB9gDYeahjmysnvnTq)\n\n\u003cimg width='100%' src='https://user-images.githubusercontent.com/4397546/69003615-582b2180-0940-11ea-9faa-2f2ae6b1d5ba.png'/\u003e\n\n\u003ci\u003eWe plot a result of our model with the input shown in the yellow square. From two zoomed regions, our method removes the shadow and reduces the ghost successfully.\u003c/i\u003e\n\n## Known Issues\n[#4](https://github.com/vinthony/ghost-free-shadow-removal/issues/4) inconsistency between the code and Figure.2, Thanks [@naoto0804](https://github.com/naoto0804)\n\n## SOME LINKS ARE BROKEN due to only OneDrive account, I will try to recover it soon.\n\n\n## **Introduction**\n\u003cp style=\"text-align:justify\"\u003e\u003ci\u003eShadow removal is an essential task for scene understanding. Many studies consider only matching the image contents, which often causes two types of ghosts: color in-consistencies in shadow regions or artifacts on shadow boundaries. In this paper, we try to tackle these issues in two aspects. On the one hand, to carefully learn the border artifacts-free image, we propose a novel network structure named the Dual Hierarchically Aggregation Network(DHAN). It contains a series of growth dilated convolutions as the backbone without any down-samplings, and we hierarchically aggregate multi-context features for attention and prediction respectively. On the other hand, we argue that training on a limited dataset restricts the textural understanding of the network, which leads to the shadow region color in-consistencies. Currently, the largest dataset contains 2k+ shadow/shadow-free images in pairs. However, it has only 0.1k+ unique scenes since many samples share exactly the same background with different shadow positions. Thus, we design a Shadow Matting Generative Adversarial Network~(SMGAN) to synthesize realistic shadow mattings from a given shadow mask and shadow-free image. With the help of novel masks or scenes, we enhance the current datasets using synthesized shadow images. Experiments show that our DHAN can erase the shadows and produce high-quality ghost-free images. After training on the synthesized and real datasets, our network outperforms other state-of-the-art methods by a large margin. \u003c/i\u003e\u003c/p\u003e\n\n## Sample Comparison\n![fig1857_5](https://user-images.githubusercontent.com/4397546/69911139-dbc13400-1451-11ea-8c1b-3b587b4f8727.png)\n\n\u003ci\u003eComparison of the shadow removal datasets, The first two samples are from the ISTD dataset while the bottom two samples are from the SRD dataset. In (d), the top two samples are from ST-CGAN and the bottom two samples are from DeShadowNet.\u003c/i\u003e\n\n\n## **Resources**\n\n- Pre-trained Models: \u003cb\u003e[SRD](https://uofmacau-my.sharepoint.com/:u:/g/personal/yb87432_umac_mo/EVjCDVbv4AhAsco1IqCTCnoBMVJt-f6pIFU603G0EEZ5CA?e=DDvg2v) | [SRD+](https://drive.google.com/uc?id=1rEIWWLwEpbZGPyFUc9jSIQr78ZeQy5eZ) | \n[ISTD](https://uofmacau-my.sharepoint.com/:u:/g/personal/yb87432_umac_mo/EdGF_2nSCZdMgbL0cz4aCt4BvEtAZ0xNsy81rloxJy5m7g?e=orI9i1) | \n[ISTD+](https://uofmacau-my.sharepoint.com/:u:/g/personal/yb87432_umac_mo/EYfdWFETlmZNqFn2gaz4juEBy6E-Rgyz7JoyabEv4iJQkw?e=poA4Wa) \u003c/b\u003e\n\n- Results on Shadow Removal: \u003cb\u003e\n[SRD](https://drive.google.com/file/d/1CJ2UTDGWm0xJEflz9tW327XqMcSkEF_D/view?usp=drive_link) | \n[SRD+](https://uofmacau-my.sharepoint.com/:u:/g/personal/yb87432_umac_mo/EYLodBImcw1AlfQZsh71HuYB_TalzP0uTBEtS-9atEdc_Q?e=DODEKk) | \n[ISTD](https://drive.google.com/file/d/1qoHVCuU4baRBhPVicNT6HVl4c4nJqiPw/view?usp=drive_link) | \n[ISTD+](https://drive.google.com/file/d/14iqUPebhJ0mnMw26_w67jp0LZgsxf9LK/view?usp=drive_link) \u003c/b\u003e\n\n- Results on Shadow Detection: \u003cb\u003e \n[SBU](https://uofmacau-my.sharepoint.com/:u:/g/personal/yb87432_umac_mo/EYByu0IMTQFHl__lK7GA1DAB0crwq0i49SIVLcdQWmnq_w?e=XO5OHg) | \n[SBU+](https://uofmacau-my.sharepoint.com/:u:/g/personal/yb87432_umac_mo/EQMCyGNUo3xJg8fInF7LWQAB0g9HFZHRBuBoxlzEL5CNUg?e=ENfsZV) \u003c/b\u003e\n\n- Training on ISTD dataset and generating shadow using USR dataset: \u003cb\u003e \n[Syn. Shadow](https://uofmacau-my.sharepoint.com/:u:/g/personal/yb87432_umac_mo/EW8-rjV5MX5BtoNSoDuzQg8B2lk4QHZS9jZzDDPfrEZVfg?e=DxPVfR) |\n[Pre-trained Model (re-trained)](https://drive.google.com/file/d/1Mlq4rQAj9jGiomrbQzjKuyHiVI8tgIhw/view?usp=sharing) \u003c/b\u003e\n\n- Extracted Shadow Mask in SRD dataset:\u003cb\u003e\n[SRD Mask](https://uofmacau-my.sharepoint.com/:u:/g/personal/yb87432_um_edu_mo/EZ8CiIhNADlAkA4Fhim_QzgBfDeI7qdUrt6wv2EVxZSc2w?e=wSjVQT)\n\u003c/b\u003e\n\n\n## **Other Resources**\n\n- [ISTD dataset](https://github.com/DeepInsight-PCALab/ST-CGAN)\n- [USR: Unpaired Shadow Removal dataset](https://drive.google.com/open?id=1PPAX0W4eyfn1cUrb2aBefnbrmhB1htoJ)\n- SRD Dataset (please email the [authors](http://vision.sia.cn/our%20team/JiandongTian/JiandongTian.html) to get an assessment).\n\n\n## **Setup**\nCreating the conda environments following [here](https://github.com/ceciliavision/perceptual-reflection-removal#conda-environment).\n\n## **Demo**\n\n#### 1. Local ipynb demo\n1. download the pre-trained model from above. **SRD+** is recommended.\n2. download pretrained-vgg19 from [MatConvNet](http://www.vlfeat.org/matconvnet/pretrained/#downloading-the-pre-trained-models).\n3. Uncompress pre-trained models into 'Models/' as shown in the folders.\n4. Starting a [jupyter](https://jupyter.org/) server and run the demo code following the instructions in `demo.ipynb`\n\nIt has been tested both in MacOS 10.15 and Ubuntu 18.04 LTS. Both CPU and GPU are supported (But running on CPU is quite slow).\n\n#### 2. Online google colab demo\n\nan online **shadow removal** demo is hosted in Google CoLab by [this url](https://colab.research.google.com/drive/1cJ_dsBUXFaFtjoZB9gDYeahjmysnvnTq)\n\nan online **shadow synthesis** demo is hosted in Google CoLab by [this url](https://colab.research.google.com/drive/1WGtsxKxogxgusFJhLJMjM8ZpMsVYjuDg#scrollTo=SxF4uQHmEiv0)\n\n#### 3. Demo from command line (Thanks [@aliericcantona](https://github.com/aliericcantona))\n\n```\npython demo.py --model PATH_TO_PRETRAINED_MODEL --vgg_19_path PATH_TO_VGG19 --input_dir SAMPLES_DIR --result_dir RESULTS_DIR\n```\n\n## **Training**\nThe data folders should be:\n```\nISTD_DATA_ROOT\n    * train\n        - train_A # shadow image\n        - train_B # shadow mask\n        - train_C # shadowfree image\n        - shadow_free # USR shadowfree images\n        - synC # our Syn. shadow\n    * test\n        - test_A # shadow image\n        - test_B # shadow mask\n        - test_C # shadowfree image\n\nSRD_DATA_ROOT\n    * train\n        - train_A # renaming the original `shadow` folder in `SRD`.\n        - train_B # the extracted shadow mask by ourself.\n        - train_C # renaming the original `shadow_free` folder in `SRD`.\n        - shadow_free # USR shadowfree images\n        - synC # our Syn. shadow\n    * test\n        - train_A # renaming the original `shadow` folder in `SRD`.\n        - train_B # the extracted shadow mask by ourself.\n        - train_C # renaming the original `shadow_free` folder in `SRD`.\n\n```\n### 1. Generating Synthesized Shadow\nDownloading the `ISTD` from the source, download the USR dataset and unzip it into unzip it into  `$YOUR_DATA_ROOT/ISTD_dataset/train/`. Train the GAN by:\n```\npython train_ss.py \\\n--task YOUR_TASK_NAME \\\n--data_dir $YOUR_DATA_ROOT/$ISTD_DATASET_ROOT/train/ \\\n--use_gpu 0 # \u003c0 for CPU \\\n--is_training 1 # 0 for testing \\\n```\n### 2. Training on the ISTD dataset\nDownloading the `ISTD` from the source, download our synthesized dataset and unzip it into  `$YOUR_DATA_ROOT/ISTD_dataset/train/`. Train the network by:\n```\npython train_sr.py \\\n--task YOUR_TASK_NAME \\\n--data_dir $YOUR_DATA_ROOT/$ISTD_DATASET_ROOT/train/ \\\n--use_gpu 1 # \u003c0 for cpu \\\n--is_training 1 # 0 for testing \\\n--use_da 0.5 # the percentage of synthesized dataset\n```\n### 3. Training on SRD dataset \nDownload and unzip the `SRD` dataset from the source. Reorganizing the dataset as described above. \n```\npython train_sr.py \\\n--task YOUR_TASK_NAME \\\n--data_dir $YOUR_DATA_ROOT/$SRD_DATASET_ROOT/train/ \\\n--use_gpu 1 # \u003c0 for cpu \\\n--is_training 1 # 0 for testing \\\n--use_da 0.5 # the percentage of synthesized dataset\n```\n\n## **Test**\n```\n# ISTD DATASET\npython train_sr.py \\\n--task YOUR_TASK_NAME # path to the pre-trained model [logs/YOUR_TASK_NAME] \\\n--data_dir $YOUR_DATA_ROOT/$ISTD_DATASET_ROOT/test/ \\\n--use_gpu 1 # \u003c0 for cpu \\\n--is_training 0 # 0 for testing \\\n\n# SRD DATASET\npython train_sr.py \\\n--task YOUR_TASK_NAME # path to the pre-trained model [logs/YOUR_TASK_NAME] \\\n--data_dir $YOUR_DATA_ROOT/$SRD_DATASET_ROOT/test/ \\\n--use_gpu 1 # \u003c0 for cpu \\\n--is_training 0 # 0 for testing \\\n\n```\n\n## **Acknowledgements**\nThe author would like to thanks Nan Chen for her helpful discussion.\n\nPart of the code is based upon [FastImageProcessing](https://github.com/CQFIO/FastImageProcessing) and [Perception Reflection Removal](https://github.com/ceciliavision/perceptual-reflection-removal)\n\n## **Citation**\n\nIf you find our work useful in your research, please consider citing:\n\n```\n@misc{cun2019ghostfree,\n    title={Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN},\n    author={Xiaodong Cun and Chi-Man Pun and Cheng Shi},\n    year={2019},\n    eprint={1911.08718},\n    archivePrefix={arXiv},\n    primaryClass={cs.CV}\n}\n```\n\n## **Contact**\nPlease contact me if there is any question (Xiaodong Cun yb87432@um.edu.mo)\n\n## **Related Works**\n\nZhang, Xuaner, Ren Ng, and Qifeng Chen. \"[Single Image Reflection Separation with Perceptual Losses](https://arxiv.org/abs/1806.05376).\" Proceedings of the CVPR. (2018).\n\n\nHu, Xiaowei, et al. \"[Mask-ShadowGAN: Learning to Remove Shadows from Unpaired Data](https://arxiv.org/abs/1903.10683).\" Proceedings of the ICCV (2019).\n\n[![HitCount](http://hits.dwyl.com/vinthony/ghost-free-shadow-removal.svg)](http://hits.dwyl.com/vinthony/ghost-free-shadow-removal)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvinthony%2Fghost-free-shadow-removal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvinthony%2Fghost-free-shadow-removal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvinthony%2Fghost-free-shadow-removal/lists"}