{"id":18730739,"url":"https://github.com/ameraner/dsen2-cr","last_synced_at":"2025-05-10T20:07:12.917Z","repository":{"id":41066806,"uuid":"310143407","full_name":"ameraner/dsen2-cr","owner":"ameraner","description":"DSen2-CR: A network for removing clouds from Sentinel-2 images. This repo contains the model code, written in Python/Keras, as well as links to pre-trained checkpoints and the SEN12MS-CR dataset.","archived":false,"fork":false,"pushed_at":"2024-05-22T16:51:10.000Z","size":693,"stargazers_count":151,"open_issues_count":11,"forks_count":34,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-09T14:00:26.050Z","etag":null,"topics":["cloud-removal","deep-learning","optical","residual-neural-network","sar","satellite","satellite-data","satellite-imagery","sentinel","sentinel-1","sentinel-2"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ameraner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-04T23:45:07.000Z","updated_at":"2025-05-04T12:33:54.000Z","dependencies_parsed_at":"2023-09-22T07:08:07.447Z","dependency_job_id":"a26f4e87-bad6-43cb-ba86-8c08919d007e","html_url":"https://github.com/ameraner/dsen2-cr","commit_stats":{"total_commits":14,"total_committers":4,"mean_commits":3.5,"dds":0.4285714285714286,"last_synced_commit":"9d6cdcf9e262920cb971e34fe298874d77e7a269"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ameraner%2Fdsen2-cr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ameraner%2Fdsen2-cr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ameraner%2Fdsen2-cr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ameraner%2Fdsen2-cr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ameraner","download_url":"https://codeload.github.com/ameraner/dsen2-cr/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253292089,"owners_count":21885035,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-removal","deep-learning","optical","residual-neural-network","sar","satellite","satellite-data","satellite-imagery","sentinel","sentinel-1","sentinel-2"],"created_at":"2024-11-07T14:45:18.143Z","updated_at":"2025-05-10T20:07:12.893Z","avatar_url":"https://github.com/ameraner.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cloud removal in Sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion\n[![BPA](https://img.shields.io/badge/Winner-ISPRS%20U.V.%20Helava%20Award%202020--2021-brightgreen)](https://www.isprs.org/society/awards/helava/2020_2021award.aspx)\n[![BPA](https://img.shields.io/badge/Winner-ISPRS%20JPRS%20Best%20Paper%20Award%202020-brightgreen)](https://www.journals.elsevier.com/isprs-journal-of-photogrammetry-and-remote-sensing/news/the-u-v-helava-award-best-paper-volumes-159-170-2020)\n\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/cloud-removal-in-sentinel-2-imagery-using-a/cloud-removal-on-sen12ms-cr)](https://paperswithcode.com/sota/cloud-removal-on-sen12ms-cr?p=cloud-removal-in-sentinel-2-imagery-using-a)\n\n![Paper preview](doc/paper.JPG)\n\u003e\n\u003e _Example results from the final setup of DSen2-CR using the CARL loss. Left is the input cloudy image, middle is the predicted image, right is the cloud-free target image._\n----\nThis repository contains the code and models for the paper\n\u003e Meraner, A., Ebel, P., Zhu, X. X., \u0026 Schmitt, M. (2020). Cloud removal in Sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion. ISPRS Journal of Photogrammetry and Remote Sensing, 166, 333-346.\n\nThe **open-access paper** is available at [the Elsevier ISPRS page](https://doi.org/10.1016/j.isprsjprs.2020.05.013).\n\nThe paper [won the ISPRS 2020 Best Paper Award](https://www.journals.elsevier.com/isprs-journal-of-photogrammetry-and-remote-sensing/news/the-u-v-helava-award-best-paper-volumes-159-170-2020), and \nand then went on to [win the U.V. Helava Award as the best paper for the period 2020-2021](https://www.isprs.org/society/awards/helava/2020_2021award.aspx).\n\nIf you use this code, models or dataset for your research, please cite us accordingly:\n```bibtex\n@article{Meraner2020,\ntitle = \"Cloud removal in Sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion\",\njournal = \"ISPRS Journal of Photogrammetry and Remote Sensing\",\nvolume = \"166\",\npages = \"333 - 346\",\nyear = \"2020\",\nissn = \"0924-2716\",\ndoi = \"https://doi.org/10.1016/j.isprsjprs.2020.05.013\",\nurl = \"http://www.sciencedirect.com/science/article/pii/S0924271620301398\",\nauthor = \"Andrea Meraner and Patrick Ebel and Xiao Xiang Zhu and Michael Schmitt\",\nkeywords = \"Cloud removal, Optical imagery, SAR-optical, Data fusion, Deep learning, Residual network\",\n}\n```\n\n# Code\n\n---\n\n**NOTE**\n\nThe code in this repository has been created in my early Python years and might not be the most elegant in some parts. I apologize for eventual issues or possible bugs. \n\nShould you notice something in the code, please feel free to create a Github issue (or, even better, a pull request :)), or let me know at the address  [andrea.meraner [at] eumetsat.int](mailto:andrea.meraner@eumetsat.int) ! \n\n---\n\n## Installation\nThe network is written in Keras with Tensorflow as backend. It is strongly advised to use GPU support to run the models.\n\nA conda environment with the required dependencies can be created with\n```bash\nconda create -n dsen2cr_env\nconda activate dsen2cr_env\nconda install -c conda-forge python=3.7 tensorflow-gpu=1.15.0 keras=2.2.4 numpy=1.17 scipy rasterio pydot graphviz h5py=2.10.0\n```\n\nAlternatively, a Dockerfile is provided in `Docker/Dockerfile` which can be used to create a Docker image including CUDA.\n\nNote: \nThis code has been mainly written at the end of 2018/start of 2019 with the Python packages versions available at that time. A usage with updated packages might require some modification of the code.\nIf you try this code with updated libraries, please let me know your findings ([andrea.meraner [at] eumetsat.int](mailto:andrea.meraner@eumetsat.int)).\n\nTo clone the repo:\n```bash\ngit clone git@github.com:ameraner/dsen2-cr.git\ncd dsen2-cr\n```\n\n## Usage\n### Basic Commands\nA new model can be trained from scratch by simply launching\n```bash\ncd Code/\npython dsen2cr_main.py\n```\nThe setup and hyperparameters can be tuned directly in the first lines of the main code.\n\nTo resume the training from a previoulsy saved checkpoint, type\n```bash\npython dsen2cr_main.py --resume path/to/checkpoint.h5\n```\n\nTo predict images and evaluate the metrics of a trained network, do\n```bash\npython dsen2cr_main.py --predict path/to/checkpoint.h5\n```\n\n### Dataset Paths\nThe main code will look for the paths to training/validation/test data in the csv file `Data/datasetfilelist.csv`.\nAn example is provided in the repository. The first column of each entry is an integer, where `1` defines a training sample, \n`2` a validation sample, and `3` a test sample. The second, third, and fourth column indicate the subfolder names where the\nSentinel-1, Sentinel-2 Cloudfree, and Sentinel-2 Cloudy images are located respectively. The fifth column finally states the \nfilename of the image, that must be the same in the three folders.\nThe three subfolders must be located in the path defined by the variable `input_data_folder` in the main script.  \n\nIf you wish to download the full list of patches including the indication of the train/val/test split, \nyou can find the full csv files [here](https://drive.google.com/file/d/1e1u1ARVf4rRYugiyI-pYrVcK29drvIq5/view?usp=sharing).\nPlease see the Dataset section below for more information.\n\n# Trained Model Checkpoints\nThe full DSen2-CR model trained by optimizing the CARL loss can be downloaded from Google Drive [here](https://drive.google.com/file/d/1L3YUVOnlg67H5VwlgYO9uC9iuNlq7VMg/view?usp=sharing).\n\nThe full model trained on a plain L1 loss can be downloaded [here](https://drive.google.com/file/d/1zv4_91Yr2IYyYDoqhZw8KpnfvfLhkuBB/view?usp=sharing). The network trained on CARL but without SAR input \ncan be found [here](https://drive.google.com/file/d/1VHZa5-lX68mA2FbHeCiQsUq13oECw9DA/view?usp=sharing). The network trained without SAR, and on plain L1 loss, can be found [here](https://drive.google.com/file/d/11Th6UwKMXla7LGxsFJXwj-Jx9bSKlWUH/view?usp=sharing).\n\n\n# Dataset\nThe dataset used in this work is called SEN12MS-CR. A slightly reprocessed version of it\nis publicly available for download [here](https://mediatum.ub.tum.de/1554803).\nIf you use this dataset for your research, please cite our related IEEE TGRS paper \n\u003e Ebel, P., Meraner, A., Schmitt, M., \u0026 Zhu, X. X. (2020). Multisensor Data Fusion for Cloud Removal in Global and All-Season Sentinel-2 Imagery. IEEE Transactions on Geoscience and Remote Sensing.\n\ndescribing the dataset release. The paper can be accessed\nfor free at [the IEEE Explore page](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=\u0026arnumber=9211498). See \nalso a related website [here](https://patricktum.github.io/cloud_removal/).\n```bibtex\n@article{Ebel2020,\n  author={P. {Ebel} and A. {Meraner} and M. {Schmitt} and X. X. {Zhu}},\n  journal={IEEE Transactions on Geoscience and Remote Sensing}, \n  title={Multisensor Data Fusion for Cloud Removal in Global and All-Season Sentinel-2 Imagery}, \n  year={2020},\n  volume={},\n  number={},\n  pages={1-13},\n  doi={10.1109/TGRS.2020.3024744}}\n```  \n---\n\n**NOTE**\n\nThe published SEN12MS-CR dataset described above is a reprocessed version of the one used in this work.  \n\nThe main difference is that the one from this work was in the WGS84 coordinate system, \nwhereas the released one was a reprocessing with a UTM CRS transform (in order to make the patches\nco-registered with available semantic segmentations and scene-wise labels - see paper). The differences will most\nprobably not affect the network performance, and the pre-trained models can still be used.\n\nThe csv files to be used as inputs for the model linked in the \"Dataset Paths\" section above have been adapted\nto match the filenaming convention used in the published reprocessed dataset. Note that due to the reprocessing, the train/val/test \nsplits are not identical to the ones used for the paper - the differences, however, are minor.\n\n@CodyKurpanek created a Jupyter Notebook that downloads and processes the public SEN12MS-CR dataset in order to fit the expected structure by this code.\nThe notebook can be found under [Data/unpack_dataset.ipynb](https://github.com/ameraner/dsen2-cr/blob/main/Data/unpack_dataset.ipynb).\n\n# PyTorch Model\n\nIf you're interested in a PyTorch implementation of the DSen2-CR model, the according class is available in \n[Code/dsen2cr_pytorch_model.py](https://github.com/ameraner/dsen2-cr/blob/main/Code/dsen2cr_pytorch_model.py) \n(thanks to [Patrick Ebel](https://github.com/PatrickTUM)).\n\n---\n# Credits\nAlthough now heavily modified and expanded, this code was originally based on the code by [Charis Lanaras](https://github.com/lanha)\navailable in [the DSen2 repo](https://github.com/lanha/DSen2). Also the network used by this work is, as the name suggests, \nheavily based on the original DSen2 network (see [related paper](https://www.sciencedirect.com/science/article/abs/pii/S0924271618302636)). \nI am grateful to the authors for making the original source code available.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fameraner%2Fdsen2-cr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fameraner%2Fdsen2-cr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fameraner%2Fdsen2-cr/lists"}