{"id":18573556,"url":"https://github.com/prs-eth/u-tilise","last_synced_at":"2025-04-10T07:32:10.220Z","repository":{"id":170911613,"uuid":"647190346","full_name":"prs-eth/U-TILISE","owner":"prs-eth","description":null,"archived":false,"fork":false,"pushed_at":"2023-12-04T16:32:31.000Z","size":9844,"stargazers_count":43,"open_issues_count":1,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-24T18:11:24.027Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/prs-eth.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-30T08:51:09.000Z","updated_at":"2025-03-05T08:23:57.000Z","dependencies_parsed_at":"2024-11-06T23:11:00.680Z","dependency_job_id":null,"html_url":"https://github.com/prs-eth/U-TILISE","commit_stats":null,"previous_names":["stuckerc/u-tilise"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prs-eth%2FU-TILISE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prs-eth%2FU-TILISE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prs-eth%2FU-TILISE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prs-eth%2FU-TILISE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/prs-eth","download_url":"https://codeload.github.com/prs-eth/U-TILISE/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248176446,"owners_count":21060066,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T23:10:50.522Z","updated_at":"2025-04-10T07:32:09.165Z","avatar_url":"https://github.com/prs-eth.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n    \u003ch1\u003eU-TILISE: A Sequence-to-sequence Model for Cloud Removal in Optical Satellite Time Series\u003c/h1\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ch3 align=\"center\"\u003e \u003cstrong\u003e\u003csup\u003e1\u003c/sup\u003eCorinne Stucker,   \u003csup\u003e2\u003c/sup\u003eVivien Sainte Fare Garnot,   \u003csup\u003e1\u003c/sup\u003eKonrad Schindler\u003c/strong\u003e  \u003c/h3\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003cstrong\u003e\u003csup\u003e1\u003c/sup\u003e Chair of Photogrammetry and Remote Sensing, ETH Zurich\u003c/strong\u003e\u003cbr\u003e\n    \u003cstrong\u003e\u003csup\u003e2\u003c/sup\u003e Institute for Computational Science, University of Zurich\u003c/strong\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ch3 align=\"center\"\u003e[\u003ca href=\"https://doi.org/10.1109/TGRS.2023.3333391\"\u003ePaper\u003c/a\u003e] [\u003ca href=\"https://arxiv.org/abs/2305.13277\"\u003eArXiv\u003c/a\u003e]\u003c/h3\u003e\n\u003c/p\u003e\n\n\nSatellite image time series in the optical and infrared spectrum suffer from frequent data gaps due to cloud cover, cloud shadows, and temporary sensor outages. It has been a long-standing problem of remote sensing research how to best reconstruct the missing pixel values and obtain complete, cloud\u0026#8209;free image sequences. We approach that problem from the perspective of representation learning and develop U\u0026#8209;TILISE, an efficient neural model that is able to implicitly capture spatio-temporal patterns of the spectral intensities, and that can therefore be trained to map a cloud\u0026#8209;masked input sequence to a cloud\u0026#8209;free output sequence. The model consists of a convolutional *spatial encoder* that maps each individual frame of the input sequence to a latent encoding; an attention-based *temporal encoder* that captures dependencies between those per-frame encodings and lets them exchange information along the time dimension; and a convolutional *spatial decoder* that decodes the latent embeddings back into multi-spectral images. We experimentally evaluate the proposed model on EarthNet2021, a dataset of Sentinel-2 time series acquired all over Europe, and demonstrate its superior ability to reconstruct the missing pixels. Compared to a standard interpolation baseline, it increases the PSNR by 1.8 dB at previously seen locations and by 1.3 dB at unseen locations.\n\n\u003cimage src=\"docs/teaser.png\"/\u003e\n\n\n## Setup\n\n### Dependencies\nThis code was developed using Ubuntu 22.04, Python 3.10, PyTorch 1.13, and CUDA 11.6.\nFor an optimal experience, we recommend creating a new conda environment and installing the required dependencies with the following commands:\n```bash\nconda env create -f environment.yml\nconda activate u-tilise\n```\n\nAfter setting up the environment, establish a corresponding IPython kernel named ``, . This is necessary to execute the demo Jupyter Notebook [demo.ipynb](demo.ipynb):\n```bash\nipython kernel install --user --name=u-tilise\n```\n\n### Checkpoints\nYou can download our pretrained model checkpoints [here](https://share.phys.ethz.ch/~pf/stuckercdata//checkpoints/).\nIf you prefer an automated approach, execute the script below to download and extract all checkpoints to the `./checkpoints/` directory:\n```bash\nbash ./scripts/download_checkpoints.sh\n``` \n\nWe provide the following model checkpoints:\n* `utilise_earthnet2021.pth`: model weights for  trained on the [EarthNet2021](https://www.earthnet.tech/en21/quick-start-guide/) dataset.\n* `utilise_sen12mscrts_wo_s1.pth`: model weights for U\u0026#8209;TILISE trained on the [SEN12MS-CR-TS](https://patricktum.github.io/cloud_removal/sen12mscr/) dataset, *without* SAR guidance.\n* `utilise_sen12mscrts_w_s1.pth`: model weights for U\u0026#8209;TILISE trained on the [SEN12MS-CR-TS](https://patricktum.github.io/cloud_removal/sen12mscr/) dataset, *with* SAR guidance.\n\n\n### Sample data\nFor demonstration purposes, we offer the preprocessed *iid* test split of the EarthNet2021 dataset. To download this data, execute the following command:\n```bash\nbash ./scripts/download_data_earthnet2021.sh\n```\n\nThis will download and unpack the data (~ 14 GB) into the `./data/` directory. The provided data sets are:\n* `earthnet_iid_test_split.hdf5`: The original 30-frames time series with actual data gaps.\n* `earthnet_iid_test_split_simulation.hdf5`: The corresponding cloud\u0026#8209;free time series with synthetically added data gaps.\n\n\n## Data\n\n### EarthNet2021\n[EarthNet2021](https://www.earthnet.tech/en21/quick-start-guide/)[^1] provides Sentinel\u0026#8209;2 satellite image time\nseries collected over Central and Western Europe from November 2016 to May 2020. Each time series comprises \n30 images with Level-1C top-of-atmosphere (TOA) reflectances. The images are acquired in a regular temporal interval of \nfive days. Every image is composed of the four spectral bands B2 (blue), B3 (green), B4 (red), and B8 (near-infrared) \nand covers a spatial extent of 128\u0026times;128 pixels (2.56\u0026times;2.56 km in scene space), resampled to the resolution of \n20 m. Furthermore, the dataset includes pixel-wise cloud probability maps (training data only) and binary cloud (and cloud shadow) masks.\n\n\n### SEN12MS-CR-TS\n[SEN12MS-CR-TS](https://patricktum.github.io/cloud_removal/sen12mscr/)[^2] provides globally sampled Sentinel\u0026#8209;2 satellite image time\nseries from 2018 with a spatial extent of 256\u0026times;256 pixels (2.56\u0026times;2.56 km in scene space). Each time series\ncontains 30 images. The images encompass all 13 spectral bands, upsampled to 10 m resolution. Furthermore, every optical image \nis paired with a spatially co-registered, temporally close (but not synchronous) C-band SAR image with two channels representing \nthe $\\sigma_0$ backscatter coefficients in the VV and VH polarizations, in units of decibels (dB). Furthermore, the dataset \nincludes pixel-wise cloud probabilities and binary cloud masks.\n\n### Simulation of data gaps\nTo train and quantitatively assess U\u0026#8209;TILISE's performance, we use gap\u0026#8209;free (cloud\u0026#8209;free) Sentinel\u0026#8209;2 satellite image time series.\nOur preprocessing steps are as follows:\n1. We first identify all images with partially occluded pixels or images that are occluded/missing entirely by applying \na threshold to the cloud probability maps (if available) or the binary cloud masks. We then remove all images with data gaps to produce \ncloud\u0026#8209;free time series that exhibit a valid observation for every spatio-temporal location.\n2. We discard time series with less than five remaining images, as we deem such sequences too short for learning spatio-temporal patterns.\n3. To generate synthetic data gaps, we randomly sample real cloud masks from other acquisition times and/or locations within the same Sentinel-2 tile\n   and superimpose those masks onto the gap-free time series by setting the reflectance of occluded pixels (according to the masks) to the maximum value~1.\n\n\n### Custom dataset and data loader\nEnsure the output of your custom data loader meets the following minimum requirements:\n\n- `x`: (Masked) input time series, of shape $(T \\times C \\times H \\times W)$.\n- `masks`: Masks used to mark occluded/missing pixels in `y` with dimensions $(T \\times 1 \\times H \\times W)$.\n- `position_days`: Positions used for positional encoding, of shape $(T, )$.\n- `y`: Observed/Target time series, of shape $(T \\times C \\times H \\times W)$.\n\nFor visualization during training in Weights \u0026 Biases:\n- Include `c_index_rgb`, `c_index_nir`, and `sample_index`.\n\nFor visualization in [demo.ipynb](demo.ipynb):\n- Include `c_index_rgb`.\n\n\n## Preprocessing\n\n:warning: **Note:** Skip this section if you don't intend to retrain U\u0026#8209;TILISE on the EarthNet2021 or the SEN12MS\u0026#8209;CR\u0026#8209;TS dataset.\n\nTo train U\u0026#8209;TILISE on the EarthNet2021 or the SEN12MS\u0026#8209;CR\u0026#8209;TS dataset, you will need to complete several steps:\ndownloading the dataset, converting it from its native format to one compatible with our data loaders, and running a simulation to \ngenerate cloud\u0026#8209;free sequences with synthetically added data gaps. Here's a step-by-step guide:\n\n**EarthNet2021**\n1. **Dataset download**\n   \n   Obtain the dataset by following the download instructions available [here](https://www.earthnet.tech/en21/ds-download/).\n\n2. **Data conversion**\n   \n   The EarthNet2021 dataset comprises two .npz files for every time series. We aggregate the data of each data split\n   (i.e., train, iid, ood) into a single HDF5 file for further use. To run the npz-to-HDF5 conversion, execute the commands below:\n\n   ```bash\n   python ./toolbox/EarthNet2021_npz2hdf5.py --root_source \u003cdata_directory\u003e --root_dest \u003coutput_directory\u003e --split train --mode train\n   python ./toolbox/EarthNet2021_npz2hdf5.py --root_source \u003cdata_directory\u003e --root_dest \u003coutput_directory\u003e --split train --mode val\n   python ./toolbox/EarthNet2021_npz2hdf5.py --root_source \u003cdata_directory\u003e --root_dest \u003coutput_directory\u003e --split iid\n   python ./toolbox/EarthNet2021_npz2hdf5.py --root_source \u003cdata_directory\u003e --root_dest \u003coutput_directory\u003e --split ood\n   ```\n    \n   Replace `data_directory` with the root directory where you have saved your downloaded data and `output_directory` with\n   your preferred destination for the HDF5 files.\n   \n   Upon executing the above commands, you should find the following HDF5 files in `output_directory`:\n   - `train.hdf5`\n   - `iid_test_split.hdf5`\n   - `ood_test_split.hdf5`\n   \n   Besides the npz-to-HDF5 conversion, the script also stores the indices of unavailable frames and identifies all frames with partially\n   or fully occluded pixels for each time series in the respective data split.\n\n\n3. **Preprocessing of the validation split**\n\n   To generate time series with artificial data gaps for training, run the following command:\n   \n   ```bash\n   python ./toolbox/simulate_dataset.py --config_file ./data/configs/config_earthnet2021_simulation_val.yaml --out_dir \u003coutput_directory\u003e --out_hdf5_filename earthnet2021_val_simulation.hdf5\n   ```\n\n   Make sure to set `root` in the [config_earthnet2021_simulation_val.yaml](./data/configs/config_earthnet2021_simulation_val.yaml)\n   configuration file to match the `output_directory` used in step 2. Optionally, change `max_seq_length` to modify the fixed\n   maximal temporal length $T$.\n\n   This step creates a new HDF5 file `/output_directory/earthnet2021_val_simulation.hdf5`. Among others, this file contains the\n   temporally trimmed cloud\u0026#8209;free validation time series, the corresponding time series with synthetically introduced data gaps, the masks\n   used for masking, and the acquisition dates. \n\n\n4. **Preprocessing of the test splits**\n\n   To generate time series with artificial data gaps for testing and evaluation, execute the command below:\n   \n   ```bash\n   python ./toolbox/simulate_dataset.py --config_file ./data/configs/config_earthnet2021_simulation_test.yaml --out_dir \u003coutput_directory\u003e --out_hdf5_filename earthnet2021_iid_test_split_simulation.hdf5\n   ```\n\n   Again, configure `root` in the [config_earthnet2021_simulation_test.yaml](./data/configs/config_earthnet2021_simulation_test.yaml)\n   configuration file to match the `output_directory` used in step 2. If you wish to process the *ood* test split instead, set the\n   `split` parameter to *ood* and adjust `--out_hdf5_filename` accordingly.\n\n   \u003e Note that `max_seq_length` in the [config_earthnet2021_simulation_test.yaml](./data/configs/config_earthnet2021_simulation_test.yaml)\n   configuration file is set to None to skip the temporal trimming of the test time series.\n\n\n\n**SEN12MS-CR-TS**\n1. **Dataset download**\n\n   Follow the provided [instructions](https://patricktum.github.io/cloud_removal/sen12mscr/) to download the dataset.\n\n2. **Data conversion**\n\n   To convert and aggregate the .tif files from individual acquisitions into a single HDF5 file for each data split\n   (i.e., train, val, test), use the functionalities provided [here](https://github.com/PatrickTUM/SEN12MS-CR-TS/blob/master/util/hdf5converter/).\n\n3. **Detection of real data gaps**\n\n   Execute the following script to identify all frames that have partial or complete occlusions:\n\n   ```bash\n   python ./toolbox/SEN12MSCRTS_detect_cloudy_frames.py --root \u003cdata_directory\u003e --split train\n   python ./toolbox/SEN12MSCRTS_detect_cloudy_frames.py --root \u003cdata_directory\u003e --split val\n   python ./toolbox/SEN12MSCRTS_detect_cloudy_frames.py --root \u003cdata_directory\u003e --split test\n   ```\n\n   Upon completion, the HDF5 files created in step 2 will be updated with indices corresponding to images exhibiting data gaps.\n\n4. **Preprocessing of the validation and test splits**\n\n   To generate time series with artificial data gaps for training and evaluation, execute the commands below:\n   \n   ```bash\n   python ./toolbox/simulate_dataset.py --config_file ./data/configs/config_sen12mscrts_simulation_val.yaml --out_dir \u003coutput_directory\u003e --out_hdf5_filename sen12mscrts_val_simulation.hdf5\n   python ./toolbox/simulate_dataset.py --config_file ./data/configs/config_sen12mscrts_simulation_test.yaml --out_dir \u003coutput_directory\u003e --out_hdf5_filename sen12mscrts_test_simulation.hdf5\n   ```\n\n   For detailed instructions on each parameter and step, refer to the instructions provided for the EarthNet2021 dataset above.\n\n\n\n## Training \n\nTo initiate training, execute the following command:\n```bash\npython run_train.py /path/to/config_file.yaml --save_dir \u003coutput_directory\u003e \n```\n\nwhere:\n\n- `/path/to/config_file.yaml` is the YAML configuration file specifying all runtime arguments.\n- `--save_dir` specifies the output directory.\n\nAll training hyperparameters are predefined in [default.yaml](configs/default.yaml) and set to the values used\nin the main experiments of the paper. If a parameter is specified in `/path/to/config_file.yaml`, it will override the default value.\n\nTo view all available training options, run:\n```bash\npython run_train.py -h\n```\n\n**Example configuration files**\n\nWe provide a collection of configuration files within the `./configs/` directory:\n* `default.yaml`: Default parameter settings.\n* `config_earthnet2021_train.yaml`: Additional settings used to train on the EarthNet2021 dataset.\n* `config_sen12mscrts_train.yaml`: Additional settings used to train on the SEN12MS\u0026#8209;CR\u0026#8209;TS dataset.\n* `config_earthnet2021_test_simulation.yaml`: Test settings for the EarthNet2021 dataset (synthetic data gaps).\n* `config_earthnet2021_test.yaml`: Test settings for the EarthNet2021 dataset (actual data gaps).\n\nUpon execution, the `run_train.py` script combines the `default.yaml` with the provided `--config_file`, \nsaving the resultant runtime configuration as a YAML file in the directory defined by `--save_dir`. As a point of reference, \nyou can find an example runtime configuration at [demo_train_config.yaml](configs/demo_train_config.yaml). We will use this YAML file below to \ndemonstrate the evaluation procedure.\n\n\n## Evaluation\n\nTo perform the evaluation, execute the `run_eval.py` script using the following command:\n```bash\npython run_eval.py /path/to/config.yaml \u003cmethod_name\u003e --checkpoint \u003cpath_to_checkpoint\u003e --test-data.data-dir \u003cdata_directory\u003e --test-data.hdf5-file \u003chdf5_file_name\u003e --test-data.split \u003cdata_split\u003e\n```\n\nTo view all available input options, run:\n```bash\npython run_eval.py -h\n```\n\n\n**Examples**\n\n1. To evaluate the *iid* test split of the EarthNet2021 dataset, use the following command:\n```bash\npython run_eval.py ./configs/demo_train_config.yaml utilise --test-data.data-dir ./data/ --test-data.hdf5-file earthnet2021_iid_test_split_simulation.hdf5 --test-data.split iid --checkpoint ./checkpoints/utilise_earthnet2021.pth \n```\n\n2. Similarly, to evaluate the linear interpolation baseline, run:\n```bash\npython run_eval.py ./configs/demo_train_config.yaml trivial --mode linear_interpolation --test-data.data-dir ./data/ --test-data.hdf5-file earthnet2021_iid_test_split_simulation.hdf5 --test-data.split iid\n```\n\n\n## Demo\n\n### Inference\n\nCheck out the Jupyter Notebook [demo.ipynb](demo.ipynb), which provides a step-by-step demonstration of using U\u0026#8209;TILISE to impute a \ngiven time series and visualize the associated attention masks.\n\n\n## Citation\n```bibtex\n@article{stucker2023u,\n  title={{U-TILISE}: A Sequence-to-sequence Model for Cloud Removal in Optical Satellite Time Series},\n  author={Stucker, Corinne and Garnot, Vivien Sainte Fare and Schindler, Konrad},\n  journal={IEEE Transactions on Geoscience and Remote Sensing},\n  year={2023},\n  volume={61}\n}\n```\n\n## Acknowledgements\nU\u0026#8209;TILISE extends the architecture of [U-TAE](https://github.com/VSainteuf/utae-paps) to a full 3D spatio-temporal sequence-to-sequence model that preserves the temporal dimension.\n\nWe thank Vivien Sainte Fare Garnot for his efforts in open sourcing and maintaining U-TAE.[^3]\n\n\n[^1]: C. Requena-Mesa, V. Benson, M. Reichstein, J. Runge, and J. Denzler, *EarthNet2021 :A large-scale dataset and challenge for earth surface forecasting as a guided video prediction task*, in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2021, pp.1132–1142.\n[^2]: P. Ebel, Y. Xu, M. Schmitt, and X. X. Zhu, *SEN12MS-CR-TS: A remote-sensing dataset for multimodal multitemporal cloud removal*, IEEE Transactions on Geoscience and Remote Sensing, vol.60, pp. 1-14, 2022.\n[^3]: V.S.F. Garnot and L. Landrieu, *Panoptic segmentation of satellite image time series with convolutional temporal attention networks*, in IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp.4872–4881.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprs-eth%2Fu-tilise","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprs-eth%2Fu-tilise","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprs-eth%2Fu-tilise/lists"}