{"id":13736972,"url":"https://github.com/VSainteuf/utae-paps","last_synced_at":"2025-05-08T13:32:17.247Z","repository":{"id":53577162,"uuid":"378926281","full_name":"VSainteuf/utae-paps","owner":"VSainteuf","description":"PyTorch implementation of U-TAE and PaPs for satellite image time series panoptic segmentation.","archived":false,"fork":false,"pushed_at":"2024-08-20T13:48:35.000Z","size":3175,"stargazers_count":136,"open_issues_count":9,"forks_count":54,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-08-20T15:53:49.787Z","etag":null,"topics":["agriculture","computer-vision","deep-learning","iccv","iccv2021","panoptic-segmentation","satellite-imagery","self-attention","semantic-segmentation","sentinel-2"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VSainteuf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-21T12:40:13.000Z","updated_at":"2024-08-20T13:48:38.000Z","dependencies_parsed_at":"2024-08-20T15:46:05.666Z","dependency_job_id":"8130855d-606b-4ce2-9c96-dfdde1c62622","html_url":"https://github.com/VSainteuf/utae-paps","commit_stats":{"total_commits":16,"total_committers":2,"mean_commits":8.0,"dds":0.125,"last_synced_commit":"001fcbbada7988e4bd4c1c37bfead2c23d21df4a"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VSainteuf%2Futae-paps","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VSainteuf%2Futae-paps/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VSainteuf%2Futae-paps/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VSainteuf%2Futae-paps/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VSainteuf","download_url":"https://codeload.github.com/VSainteuf/utae-paps/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224737431,"owners_count":17361345,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agriculture","computer-vision","deep-learning","iccv","iccv2021","panoptic-segmentation","satellite-imagery","self-attention","semantic-segmentation","sentinel-2"],"created_at":"2024-08-03T03:01:32.794Z","updated_at":"2025-05-08T13:32:17.227Z","avatar_url":"https://github.com/VSainteuf.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook"],"sub_categories":[],"readme":"# Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks (ICCV 2021)\n\n![](gfx/predictions.png)\n\nThis repository is the official implementation of [Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks\n](https://arxiv.org/abs/2107.07933). \n\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/panoptic-segmentation-of-satellite-image-time/semantic-segmentation-on-pastis)](https://paperswithcode.com/sota/semantic-segmentation-on-pastis?p=panoptic-segmentation-of-satellite-image-time)\n\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/panoptic-segmentation-of-satellite-image-time/panoptic-segmentation-on-pastis)](https://paperswithcode.com/sota/panoptic-segmentation-on-pastis?p=panoptic-segmentation-of-satellite-image-time)\n\n## Updates\n\n- **27.06.2022 Major Bugfix** :beetle: A bug in the panoptic metrics was driving the Recognition Quality down artificially. The bug is now fixed and the metrics have been updated here and on Arxiv. Across experiments, solving this bug improved PQ by ~2-3pts on PASTIS. See [this issue](https://github.com/VSainteuf/utae-paps/issues/11) for more details. \n\n## Contents\nThis repository contains the following PyTorch code:\n- [Implementation](https://github.com/VSainteuf/utae-paps/blob/main/src/backbones/utae.py) of **U-TAE** spatio-temporal encoding architecture for satellite image time series\n![UTAE](gfx/utae.png)\n- [Implementation](https://github.com/VSainteuf/utae-paps/blob/main/src/panoptic/paps.py) of **Parcels-as-Points (PaPs)** module for panoptic segmentation of agricultural parcels\n![PaPs](gfx/paps.png)\n- Code for reproduction of the paper's results for panoptic and semantic segmentation.\n\n\n\n\n## Results\n\nOur model achieves the following performance on :\n\n### PASTIS - Panoptic segmentation\n\nOur spatio-temporal encoder U-TAE combined with our PaPs instance segmentation module achieves 40.4 Panoptic Quality (PQ) on PASTIS for panoptic segmentation.\nWhen replacing U-TAE with a convolutional LSTM the performance drops to 33.4 PQ.\n\n| Model name         | SQ  | RQ | PQ|\n| ------------------ |--- | --- |--- |\n| **U-TAE + PaPs** (ours)      | **81.5**|**53.2** |**43.8**|\n| UConvLSTM+PaPs  | 80.2|   43.9   |  35.6|\n\n### PASTIS - Semantic segmentation\nOur spatio-temporal encoder U-TAE yields a semantic segmentation score of 63.1 mIoU on PASTIS, achieving an improvement of approximately 5 points compared to the best existing methods that we re-implemented (Unet-3d, Unet+ConvLSTM and Feature Pyramid+Unet).\nSee the paper for more details.\n\n| Model name         | #Params| OA  |  mIoU |\n| ------------------ |---- |---- | ---|\n| **U-TAE**  (ours) |   **1.1M**|  **83.2%**    | **63.1%**|\n| Unet-3d   | 1.6M|    81.3%    |  58.4%|\n| Unet-ConvLSTM |1.5M  |     82.1%    |  57.8%|\n| FPN-ConvLSTM  | 1.3M|    81.6%   |  57.1%|\n\n\n\n## Requirements\n\n### PASTIS Dataset download\nThe Dataset is freely available for download [here](https://github.com/VSainteuf/pastis-benchmark). \n\n\n\n### Python requirements\nTo install requirements:\n\n```setup\npip install -r requirements.txt\n```\n\n(`torch_scatter` is required for the panoptic experiments. \nInstalling this library requires a little more effort, see [the official repo](https://github.com/rusty1s/pytorch_scatter))\n\n\n\n## Inference with pre-trained models\n\n### Panoptic segmentation\n\n\nPre-trained weights of U-TAE+Paps are available [here](https://zenodo.org/record/5172301)\n\nTo perform inference of the pre-trained model on the test set of PASTIS run:\n\n```test\npython test_panoptic.py --dataset_folder PATH_TO_DATASET --weight_folder PATH_TO_WEIGHT_FOLDER --res_dir OUPUT_DIR\n```\n\n\n### Semantic segmentation\n\n\nPre-trained weights of U-TAE are available [here](https://zenodo.org/record/5172293)\n\nTo perform inference of the pre-trained model on the test set of PASTIS run:\n\n```test\npython test_semantic.py --dataset_folder PATH_TO_DATASET --weight_folder PATH_TO_WEIGHT_FOLDER --res_dir OUPUT_DIR\n```\n\n\n## Training models from scratch\n\n### Panoptic segmentation\n\nTo reproduce the main result for panoptic segmentation (with U-TAE+PaPs) run the following :\n\n```train\npython train_panoptic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR\n```\nOptions are also provided in `train_panoptic.py` to reproduce the other results of Table 2:\n\n```train\npython train_panoptic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_NoCNN --no_mask_conv\npython train_panoptic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_UConvLSTM --backbone uconvlstm\npython train_panoptic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_shape24 --shape_size 24\n```\n\nNote: By default this script runs the 5 folds of the cross validation, which can be quite long (~12 hours per fold on a Tesla V100). \nUse the fold argument to execute one of the 5 folds only \n(e.g. for the 3rd fold : `python train_panoptic.py --fold 3 --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR`).\n\n### Semantic segmentation\n\nTo reproduce results for semantic segmentation (with U-TAE) run the following :\n\n```train\npython train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR\n```\n\nAnd in order to obtain the results of the competing methods presented in Table 1 :\n\n```train\npython train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_UNET3d --model unet3d\npython train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_UConvLSTM --model uconvlstm\npython train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_FPN --model fpn\npython train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_BUConvLSTM --model buconvlstm\npython train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_COnvGRU --model convgru\npython train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_ConvLSTM --model convlstm\n\n```\nFinally, to reproduce the ablation study presented in Table 1 :\n\n```train\npython train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_MeanAttention --agg_mode att_mean\npython train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_SkipMeanConv --agg_mode mean\npython train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_BatchNorm --encoder_norm batch\npython train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_SingleDate --mono_date \"08-01-2019\"\n\n```\n\n### Reference\n\nPlease include a citation to the following paper if you use the U-TAE, PaPs or the PASTIS benchmark.\n\n```\n@article{garnot2021panoptic,\n  title={Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks},\n  author={Sainte Fare Garnot, Vivien  and Landrieu, Loic },\n  journal={ICCV},\n  year={2021}\n}\n\n```\n\n### Credits\n \n- This work was partly supported by [ASP](https://www.asp-public.fr), the French Payment Agency. \n\n- Code for the presented methods and dataset is original code by Vivien Sainte Fare Garnot,\n competing methods and some utility functions were adapted from existing repositories which are credited in the corresponding files.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FVSainteuf%2Futae-paps","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FVSainteuf%2Futae-paps","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FVSainteuf%2Futae-paps/lists"}