Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/cordmaur/WaterDetect
Water Detect Algorithm
https://github.com/cordmaur/WaterDetect
Last synced: 3 months ago
JSON representation
Water Detect Algorithm
- Host: GitHub
- URL: https://github.com/cordmaur/WaterDetect
- Owner: cordmaur
- License: apache-2.0
- Created: 2019-11-29T10:34:57.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2024-06-10T11:30:16.000Z (5 months ago)
- Last Synced: 2024-06-11T19:51:20.935Z (5 months ago)
- Language: Jupyter Notebook
- Size: 943 KB
- Stars: 186
- Watchers: 6
- Forks: 35
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- open-sustainable-technology - WaterDetect - End-to-end algorithm to generate open water cover mask, specially conceived for L2A Sentinel 2 imagery from MAJA1 processor, without any a priori knowledge on the scene. (Hydrosphere / Freshwater and Hydrology)
README
# WaterDetect
[](https://zenodo.org/badge/latestdoi/224832878)
## Synopsis
WaterDetect is an end-to-end algorithm to generate open water cover mask, specially conceived for L2A Sentinel 2 imagery from [MAJA](https://logiciels.cnes.fr/en/content/maja)1 processor, without any a priori knowledge on the scene. It can also be used for Landsat 8 images and for other multispectral clustering/segmentation tasks.
The water masks produced by WaterDetect were primarily designed for water quality product computation (Obs2Co processing chain) and are also used for multi-temporal water maps (Surfwater processing chain). Both chains are supported by the "SWOT-Downstream" and TOSCA programs by CNES. Products are provided by the THEIA / Hydroweb-NG platform.
The WaterDetect algorithm uses a multidimensional agglomerative clustering technique on a subsample of the scene's pixels, to group them in classes, and a naive bayes classifier to generalize the results for the whole scene, as summarized in the following picture:
All the details and tests has been described in the article Automatic Water Detection from Multidimensional Hierarchical Clustering for Sentinel-2 Images and a Comparison with Level 2A Processors, under revision by the journal Remote Sensing of Environment.
## How to cite
Cordeiro, M. C. R.; Martinez, J.-M.; Peña-Luque, S. Automatic Water Detection from Multidimensional Hierarchical Clustering for Sentinel-2 Images and a Comparison with Level 2A Processors. Remote Sensing of Environment 2021, 253, 112209. https://doi.org/10.1016/j.rse.2020.112209.
## Changelog
### Release 1.5.15
- Corrected PyPDF2 deprecation error.
### Release 1.5.13
- Code updated to comply with Sen2Cor processing baseline 04.00, that started in January, 25th 2022.
BoA offset was added to the Sen2Cor workflow to deal with negative values. Oficial release:
https://sentinels.copernicus.eu/documents/247904/4830984/OMPC.CS.DQR.002.07-2022%20-%20i52r0%20-%20MSI%20L2A%20DQR%20August%202022.pdf/36edbb04-0c6c-fba3-5c34-0ba3be82e91c
### Release 1.5.12
- Minor updates to make it compatible with the `waterquality` package. For more information, check the waterquality package here: https://github.com/cordmaur/WaterQuality
### Release 1.5.11
- Bug fix when loading L1C and S2COR images, from MacOS.
### Release 1.5.9
- Added external mask processing through the command `process_ext_masks`. It prepares an external mask (ex. created by FMask) to be used by WaterDetect.
### Release 1.5.8
- correct GlintMode to work on S2_THEIA images
- Add calculation of MBWI index inside DWImageClustering class
### Release 1.5.7
- new entry point runWaterDetect.py
- Namespace correction for different versions of sklearn package
- New treatment for negative reflectance values (individual pixel correction)
- Added regularization option to avoid extreme values on Normalized Difference indices.
- New water cluster detection method based on lowest Nir reflectance ("minnir")
- Updated reporting. Invalid mask is superimposed to the RGB scene representation
- Added support for Sen2Cor internal masks
- GLINT mode (For entire scenes only). Creates a Glint heatmap based on viewing and solar angles and updates the thresholds to include waters with sun glint in the final mask
## Tutorial
The following topics have the first steps to install and run the library. For a more comprehensive tutorial with code samples and results please refer to this tutorial https://cordmaur.medium.com/water-detection-in-high-resolution-satellite-images-using-the-waterdetect-python-package-7c5a031e3d16.
## Supported Formats
The algorithm has been developed taking into account atmospherically corrected images from MAJA, as described in the paper. However other image formats are also supported.
To the present, the following image formats are supported:
* Sentinel 2 - L2A from MAJA: the products can be downloaded from (https://www.theia-land.fr/en/product/sentinel-2-surface-reflectance/)
* Sentinel 2 - L2A from Sen2Cor: The L2A processed by Sen2Cor are available at Copernicus SciHub (https://scihub.copernicus.eu/)
* Sentinel 2 - L1C: L1C Sentinel 2 images can be downloaded from Copernicus SciHub (https://scihub.copernicus.eu/)
* Landsat 8 - To be validated
## Dependencies
The required libraries are:
```
GDAL>=3.0.2
matplotlib>=3.1.2
PyPDF2>=1.26.0
scipy>=1.3.2
scikit-learn>=0.22
scikit-image>=0.16.2
numpy>=1.17
PIL>=8.0
lxml>=4.5
```
### Note 1:
GDAL is required to open the satellite images. It's still possible to use without GDAL, from a python console or jupyter notebook, loading the rasters manually and passing all the necessary bands to the DWImageClustering class. Check the topic "Usage from Console" for more information.
### Note 2:
Scikit-Image is only necessary to run Otsu threshold method.
The test_dependencies.py can be used to check if all libraries are loading correctly. Simply run:
## Instalation
The easiest way to install waterdetect package is with `pip` command:
`pip install waterdetect`
Alternatively, you can clone the repository and install from its root throught the following commands:
```
git clone https://github.com/cordmaur/WaterDetect.git
cd WaterDetect
pip install .
```
Once installed, a `waterdetect` entry point is created in the path of the environment.
One can check the installation and options by running `waterdetect --help`. If GDAL is not found, a message will raise indicating that waterdetect will only run from a console.
```
usage: waterdetect [-h] [-GC] [-i INPUT] [-o OUT] [-s SHP] [-p PRODUCT]
[-c CONFIG]
The waterdetect is a high speed water detection algorithm for satellite
images. It will loop through all images available in the input folder and
write results for every combination specified in the .ini file to the output
folder. It can also run for single images from Python console or Jupyter
notebook. Refer to the onlinedocumentation
optional arguments:
-h, --help show this help message and exit
-GC, --GetConfig Copy the WaterDetect.ini from the package into the
specifieddirectory and skips the processing. Once
copied you can edit the .ini file and launch the
waterdetect without -c option.
-i INPUT, --input INPUT
The products input folder. Required.
-o OUT, --out OUT Output directory. Required.
-s SHP, --shp SHP SHP file. Optional.
-p PRODUCT, --product PRODUCT
The product to be processed (S2_THEIA, L8_USGS, S2_L1C
or S2_S2COR)
-c CONFIG, --config CONFIG
Configuration .ini file. If not specified
WaterDetect.ini from current dir and used as default
To copy the package's default .ini file into the current directory, type:
`waterdetect -GC .` without other arguments and it will copy WaterDetect.ini
into the current directory.
```
### Config File
The waterdetect needs a config file that specifies the bands used in the clustering process as well as other parameters.
To obtain the default version of this file, one can use `waterdetec -GC` and the file WaterDetect.ini will be copied into the current working folder.
## Usage as Script
The basic usage for the waterdetect is:
`waterdetect -i c:/input_folder -o -c:/output_folder -p S2_THEIA`
The input directory should contain the uncompressed folders for the images. The script will loop through all folders in the input directory and save the water masks, graphs and reports to the output folder. The output folder must be created beforehand.
If the config file is not specified, the script will search for WaterDetect.ini in the current folder.
## Usage from Console
Once properly installed, the WaterDetect can be run from a console or a Jupyter Notebook, by importing the package and calling DWDetectWater.
```
>>> import waterdetect as wd
>>> !waterdetect -GC
>>> wd.DWWaterDetect.run_water_detect(input_folder='D:\Images\Input\',
output_folder='D:\Images\Output',
shape_file='D:\Shp\SomeShapefile.shp',
single_mode=False,
product=wd.DWProducts.Sentinel2_THEIA,
config_file='WaterDetect.ini'
)
```
For more information on how to use it from jupyter notebook, in batch or single mode or to use it with other satellite images or without GDAL, please refer to the tutorial available here https://towardsdatascience.com/water-detection-in-high-resolution-satellite-images-using-the-waterdetect-python-package-7c5a031e3d16.
## Contributors
> Author: Maurício Cordeiro (ANA/GET)
> Supervisor: Jean-Michel Martinez (IRD/GET)
> Validation dataset: Santiago Pena Luque (CNES)
### Institutions
* ANA - Agência Nacional de Águas (https://www.gov.br/ana/en/)
* GET - Géosciences Environnement Toulouse (https://www.get.omp.eu/)
* IRD - Institut de Recherche pour le Développement (https://en.ird.fr/)
* CNES - Centre National d'Études Spatiales (https://cnes.fr/fr)
## License
This code is licensed under the [GNU General Public License v3.0](https://github.com/cordmaur/WaterDetect/blob/master/LICENSE) license. Please, refer to GNU's webpage (https://www.gnu.org/licenses/gpl-3.0.en.html) for details.
## Reference
(1) Hagolle, O.; Huc, M.; Pascual, D. V.; Dedieu, G. A Multi-Temporal Method for Cloud Detection, Applied to FORMOSAT-2, VENµS, LANDSAT and SENTINEL-2 Images. Remote Sensing of Environment 2010, 114 (8), 1747–1755. https://doi.org/10.1016/j.rse.2010.03.002.
(2) Cordeiro, M. C. R.; Martinez, J.-M.; Peña-Luque, S. Automatic Water Detection from Multidimensional Hierarchical Clustering for Sentinel-2 Images and a Comparison with Level 2A Processors. Remote Sensing of Environment 2021, 253, 112209. https://doi.org/10.1016/j.rse.2020.112209.