https://github.com/azavea/cloud-model
Cloud Detection Model for Sentinel-2 Imagery (see https://registry.opendata.aws/sentinel-2/)
https://github.com/azavea/cloud-model
cloud-detection machine-learning sentinel-2
Last synced: 2 months ago
JSON representation
Cloud Detection Model for Sentinel-2 Imagery (see https://registry.opendata.aws/sentinel-2/)
- Host: GitHub
- URL: https://github.com/azavea/cloud-model
- Owner: azavea
- License: mit
- Created: 2020-09-03T12:33:08.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2023-09-05T17:35:55.000Z (almost 2 years ago)
- Last Synced: 2025-04-03T07:12:43.807Z (3 months ago)
- Topics: cloud-detection, machine-learning, sentinel-2
- Language: Python
- Homepage:
- Size: 42 KB
- Stars: 7
- Watchers: 5
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
# Inference #
## Obtain Docker Image ##
Either build the docker image or pull it from quay.io.
### Build ###
Note that the model bundles necessary to build this image are not checked into this repository.
The model bundless can be obtained by typing the following.
```bash
cd inference/
aws s3 sync s3://azavea-cloud-model/models models --request-payer requester
```
The docker image can be built by typing the following (with or without the change of directory, as appropriate).
```bash
cd inference/
docker build -f Dockerfile -t quay.io/jmcclain/cloud-model:latest .
```
### Pull from quay.io ###
```
docker pull quay.io/jmcclain/cloud-model:latest
```
## Perform Inference ##
```bash
cd inference/
docker run -it --rm \
--runtime=nvidia --shm-size 16G \
-v $HOME/Desktop/imagery:/input:ro \
-v /tmp:/output \
quay.io/jmcclain/cloud-model \
--infile /input/greenwhich/L2A-0.tif \
--outfile-final /output/final.tif \
--outfile-raw /output/raw.tif \
--level L2A \
--architectures both
```
# Training #
## Build Docker Image ##
```bash
cd training/
docker build -t azavea-cloud-model-training -f Dockerfile .
```
## Run Container ##
```bash
cd training/
docker run -it --rm \
--name azavea-cloud-model-training --runtime=nvidia \
--shm-size 16G \
-v $HOME/.aws:/root/.aws:ro \
azavea-cloud-model-training bash
```
## Invoke Raster-Vision ##
### Local ###
```bash
export AWS_REQUEST_PAYER=requester
export ROOT=/tmp/xxx
rastervision run inprocess /workdir/pipeline.py \
-a root_uri ${ROOT} \
-a analyze_uri ${ROOT}/analyze \
-a chip_uri ${ROOT}/chips \
-a json catalogs.json \
-a epochs 2 \
-a batch_sz 2 \
-a small_test True \
chip train
```
### On AWS ###
It is required to have a compute environment with [additional storage](https://aws.amazon.com/premiumsupport/knowledge-center/batch-ebs-volumes-launch-template/) for the `p3.2xlarge` batch instance that is used for training. (The large number of chips will not fit on a volume of the default size.)
#### Chip ####
```bash
export AWS_REQUEST_PAYER='requester'
export LEVEL='L1C'
export ROOT='s3://bucket/prefix'
rastervision run batch /workdir/pipeline.py \
-a root_uri ${ROOT}/xxx \
-a analyze_uri ${ROOT}/${LEVEL}/analyze \
-a chip_uri ${ROOT}/${LEVEL}/chips \
-a json catalogs.json \
-a level ${LEVEL} \
-s 800 \
chip
```
#### Train ####
```bash
export LEVEL='L1C'
export ARCH='cheaplab'
export ROOT='s3://bucket/prefix'
rastervision run batch /workdir/pipeline.py \
-a root_uri ${ROOT}/${ARCH}-${LEVEL} \
-a analyze_uri ${ROOT}/${LEVEL}/analyze \
-a chip_uri ${ROOT}/${LEVEL}/chips \
-a json catalogs.json \
-a level ${LEVEL} \
-a architecture ${ARCH} \
train
```
# Licenses #
The code in this repository is licensed under the terms given in the [LICENSE.md](LICENSE.md) file.
The dataset is licensed under the terms of the [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/).