{"id":19422337,"url":"https://github.com/azavea/cloud-model","last_synced_at":"2025-07-24T23:37:17.545Z","repository":{"id":46833490,"uuid":"292564897","full_name":"azavea/cloud-model","owner":"azavea","description":"Cloud Detection Model for Sentinel-2 Imagery (see https://registry.opendata.aws/sentinel-2/)","archived":false,"fork":false,"pushed_at":"2023-09-05T17:35:55.000Z","size":43,"stargazers_count":7,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-03T07:12:43.807Z","etag":null,"topics":["cloud-detection","machine-learning","sentinel-2"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/azavea.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-09-03T12:33:08.000Z","updated_at":"2025-02-16T04:51:23.000Z","dependencies_parsed_at":"2024-11-10T13:35:25.071Z","dependency_job_id":"8065b38c-1bdd-4eff-bd19-40734452612d","html_url":"https://github.com/azavea/cloud-model","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azavea%2Fcloud-model","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azavea%2Fcloud-model/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azavea%2Fcloud-model/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azavea%2Fcloud-model/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/azavea","download_url":"https://codeload.github.com/azavea/cloud-model/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250654450,"owners_count":21465885,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-detection","machine-learning","sentinel-2"],"created_at":"2024-11-10T13:33:25.017Z","updated_at":"2025-04-24T15:32:14.787Z","avatar_url":"https://github.com/azavea.png","language":"Python","readme":"![The Clouds](https://upload.wikimedia.org/wikipedia/commons/4/46/Socrates_in_a_basket.jpg)\n\n# Inference #\n\n## Obtain Docker Image ##\n\nEither build the docker image or pull it from quay.io.\n\n### Build ###\n\nNote that the model bundles necessary to build this image are not checked into this repository.\nThe model bundless can be obtained by typing the following.\n```bash\ncd inference/\naws s3 sync s3://azavea-cloud-model/models models --request-payer requester\n```\n\nThe docker image can be built by typing the following (with or without the change of directory, as appropriate).\n```bash\ncd inference/\ndocker build -f Dockerfile -t quay.io/jmcclain/cloud-model:latest .\n```\n\n### Pull from quay.io ###\n\n```\ndocker pull quay.io/jmcclain/cloud-model:latest\n```\n\n## Perform Inference ##\n\n```bash\ncd inference/\ndocker run -it --rm \\\n       --runtime=nvidia --shm-size 16G \\\n       -v $HOME/Desktop/imagery:/input:ro \\\n       -v /tmp:/output \\\n       quay.io/jmcclain/cloud-model \\\n          --infile /input/greenwhich/L2A-0.tif \\\n          --outfile-final /output/final.tif \\\n          --outfile-raw /output/raw.tif \\\n          --level L2A \\\n          --architectures both\n```\n\n# Training #\n\n## Build Docker Image ##\n\n```bash\ncd training/\ndocker build -t azavea-cloud-model-training -f Dockerfile .\n```\n\n## Run Container ##\n\n```bash\ncd training/\ndocker run -it --rm \\\n       --name azavea-cloud-model-training --runtime=nvidia \\\n       --shm-size 16G \\\n       -v $HOME/.aws:/root/.aws:ro \\\n       azavea-cloud-model-training bash\n```\n\n## Invoke Raster-Vision ##\n\n### Local ###\n\n```bash\nexport AWS_REQUEST_PAYER=requester\nexport ROOT=/tmp/xxx\nrastervision run inprocess /workdir/pipeline.py \\\n       -a root_uri ${ROOT} \\\n       -a analyze_uri ${ROOT}/analyze \\\n       -a chip_uri ${ROOT}/chips \\\n       -a json catalogs.json \\\n       -a epochs 2 \\\n       -a batch_sz 2 \\\n       -a small_test True \\\n       chip train\n```\n\n### On AWS ###\n\nIt is required to have a compute environment with [additional storage](https://aws.amazon.com/premiumsupport/knowledge-center/batch-ebs-volumes-launch-template/) for the `p3.2xlarge` batch instance that is used for training.  (The large number of chips will not fit on a volume of the default size.)\n\n#### Chip ####\n\n```bash\nexport AWS_REQUEST_PAYER='requester'\nexport LEVEL='L1C'\nexport ROOT='s3://bucket/prefix'\nrastervision run batch /workdir/pipeline.py \\\n       -a root_uri ${ROOT}/xxx \\\n       -a analyze_uri ${ROOT}/${LEVEL}/analyze \\\n       -a chip_uri ${ROOT}/${LEVEL}/chips \\\n       -a json catalogs.json \\\n       -a level ${LEVEL} \\\n       -s 800 \\\n       chip\n```\n\n#### Train ####\n\n```bash\nexport LEVEL='L1C'\nexport ARCH='cheaplab'\nexport ROOT='s3://bucket/prefix'\nrastervision run batch /workdir/pipeline.py \\\n       -a root_uri ${ROOT}/${ARCH}-${LEVEL} \\\n       -a analyze_uri ${ROOT}/${LEVEL}/analyze \\\n       -a chip_uri ${ROOT}/${LEVEL}/chips \\\n       -a json catalogs.json \\\n       -a level ${LEVEL} \\\n       -a architecture ${ARCH} \\\n       train\n```\n\n# Licenses #\n\nThe code in this repository is licensed under the terms given in the [LICENSE.md](LICENSE.md) file.\n\nThe dataset is licensed under the terms of the [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fazavea%2Fcloud-model","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fazavea%2Fcloud-model","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fazavea%2Fcloud-model/lists"}