{"id":28789965,"url":"https://github.com/sralter/low-res-change-detection","last_synced_at":"2026-02-10T19:06:46.894Z","repository":{"id":298967788,"uuid":"1001698232","full_name":"sralter/low-res-change-detection","owner":"sralter","description":"Using free, low-resolution satellite imagery sourced from Google Earth Engine and organized using xarray, I trained a change detection image model based in PyTorch.","archived":false,"fork":false,"pushed_at":"2025-06-16T15:50:43.000Z","size":32987,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-30T12:45:07.817Z","etag":null,"topics":["docker","google-earth-engine","pytorch","streamlit","xarray"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sralter.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-13T20:56:48.000Z","updated_at":"2025-06-19T07:05:31.000Z","dependencies_parsed_at":null,"dependency_job_id":"bb43ec15-7ae9-4b90-afa2-35d104614f57","html_url":"https://github.com/sralter/low-res-change-detection","commit_stats":null,"previous_names":["sralter/low-res-change-detection"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sralter/low-res-change-detection","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sralter%2Flow-res-change-detection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sralter%2Flow-res-change-detection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sralter%2Flow-res-change-detection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sralter%2Flow-res-change-detection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sralter","download_url":"https://codeload.github.com/sralter/low-res-change-detection/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sralter%2Flow-res-change-detection/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29313092,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-10T17:48:59.043Z","status":"ssl_error","status_checked_at":"2026-02-10T17:45:37.240Z","response_time":65,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","google-earth-engine","pytorch","streamlit","xarray"],"created_at":"2025-06-17T22:38:43.675Z","updated_at":"2026-02-10T19:06:46.880Z","avatar_url":"https://github.com/sralter.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# low-res-change-detection\n\nBy Samuel Alter\n\nThe project was completed with the support of [Reveal Global Consulting](https://www.revealgc.com).  \nThank you to the whole team for their helpful comments and contributions.\n\n## Overview\n\nThis repository provides a proof-of-concept pipeline for training a change detection model using low-resolution satellite imagery. The key tools used include Google Earth Engine, xarray, and PyTorch. The workflow consists of data collection, preprocessing, model training (including hyperparameter search), inference, and optional visualization of the neural network architecture.\n\nThe model integrates the following data:\n* True color imagery\n* NDVI\n* Elevation\n* Slope\n* Aspect\n\n## Table of Contents \u003ca name='table-of-contents'\u003e\u003c/a\u003e\n\n1. [Requirements](#requirements)\n   * [Dependencies](#dependencies)\n2. [Project Structure](#project_structure)\n3. [Secrets Configuration](#secrets)\n4. [Usage](#usage)\n   * [Containerized Usage with Docker](#container)\n   * [Running the Scripts](#script)\n      * [Build Dataset](#build_dataset)\n      * [Inspect Dataset](#inspect_dataset)\n      * [Train Model](#train_vae)\n      * [Run Hyperparameter Search](#hp_search)\n      * [Inference](#inference)\n      * [Visualize Neural Network (optional)](#visualize_nn)\n5. [For Further Reading](#further)\n\n## 1. Requirements \u003ca name='requirements'\u003e\u003c/a\u003e\n\n[Back to TOC](#table-of-contents)\n\n* Python 3.12.9+\n* Google Earth Engine Python API Key (see [below](#gee))\n* AWS credentials JSON (see [below](#gee))\n* xarray, rasterio, rio-xarray\n* PyTorch\n* Optuna (for hyperparameter search)\n* numpy, pandas, matplotlib\n* streamlit (optional, for front-end interaction)\n\n### Install\n\n```bash\nuv pip install --upgrade pip\nuv pip install -r requirements.txt\n```\n\n### API Keys \u003ca name='gee'\u003e\u003c/a\u003e\n\n[Back to TOC](#table-of-contents)\n\n**For Google Earth Engine**:  \nGo to the [Earth Engine Apps guide](https://developers.google.com/earth-engine/guides/app_key) and create an API key.\n\n**For AWS**:  \nWe recommend creating your own S3 bucket when saving datasets to the cloud. Follow AWS' instructions on how to retrieve the bucket's credentials for your use.\n\n### Install dependencies \u003ca name='dependencies'\u003e\u003c/a\u003e\n\n[Back to TOC](#table-of-contents)\n\nUse the provided requirements.txt to install the dependencies for this project:\n\n```bash\nuv pip install -r requirements.txt\n```\n\n## 2. Project Structure \u003ca name='project_structure'\u003e\u003c/a\u003e\n\n[Back to TOC](#table-of-contents)\n\n```text\n├── documents/                  # Project reports, presentations, and other literature\n└── results/                    # Where the best trained model and older training runs are saved\n    ├── _old/                   # Older training runs are saved here\n    └── hp_1/                   # Best trained model, inference, and tensorboard files\n└── secrets_templates/          # Template secret files for credentials\n    ├── aws_creds_file_template.json\n    ├── google_earth_engine_creds_file_template.json\n    └── email_creds_file_template.json\n└── sources/                    # Folders for other projects that inspired this one\n└── streamlit/                  # Files supporting the presentation and project demo\n    ├── Home.py                 # Script that defines Streamlit dashboard\n    ├── best_model.pt           # Copy of best model found in hp_1 folder above\n    ├── build_dataset_streamlit.py      # Streamlit-specific script of build_dataset.py\n    ├── helpers.py              # Script with helper functions for Streamlit app\n    ├── inference_streamlit.py  # Streamlit-specific script of inference.py\n    ├── inspect_dataset_streamlit.py    # Streamlit-specific script of inspect_dataset.py\n    ├── prepare_dataset.py      # Copy of prepare_dataset.py for Streamlit app\n    ├── vae.py                  # Copy of vae.py for Streamlit app\n    └── pages/\n        ├── 1_Presentation_and_Demo.py  # Script to show Google Doc presentation on one side and demo on the other\n        └── 2_Only_Presentation.py      # Script to show just the Google Doc presentation\n├── .gitignore.txt              # .gitignore document\n├── README.md                   # README document\n├── build_dataset.py            # Collect and organize Sentinel-2 imagery via GEE and xarray into a dataset\n├── hp_search.py                # Entry point: hyperparameter search orchestration with Optuna\n├── inference.py                # Run inference on new imagery using trained model\n├── inspect_dataset.py          # Inspect and summarize dataset statistics\n├── prepare_dataset.py          # Further dataset preparation (e.g., NDVI computation)\n├── requirements.txt            # Python dependencies\n├── train_vae.py                # Entry point: train VAE model in one training run\n├── trainer.py                  # Trainer utilities (logging, checkpointing, metrics)\n├── vae.py                      # Define the VAE model architecture\n├── visualize_nn.py             # (Optional) Visualize network architecture\n```\n\n## 3. Secrets Configuration \u003ca name='secrets'\u003e\u003c/a\u003e\n\n[Back to TOC](#table-of-contents)\n\nYou will need to get an AWS credentials file, a Google Earth Engine account, and if you want the `hp_search.py` script to send you an email, you will also need to set up an app-specific password for your mail client.\n\n## 4. Usage \u003ca name='usage'\u003e\u003c/a\u003e\n\n[Back to TOC](#table-of-contents)\n\n### Containerized Usage with Docker \u003ca name='container'\u003e\u003c/a\u003e\n\n[Back to TOC](#table-of-contents)\n\nWe’ve provided a `Dockerfile` and `docker-compose.yml` so you can “clone → build → run” without worrying about system setup.\n\nThe workflow is simple:\n1. Build the docker image\n   ```bash\n   git clone https://github.com/sralter/low-res-change-detection.git\n   cd low-res-change-detection\n   docker compose up --build -d\n   ```\n2. Prepare secrets folder...\n   ```bash\n   mkdir secrets\n   cp secrets_templates/google_earth_engine_creds_file_template.json secrets/google_earth_engine_creds_file.json\n   cp secrets_templates/aws_creds_file_template.json secrets/aws_creds_file.json\n   cp secrets_templates/email_creds_file_template.json secrets/email_creds_file.json\n   ```\n3. ...And populate the .JSONs in the secrets folder with your credentials\n4. Execute the scripts\n   ```bash\n   docker compose run --rm app build_dataset.py [your args …]\n   ```\n\n### Running the Scripts \u003ca name='script'\u003e\u003c/a\u003e\n\n[Back to TOC](#table-of-contents)\n\nRefer to the below sections when running each process. All scripts are run from the command line. Our guidance is to run the following scripts in this order. With `train_vae` and `hp_search`, you can choose either a single training run with `train_vae.py`, or a hyperparameter search with `hp_search.py`.\n1. [Build Dataset.py](#build_dataset)\n2. [Inspect Dataset.py](#inspect_dataset)\n3. [Train VAE](#train_vae) **OR** [HP Search](#hp_search)\n4. [Inference](#inference)\n5. [Visualize NN](#visualize_nn)\n\nIf you want to run the scripts within Docker, instead of starting with `python` as in: `python script_name.py --arg1 --arg2 ...`, start with: `docker compose run --rm app script_name.py --arg1 --arg2 ...` instead.\n\n#### 1. Build Dataset \u003ca name='build_dataset'\u003e\u003c/a\u003e\n\n[Back to TOC](#table-of-contents)\n\n```bash\npython build_dataset.py \\\n    --bucket rgc-zarr-store \\\n    --folder data \\\n    --ee-account-key secrets/low-res-sat-change-detection-f7e0f971189b.json \\\n    --ee-account-email low-res-sat-change-detection@low-res-sat-change-detection.iam.gserviceaccount.com \\\n    --aws-creds-file secrets/aws_rgc-zarr-store.json \\\n    --geohashes 9vgm0,9vgm1\n```\n\n#### 2. Inspect Dataset \u003ca name='inspect_dataset'\u003e\u003c/a\u003e\n\n[Back to TOC](#table-of-contents)\n\n```bash\npython inspect_dataset.py \\\n    --bucket rgc-zarr-store \\\n    --folder data \\\n    --geohashes 9v1z2 \\\n    --output outputs \\\n    --aws-creds secrets/aws_rgc-zarr-store.json \\\n    --dates first last\n```\n\n#### 3a. Train Model \u003ca name='train_vae'\u003e\u003c/a\u003e\n\n[Back to TOC](#table-of-contents)\n\nWe recommend using the `hp_search.py` script first so that Optuna can search the hyperparameter space, then run those best parameters in `train_vae.py`\n\n```bash\npython train_vae.py \\\n    --bucket rgc-zarr-store \\\n    --folder data \\\n    --train-geohashes 9vgm0,9vgm1,9vgm2 \\\n    --val-geohashes 9vgm3 \\\n    --test-geohashes 9vgm4 \\\n    --out results \\\n    --aws-creds secrets/aws_rgc-zarr-store.json \\\n    --num-workers 4 \\\n    --stage-zarr\n```\n\n#### 3b. Run Hyperparameter Search \u003ca name='hp_search'\u003e\u003c/a\u003e\n\n[Back to TOC](#table-of-contents)\n\n```bash\npython hp_search.py \\\n    --trials 20 \\\n    --bucket rgc-zarr-store \\\n    --folder data \\\n    --train-geohashes 9vgm0,9vgm1,9vgm2 \\\n    --val-geohashes 9vgm3 \\\n    --test-geohashes 9vgm4 \\\n    --out results/hp_run_1 \\\n    --aws-creds secrets/aws_rgc-zarr-store.json \\\n    --num-workers 0 \\\n    --stage-zarr \\\n    --trial-epochs 5 \\\n    --trial-patience 3\n```\n\n#### 4. Inference \u003ca name='inference'\u003e\u003c/a\u003e\n\n[Back to TOC](#table-of-contents)\n\n```bash\npython inference.py \\\n    --bucket rgc-zarr-store \\\n    --folder data \\\n    --geohash 9vgm0 \\\n    --model path/to/best/model.pt \\\n    --dates first last \\\n    --aws-creds-file secrets/aws_rgc-zarr-store.json \\\n    --out outputs \\\n    --stage-zarr\n```\n\n#### 5. Visualize Neural Network (Optional) \u003ca name='visualize_nn'\u003c/a\u003e\n\n[Back to TOC](#table-of-contents)\n\n```bash\npython visualize_nn.py \\\n    --model path/to/best/model.pt \\\n```\n\n## 5. Further Reading \u003ca name='further'\u003e\u003c/a\u003e\n\n[Back to TOC](#table-of-contents)\n\nIf you want to learn more about the project and learn the \"how\" and \"why\", I urge you to read the final project report and look at the slideshow presentation, both in the documents folder.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsralter%2Flow-res-change-detection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsralter%2Flow-res-change-detection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsralter%2Flow-res-change-detection/lists"}