{"id":13606210,"url":"https://github.com/developmentseed/skynet-train","last_synced_at":"2025-06-23T21:40:59.413Z","repository":{"id":66924979,"uuid":"60793169","full_name":"developmentseed/skynet-train","owner":"developmentseed","description":"Training and test the SegNet neural network on satellite imagery","archived":false,"fork":false,"pushed_at":"2018-05-02T14:35:01.000Z","size":5907,"stargazers_count":206,"open_issues_count":15,"forks_count":50,"subscribers_count":75,"default_branch":"master","last_synced_at":"2025-04-12T08:40:02.475Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/developmentseed.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-06-09T17:23:10.000Z","updated_at":"2024-10-05T07:25:48.000Z","dependencies_parsed_at":"2023-05-29T01:00:12.132Z","dependency_job_id":null,"html_url":"https://github.com/developmentseed/skynet-train","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/developmentseed/skynet-train","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developmentseed%2Fskynet-train","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developmentseed%2Fskynet-train/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developmentseed%2Fskynet-train/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developmentseed%2Fskynet-train/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/developmentseed","download_url":"https://codeload.github.com/developmentseed/skynet-train/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developmentseed%2Fskynet-train/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261561183,"owners_count":23177545,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T19:01:07.109Z","updated_at":"2025-06-23T21:40:59.389Z","avatar_url":"https://github.com/developmentseed.png","language":"JavaScript","readme":"\n# SegNet training and testing scripts\n\nThese scripts are for use in training and testing the [SegNet neural\nnetwork](http://mi.eng.cam.ac.uk/projects/segnet/), particularly with\nOpenStreetMap + Satellite Imagery training data generated by\n[skynet-data](https://github.com/developmentseed/skynet-data).\n\nContributions are very welcome!\n\n# Quick start\n\nThe quickest and easiest way to use these scripts is via the\n`developmentseed/skynet-train` docker image, but note that to make this work\nwith a GPU--necessary for reasonable training times---you will need a machine\nset up to use [`nvidia-docker`](https://github.com/NVIDIA/nvidia-docker).  (The\n[start_instance](https://github.com/developmentseed/skynet-train/blob/master/start_instance)\nscript uses `docker-machine` to spin up an AWS EC2 g2 instance and set it up with\nnvidia-docker. The [start_spot_instance](https://github.com/developmentseed/skynet-train/blob/master/start_spot_instance)\nscript does the same thing but creates a [spot](https://aws.amazon.com/ec2/spot/)\ninstance instead of an on demand one.)\n\n1. Create a training dataset with [skynet-data](https://github.com/developmentseed/skynet-data).\n2. Run:\n\n```sh\nnvidia-docker run \\\n    -v /path/to/training/dataset:/data \\\n    -v /path/to/training/output:/output \\\n    -e AWS_ACCESS_KEY_ID=... \\\n    -e AWS_SECRET_ACCESS_KEY=... \\\n    developmentseed/skynet-train:gpu \\\n      --sync s3://your-bucket/training/blahbla\n```\n\nThis will kick off a training run with the given data. Every 10000 iterations,\nthe model will be snapshotted and run on the test data, the training \"loss\"\nwill be plotted, and all of this uploaded to s3. (Omit the `--sync` argument\nand AWS creds to skip the upload.)\n\nEach batch of test results includes a `view.html` file that shows a bare-bones\nviewer allowing you to browse the results on a map and compare model outputs to\nthe ground truth data.  Use it like:\n - http://your-bucket-url/...test-dir.../view.html?imagery_source=MAPID\u0026access_token=MAPBOX_ACCESS_TOKEN where `MAPID` points to Mapbox-hosted raster tiles used for training. (Defaults to `mapbox.satellite`.)\n - http://your-bucket-url/...test-dir.../view.html?imagery_source=http://yourtiles.com/{z}/{x}/{y} for non-Mapbox imagery tiles\n\n\nCustomize the training run with these params:\n\n```\n--model MODEL # segnet or segnet_basic, defaults to segnet\n--output OUTPUT # directory in which to output training assets\n--data DATA # training dataset\n[--fetch-data FETCH_DATA] # s3 uri from which to download training data into DATA\n[--snapshot SNAPSHOT] # snapshot frequency\n[--cpu] # sets cpu mode\n[--gpu [GPU [GPU ...]]] # set gpu devices to use\n[--display-frequency DISPLAY_FREQUENCY] # frequency of logging output (affects granularity of plots)\n[--iterations ITERATIONS] # total number of iterations to run\n[--crop CROP] # crop trianing images to CROPxCROP pixels\n[--batch-size BATCH_SIZE] # batch size (adjust this up or down based on GPU size. defaults to 6 for segnet and 16 for segnet_basic)\n[--sync SYNC]\n```\n\n## Monitoring\n\nOn an instance where training is happening, expose a simple monitoring page with:\n\n```sh\ndocker run --rm -it -v /mnt/training:/output -p 80:8080 developmentseed/skynet-monitor\n```\n\n# Details\n\nPrerequisites / Dependencies:\n - Node and Python\n - As of now, training SegNet requires building the [caffe-segnet fork](https://github.com/alexgkendall/caffe-segnet) fork of Caffe.\n - Install node dependencies by running `npm install` in the root directory of this repo.\n\n## Set up model definition\n\nAfter creating a dataset with the [skynet-data](https://github.com/developmentseed/skynet-data)\nscripts, set up the model `prototxt` definition files by running:\n\n```\nsegnet/setup-model --data /path/to/dataset/ --output /path/to/training/workdir\n```\n\nAlso copy `segnet/templates/solver.prototxt` to the training work directory, and\nedit it to (a) point to the right paths, and (b) set up the learning\n\"hyperparameters\".\n\n(NOTE: this is hard to get right at first; when we post links to a couple of\npre-trained models, we'll also include a copy of the solver.prototxt we used as\na reference / starting point.)\n\n\n## Train\n\nDownload the pre-trained VGG weights `VGG_ILSVRC_16_layers.caffemodel` from\nhttp://www.robots.ox.ac.uk/~vgg/research/very_deep/\n\nFrom your training work directory, run\n\n```\n$CAFFE_ROOT/build/tools/caffe train -gpu 0 -solver solver.txt \\\n    -weights VGG_ILSVRC_16_layers.caffemodel \\\n    2\u003e\u00261 | tee train.log\n```\n\nYou can monitor the training with:\n\n```\nsegnet/util/plot_training_log.py train.log --watch\n```\n\nThis will generate and continually update a plot of the \"loss\" (i.e., training\nerror) which should gradually decrease as training progresses.\n\n## Testing the Trained Network\n\n```\nsegnet/run_test --output /path/for/test/results/ --train /path/to/segnet_train.prototxt --weights /path/to/snapshots/segnet_blahblah_iter_XXXXX.caffemodel --classes /path/to/dataset/classes.json\n```\n\nThis script essentially carries out the instructions outlined here:\nhttp://mi.eng.cam.ac.uk/projects/segnet/tutorial.html\n\n## Inference\n\nAfter you have a trained and tested network, you'll often want to use it to predict over a larger area. We've included scripts for running this process locally or on AWS.\n\n### Local Inference\n\nTo run predictions locally you'll need:\n- Raster imagery (as either a GeoTIFF or a VRT)\n- A line delimited list of [XYZ tile indices](https://wiki.openstreetmap.org/wiki/Slippy_map_tilenames) to predict on (e.g. `49757-74085-17`. These can be made with [geodex](https://github.com/developmentseed/geodex))\n- A skynet model, trained weights, and class definitions ( `.prototxt`, `.caffemodel`, `.json`)\n\nTo run:\n\n```sh\ndocker run -v /path/to/inputs:/inputs -v /path/to/model:/model -v /path/to/output/:/inference \\\n  developmentseed:/skynet-run:local-gpu /inputs/raster.tif /inputs/tiles.txt \\\n  --model /model/segnet_deploy.prototxt\n  --weights /model/weights.caffemodel\n  --classes /model/classes.json\n  --output /inference\n```\n\nIf you are running on a CPU, use the `:local-cpu` docker image and add `--cpu-only` as a final flag to the above command.\n\nThe predicted rasters and vectorized geojson outputs will be located in `/inference` (and the corresponding mounted volume)\n\n### AWS Inference\n\nTODO: for now, see command line instructions in `segnet/queue.py` and `segnet/batch_inference.py`\n\n## GPU\n\nThese scripts were originally developed for use on an AWS `g2.2xlarge` instance. For support on newer GPUs, it may be required to:\n- use a [newer NVIDIA driver](https://github.com/developmentseed/skynet-train/blob/master/user_data.sh#L22)\n- use a newer version of CUDA. To support CUDA8+, you can use the docker images tagged with `:cuda8`. They are built off an updated [`caffe-segnet` fork](https://github.com/TimoSaemann/caffe-segnet-cudnn5) with support for `cuDNN5`.\n","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevelopmentseed%2Fskynet-train","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevelopmentseed%2Fskynet-train","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevelopmentseed%2Fskynet-train/lists"}