{"id":13454792,"url":"https://github.com/trailbehind/DeepOSM","last_synced_at":"2025-03-24T06:31:27.106Z","repository":{"id":41407119,"uuid":"51717338","full_name":"trailbehind/DeepOSM","owner":"trailbehind","description":"Train a deep learning net with OpenStreetMap features and satellite imagery.","archived":false,"fork":false,"pushed_at":"2017-03-13T16:09:05.000Z","size":1029,"stargazers_count":1322,"open_issues_count":13,"forks_count":182,"subscribers_count":86,"default_branch":"master","last_synced_at":"2025-03-19T11:16:47.730Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/trailbehind.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-02-14T22:30:33.000Z","updated_at":"2025-03-10T13:25:30.000Z","dependencies_parsed_at":"2022-08-27T06:41:48.685Z","dependency_job_id":null,"html_url":"https://github.com/trailbehind/DeepOSM","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailbehind%2FDeepOSM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailbehind%2FDeepOSM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailbehind%2FDeepOSM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trailbehind%2FDeepOSM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/trailbehind","download_url":"https://codeload.github.com/trailbehind/DeepOSM/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244560389,"owners_count":20472220,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T08:00:57.991Z","updated_at":"2025-03-24T06:31:27.072Z","avatar_url":"https://github.com/trailbehind.png","language":"Python","funding_links":[],"categories":["Models/Projects","Libraries","Python","模型项目","Model Deployment library","Projects"],"sub_categories":["Remote Sensing","微信群","Tensorflow \u003ca name=\"tensorflow\"/\u003e"],"readme":"# DeepOSM [![Build Status](https://travis-ci.org/trailbehind/DeepOSM.svg?branch=master)](https://travis-ci.org/trailbehind/DeepOSM)\n\nClassify roads and features in satellite imagery, by training neural networks with OpenStreetMap (OSM) data. \n\nDeepOSM can:\n\n* Download a chunk of satellite imagery\n* Download OSM data that shows roads/features for that area\n* Generate training and evaluation data\n* Display predictions of mis-registered roads in OSM data, or display raw predictions of ON/OFF\n\nRunning the code is as easy as install Docker, make dev, and run a script. \n\nContributions are welcome. Open an issue if you want to discuss something to do, or [email me](mailto:andrew@gaiagps.com).\n\n## Default Data/Accuracy\n\nBy default, DeepOSM will analyze about 200 sq. km of area in Delaware. DeepOSM will\n\n* predict if the center 9px of a 64px tile contains road.\n* use the infrared (IR) band and RGB bands.\n* be 75-80% accurate overall, training only for a minute or so.\n* use a single fully-connected relu layer in [TensorFlow](https://www.tensorflow.org/).\n* render, as JPEGs, \"false positive\" predictions in the OSM data - i.e. where OSM lists a road, but DeepOSM thinks there isn't one.\n\n![NAIP with Ways and Predictions](https://pbs.twimg.com/media/Cjk6fADUYAE0wvh.jpg)\n\n## Background on Data - NAIPs and OSM PBF\n\nFor training data, DeepOSM cuts tiles out of [NAIP images](http://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/), which provide 1-meter-per-pixel resolution, with RGB+infrared data bands.\n\nFor training labels, DeepOSM uses PBF extracts of OSM data, which contain features/ways in binary format that can be munged with Python.\n\nThe [NAIPs come from a requester pays bucket on S3 set up by Mapbox](http://www.slideshare.net/AmazonWebServices/open-data-innovation-building-on-open-data-sets-for-innovative-applications), and the OSM extracts come [from geofabrik](http://download.geofabrik.de/).\n\n## Install Requirements\n\nDeepOSM has been run successfully on both Mac (10.x) and Linux (14.04 and 16.04). You need at least 4GB of memory.\n\n### AWS Credentials\n\nYou need AWS credentials to download NAIPs from an S3 requester-pays bucket. This only costs a few cents for a bunch of images, but you need a credit card on file.\n\n * get your [AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from AWS](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html)\n\n * export them as environment variables (and maybe add to your bash or zprofile)\n\n```\nexport AWS_ACCESS_KEY_ID='FOO'\nexport AWS_SECRET_ACCESS_KEY='BAR'\n```\n\n### Install Docker\n\nFirst, [install a Docker Binary](https://docs.docker.com/engine/installation/).\n\nI also needed to set my VirtualBox default memory to 4GB, when running on a Mac. This is easy:\n\n * start Docker, per the install instructions\n * stop Docker\n * open VirtualBox, and increase the memory of the VM Docker made\n\n### (GPU Only) Install nvidia-docker\n\nIn order to use your GPU to accelerate DeepOSM, you will need to\ndownload and install the latest NVIDIA drivers for your GPU, and\n(after first installing docker itself), install `nvidia-docker`.\n\nFirst, find the latest NVIDIA drivers for your GPU on\n[NVIDIA's website](http://www.nvidia.com/Download/Find.aspx). Make\nsure you check the version number of the driver, as the most recent\nrelease isn't always the latest version.\n\nOnce you have downloaded the appropriate `NVIDIA-*.run` file, install\nit as follows (based on\n[these instructions](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/accelerated-computing-instances.html#install-nvidia-driver)):\n\nEnsure your system is up-to-date and reboot to ensure the latest\ninstalled kernel is loaded:\n\n```\n# ensure your packages are up-to-date\nsudo apt-get update\nsudo apt-get dist-upgrade\n# and reboot\nsudo reboot\n```\n\nOnce your system has rebooted, install `build-essential` and the\n`linux-headers` package for your current kernel version (or\nequivalents for your linux distribution):\n\n```\nsudo apt-get install build-essential linux-headers-$(uname -r) \n```\n\nThen run the NVIDIA driver install you downloaded earlier, and reboot\nyour machine afterwards:\n\n```\nsudo bash \u003clocation of ./NVIDIA-Linux-*.run file\u003e\nsudo reboot\n```\n\nFinally, verify that the NVIDIA drivers are installed correctly, and\nyour GPU can be located using `nvidia-smi`:\n\n```\nnvidia-smi\n```\n\n```\nThu Mar  9 03:40:33 2017       \n+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 367.57                 Driver Version: 367.57                    |\n|-------------------------------+----------------------+----------------------+\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n|===============================+======================+======================|\n|   0  GRID K520           Off  | 0000:00:03.0     Off |                  N/A |\n| N/A   54C    P0    45W / 125W |      0MiB /  4036MiB |      0%      Default |\n+-------------------------------+----------------------+----------------------+\n                                                                               \n+-----------------------------------------------------------------------------+\n| Processes:                                                       GPU Memory |\n|  GPU       PID  Type  Process name                               Usage      |\n|=============================================================================|\n|  No running processes found                                                 |\n+-----------------------------------------------------------------------------+\n```\n\nNow that the NVIDIA drivers are installed, `nvidia-docker` can be\ndownloaded and installed as follows (based on\n[these instructions](https://github.com/NVIDIA/nvidia-docker#quick-start)):\n\n```\nwget -P /tmp https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.1/nvidia-docker_1.0.1-1_amd64.deb\nsudo dpkg -i /tmp/nvidia-docker*.deb \u0026\u0026 rm /tmp/nvidia-docker*.deb\n```\n\nAnd you can confirm the installation, by attempting to run `nvida-smi`\ninside of a docker container:\n\n```\nnvidia-docker run --rm nvidia/cuda nvidia-smi\n```\n\n```\nUsing default tag: latest\nlatest: Pulling from nvidia/cuda\nd54efb8db41d: Pull complete \nf8b845f45a87: Pull complete \ne8db7bf7c39f: Pull complete \n9654c40e9079: Pull complete \n6d9ef359eaaa: Pull complete \ncdfa70f89c10: Pull complete \n3208f69d3a8f: Pull complete \neac0f0483475: Pull complete \n4580f9c5bac3: Pull complete \n6ee6617c19de: Pull complete \nDigest: sha256:2b7443eb37da8c403756fb7d183e0611f97f648ed8c3e346fdf9484433ca32b8\nStatus: Downloaded newer image for nvidia/cuda:latest\nThu Mar  9 03:44:23 2017       \n+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 367.57                 Driver Version: 367.57                    |\n|-------------------------------+----------------------+----------------------+\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n|===============================+======================+======================|\n|   0  GRID K520           Off  | 0000:00:03.0     Off |                  N/A |\n| N/A   54C    P8    18W / 125W |      0MiB /  4036MiB |      0%      Default |\n+-------------------------------+----------------------+----------------------+\n                                                                               \n+-----------------------------------------------------------------------------+\n| Processes:                                                       GPU Memory |\n|  GPU       PID  Type  Process name                               Usage      |\n|=============================================================================|\n|  No running processes found                                                 |\n+-----------------------------------------------------------------------------+\n```\n\nOnce you have confirmed `nvidia-smi` works inside of `nvidia-docker`,\nyou should be able to run DeepOSM using your GPU.\n\n### Run Scripts\n\nStart Docker, then run:\n\n```bash\nmake dev-gpu\n```\n\nOr if you don't have a capable GPU, run:\n\n```bash\nmake dev\n```\n\n### Download NAIP, PBF, and Analyze\n\nInside Docker, the following Python scripts will work. This will download all source data, tile it into training/test data and labels, train the neural net, and generate image and text output. \n\nThe default data is six NAIPs, which get tiled into 64x64x4 bands of data (RGB-IR bands). The training labels derive from PBF files that overlap the NAIPs.\n\n```\npython bin/create_training_data.py\npython bin/train_neural_net.py\n```\n\nFor output, DeepOSM will produce some console logs, and then JPEGs of the ways, labels, and predictions overlaid on the tiff.\n\n### Testing\n\nThere is a very limited test suite available at the moment, that can be accessed (from the host system) by running:\n\n`make test`\n\n### Jupyter Notebook\n\nAlternately, development/research can be done via jupyter notebooks:\n\n`make notebook`\n\nTo access the notebook via a browser on your host machine, find the IP VirtualBox is giving your default docker container by running:\n\n```\ndocker-machine ls\n\nNAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS\ndefault   *        virtualbox   Running   tcp://192.168.99.100:2376           v1.10.3\n```\n\nThe notebook server is accessible via port 8888, so in this case you'd go to:\nhttp://192.168.99.100:8888\n\n### Readings\n\n* [Learning to Detect Roads in High-Resolution Aerial\nImages](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.232.1679\u0026rep=rep1\u0026type=pdf) (Hinton) \n* [Machine Learning for Aerial Image Labeling](https://www.cs.toronto.edu/~vmnih/docs/Mnih_Volodymyr_PhD_Thesis.pdf)- Minh's 2013 thesis, student of Hinton's\nbest/recent paper on doing this, great success with these methods\n* Similar Efforts with OSM Data\n    * [OSM-Crosswalk-Detection](https://github.com/geometalab/OSM-Crosswalk-Detection) - uses Keras to detect crosswalks, a class project (Fall 2015)\n    * [OSM-HOT-ConvNet](https://github.com/larsroemheld/OSM-HOT-ConvNet) - attempted use for disaster response, author thinks it's only 69% accurate at pixel level (fall 2016)\n    * [Terrapattern](http://www.terrapattern.com/) - (spring 2016) - detect similar images, such as pools, boat wakes, or other patterns journalists/researchers might be interested in - Carnegie Mellon researchers, open source\n    * [Skynet Data](https://github.com/developmentseed/skynet-data) - (spring 2016) - data pipeline for machine learning with OpenStreetMap\n    * [Full implementation of Mnih's CNN with Massachusetts data](https://github.com/mitmul/ssai-cnn)\n\n* [Parsing Natural Scenes and Natural Language\nwith Recursive Neural Networks (RNNs)](http://ai.stanford.edu/~ang/papers/icml11-ParsingWithRecursiveNeuralNetworks.pdf)\n* Background on Neural Networks and Deep Learning\n    * [MNIST Data and Background](http://yann.lecun.com/exdb/mnist/)\n    * all the other links to Nielsen’s book and [Colah’s blog](http://colah.github.io/posts/2015-08-Backprop/)\n    * [original Information Theory paper by Shannon](http://worrydream.com/refs/Shannon%20-%20A%20Mathematical%20Theory%20of%20Communication.pdf)\n\n[Also see a work journal here](http://trailbehind.github.io/DeepOSM/).\n\n### Papers - Relevant Maybe\n\n* [Aerial image geolocalization from recognition and matching of roads and intersections - May 2016](https://arxiv.org/abs/1605.08323)\n\n* [Uses a large window to improve predictions, trying to capture broad network topology.](https://www.inf.ethz.ch/personal/ladickyl/roads_gcpr14.pdf)\n\n* [Automatically extract roads with no human labels. Not that accurate, could work for preprocessing to detect roads.](https://www.researchgate.net/publication/263892800_Tensor-Cuts_A_simultaneous_multi-type_feature_extractor_and_classifier_and_its_application_to_road_extraction_from_satellite_images)\n\n### Papers - Not All that Relevant\n\n* [Uses map data and shapes of overpasses to then detect pictures of the objects? Seems like a cool paper to read if it was free.](http://dl.acm.org/citation.cfm?id=2424336)\n\n* [New technique for classification of sub-half-meter data into different zones.](http://ieeexplore.ieee.org/xpl/login.jsp?tp=\u0026arnumber=6827949\u0026url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D6827949)\n\n* [Couldn't access text, focused on using lidar data.](http://ieeexplore.ieee.org/xpl/login.jsp?tp=\u0026arnumber=6238909\u0026url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D6238909)\n\n* [Proposes a way to extract network topology, and maybe this can be used as a post processor?](http://www.cv-foundation.org/openaccess/content_cvpr_2013/html/Wegner_A_Higher-Order_CRF_2013_CVPR_paper.html)\n\n### Papers to Review\n\nRecent Recommendations\n\n* FIND - have you seen a paper from a few years ago about estimating OSM completeness by comparing size of compressed satellite images vs number of osm nodes\n\n* READ - this presentation on using GPS traces to suggest OSM edits (Strava/Telenav): http://webcache.googleusercontent.com/search?q=cache:VoiCwRHOyLUJ:stateofthemap.us/map-tracing-for-millennials/+\u0026cd=3\u0026hl=en\u0026ct=clnk\u0026gl=us\n\n#### Citing Mnih and Hinton\n\nI am reviewing these papers from Google Scholar that both cite the key papers and seem relevant to the topic. \n\n* http://ieeexplore.ieee.org/xpl/login.jsp?tp=\u0026arnumber=6602035\u0026url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D6602035\n\n* http://www.cv-foundation.org/openaccess/content_cvpr_workshops_2015/W13/html/Paisitkriangkrai_Effective_Semantic_Pixel_2015_CVPR_paper.html\n\n* http://www.tandfonline.com/doi/abs/10.1080/15481603.2013.802870\n\n* https://www.computer.org/csdl/proceedings/icpr/2014/5209/00/5209d708-abs.html\n\n* http://opticalengineering.spiedigitallibrary.org/article.aspx?articleid=1679147\n\n* http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=1354584\n\n* http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.309.8565\n\n* https://www.itc.nl/library/papers_2012/msc/gem/shaoqing.pdf\n\n* http://ieeexplore.ieee.org/xpl/login.jsp?tp=\u0026arnumber=7326745\u0026url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7326745\n\n* http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=2191094\n\n* http://arxiv.org/abs/1509.03602\n\n* http://ieeexplore.ieee.org/xpl/login.jsp?tp=\u0026arnumber=7112625\u0026url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7112625\n\n* http://www.sciencedirect.com/science/article/pii/S0924271615001690\n\n* http://arxiv.org/abs/1405.6137\n\n* https://www.itc.nl/external/ISPRS_WGIII4/ISPRSIII_4_Test_results/papers/Onera_2D_label_Vaih.pdf\n\n* http://link.springer.com/chapter/10.1007/978-3-319-23528-8_33#page-1\n\n* http://arxiv.org/abs/1508.06163\n\n* http://www.mdpi.com/2072-4292/8/4/329\n\n* http://arxiv.org/abs/1510.00098\n\n* http://link.springer.com/article/10.1007/s10489-016-0762-6\n\n* http://www.tandfonline.com/doi/abs/10.1080/01431161.2015.1054049\n\n* http://ieeexplore.ieee.org/xpl/login.jsp?tp=\u0026arnumber=7393563\u0026url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7393563\n\n* http://www.cv-foundation.org/openaccess/content_iccv_2015/html/Mattyus_Enhancing_Road_Maps_ICCV_2015_paper.html\n\n* http://www.cv-foundation.org/openaccess/content_iccv_2015/html/Zheng_Minimal_Solvers_for_ICCV_2015_paper.html\n\n* http://arxiv.org/abs/1405.6136\n\n* http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.681.1695\u0026rep=rep1\u0026type=pdf\n\n* http://ieeexplore.ieee.org/xpl/login.jsp?tp=\u0026arnumber=7120492\u0026url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7120492\n\n* http://www.tandfonline.com/doi/abs/10.3846/20296991.2014.890271\n\n* http://ieeexplore.ieee.org/xpl/login.jsp?tp=\u0026arnumber=7362660\u0026url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7362660\n\n* http://ieeexplore.ieee.org/xpl/login.jsp?tp=\u0026arnumber=7414402\u0026url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7414402\n\n* http://ieeexplore.ieee.org/xpl/login.jsp?tp=\u0026arnumber=6663455\u0026url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D6663455\n\n* http://ieeexplore.ieee.org/xpl/login.jsp?tp=\u0026arnumber=7337372\u0026url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7337372\n\n* https://www.researchgate.net/profile/Moslem_Ouled_Sghaier/publication/280655680_Road_Extraction_From_Very_High_Resolution_Remote_Sensing_Optical_Images_Based_on_Texture_Analysis_and_Beamlet_Transform/links/55c0d9da08ae092e96678ff3.pdf\n\n* http://ieeexplore.ieee.org/xpl/login.jsp?tp=\u0026arnumber=7159022\u0026url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7159022\n\n* http://www.sciencedirect.com/science/article/pii/S0303243415300283\n\n* http://dl.acm.org/citation.cfm?id=2666389\n\n* http://www.ijicic.org/ijicic-15-04045.pdf\n\n### Original Idea\n\nThis was the general idea to start, and working with TMS tiles sort of worked (see first 50 or so commits), so DeepOSM got switched to better data:\n\n![Deep OSM Project](https://gaiagps.mybalsamiq.com/mockups/4278030.png?key=1e42f249214928d1fa7b17cf866401de0c2af867)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrailbehind%2FDeepOSM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrailbehind%2FDeepOSM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrailbehind%2FDeepOSM/lists"}