{"id":18174830,"url":"https://github.com/kylebarron/usgs-topo-tiler","last_synced_at":"2025-09-06T15:34:05.064Z","repository":{"id":62586788,"uuid":"261370671","full_name":"kylebarron/usgs-topo-tiler","owner":"kylebarron","description":"Python package to read Web Mercator map tiles from USGS Historical Topographic Maps","archived":false,"fork":false,"pushed_at":"2020-05-12T05:46:21.000Z","size":11313,"stargazers_count":78,"open_issues_count":4,"forks_count":3,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-08-31T00:36:43.715Z","etag":null,"topics":["cloud-optimized-geotiff","cogeo-mosaic","rasterio","topographic-maps","usgs","usgs-data"],"latest_commit_sha":null,"homepage":"https://kylebarron.dev/usgs-topo-mosaic","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kylebarron.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-05T05:37:13.000Z","updated_at":"2025-04-16T17:51:39.000Z","dependencies_parsed_at":"2022-11-03T20:22:29.759Z","dependency_job_id":null,"html_url":"https://github.com/kylebarron/usgs-topo-tiler","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/kylebarron/usgs-topo-tiler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylebarron%2Fusgs-topo-tiler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylebarron%2Fusgs-topo-tiler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylebarron%2Fusgs-topo-tiler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylebarron%2Fusgs-topo-tiler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kylebarron","download_url":"https://codeload.github.com/kylebarron/usgs-topo-tiler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylebarron%2Fusgs-topo-tiler/sbom","scorecard":{"id":575185,"data":{"date":"2025-08-11","repo":{"name":"github.com/kylebarron/usgs-topo-tiler","commit":"ee6bce4ce8ec62bd70eaffe90e10db3c35b0fd74"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-20T17:29:08.799Z","repository_id":62586788,"created_at":"2025-08-20T17:29:08.799Z","updated_at":"2025-08-20T17:29:08.799Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273925779,"owners_count":25192312,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-06T02:00:13.247Z","response_time":2576,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-optimized-geotiff","cogeo-mosaic","rasterio","topographic-maps","usgs","usgs-data"],"created_at":"2024-11-02T16:07:54.687Z","updated_at":"2025-09-06T15:34:04.443Z","avatar_url":"https://github.com/kylebarron.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# usgs-topo-tiler\n\nPython package to read Web Mercator map tiles from USGS Historical Topographic\nMaps, and utilities to create a [MosaicJSON][mosaicjson] collection from these\nmaps.\n\n[mosaicjson]: https://github.com/developmentseed/mosaicjson-spec/\n\n![Grand Canyon Historical Mosaic][grca_mosaic]\n\n[grca_mosaic]: https://raw.githubusercontent.com/kylebarron/usgs-topo-tiler/master/assets/grca_mosaic.jpg\n\nA mosaic of USGS historical maps around the Grand Canyon, with added relief\nshading.\n\n## Overview\n\nI stumbled upon a hidden gem: the entire USGS historical topographic map\narchive, consisting of 183,112 digitized maps created between 1884 and 2006, is\nstored in Cloud-Optimized GeoTIFF (COG) format on AWS S3.\n\nThe fact that maps are accessible publicly and stored in COG format means that\nyou can easily and cheaply set up a serverless function on AWS Lambda to serve\nmap tiles on the fly.\n\nThe [COG format][cogeo] is a backwards-compatible, cloud-native storage format\nfor raster files that allow selected portions of the file to be read over the\nnetwork without needing to download and parse the entire file. This fast random\nread access allows for dynamic tiling of map tiles on demand, without needing to\npreprocess and store any map data.\n\n[cogeo]: https://www.cogeo.org/\n\nThere are three parts to serving your own tiles:\n\n- `usgs-topo-tiler`: a library to extract a _single_ Web Mercator tile from _one_ source historical map.\n- `usgs-topo-tiler`'s CLI, which helps to construct MosaicJSON files. These files tell `usgs-topo-mosaic` what source files should be combined to create a single Web Mercator tile.\n- [`usgs-topo-mosaic`][usgs-topo-mosaic]: a library to use a MosaicJSON file created above to create a seamless mosaic of tiles. This is designed to be deployed with AWS Lambda and AWS API Gateway as a serverless tile endpoint.\n\n[usgs-topo-mosaic]: https://github.com/kylebarron/usgs-topo-mosaic\n\n## Generate a Web Mercator tile\n\n### Install\n\n```bash\npip install usgs-topo-tiler 'rio-tiler\u003e=2.0a6'\n```\n\n### Usage\n\nHere I'll show a quick overview of reading a single mercator tile from a single\nUSGS historical map. If you're looking for a specific map, the simplest way is\nprobably to use the [National Map Viewer][nationalmap]. Check the box for\n\"Historical Topographic Maps\", make sure the file format is \"GeoTIFF\". Click\n\"Find Products\", and then right click \"Download\" to get the HTTPS url to the\nGeoTIFF on S3.\n\n[nationalmap]: https://viewer.nationalmap.gov/basic/\n\nFor this demo, I'll make a mercator tile from an 1897 topographic map of\nYosemite Valley.\n\n**Read a tile from a file over the internet**\n\n```py\nfrom usgs_topo_tiler import tile\n\nurl = 'https://prd-tnm.s3.amazonaws.com/StagedProducts/Maps/HistoricalTopo/GeoTIFF/CA/CA_Yosemite_299696_1897_125000_geo.tif'\n# Mercator tile\nx, y, z = 687, 1583, 12\n\ntile, mask = tile(url, x, y, z, tilesize=512)\nprint(tile.shape)\n# (3, 512, 512)\nprint(mask.shape)\n# (512, 512)\n```\n\n**Create image from tile**\n\nNote that if you're using `rio-tiler` v1, you should replace `render` with\n`array_to_image`.\n\n```py\nfrom rio_tiler.utils import render\n\nbuffer = render(tile, mask, img_format='png')\n```\n\n**Write image to file**\n\n```py\nwith open('yose_1897.png', 'wb') as f:\n    f.write(buffer)\n```\n\nYou now have a 512x512 png image aligned with the Web Mercator grid, and because\nthe source is a _Cloud-optimized_ GeoTIFF, the image was made with a minimal\nnumber of requests to the source, and without reading the entire GeoTIF.\n\n![Yosemite, 1897 Web Mercator tile][yose_1897]\n\n[yose_1897]: https://github.com/kylebarron/usgs-topo-tiler/blob/master/assets/yose_1897.png?raw=true\n\n## Create a MosaicJSON\n\nThe process described above is for create _one_ tile. But often we want to join\nmany mercator tiles to make a seamless web map. This is where\n[MosaicJSON][mosaicjson] comes in. It describes how to join sources and where to\nplace them.\n\nThis section outlines how to create a MosaicJSON from USGS historical topo\nassets. This MosaicJSON can then be used with `usgs-topo-mosaic` to serve a web\nmap of tiles with AWS Lambda.\n\n### Install\n\nWhen you install `usgs-topo-tiler`, it doesn't include dependencies necessary\nfor the CLI commands so as to keep the deployment size small when used with\n[`usgs-topo-mosaic`][usgs-topo-mosaic].\n\nTo install the additional CLI dependencies, run:\n\n```bash\npip install 'usgs-topo-tiler[cli]'\n```\n\n### Download bulk metadata\n\nFirst you need to download metadata including at least the geospatial bounds of\neach map, and the URL of each map. It's possible to use the [National Map\nAPI][nationalmap_api] to retrieve such metadata, however I prefer to download a\nCSV of bulk metadata from S3. This file includes metadata on every image in\ntheir collection.\n\n[nationalmap_api]: https://viewer.nationalmap.gov/tnmaccess/api/index\n\n```bash\nmkdir -p data\nwget https://prd-tnm.s3-us-west-2.amazonaws.com/StagedProducts/Maps/Metadata/topomaps_all.zip -P data/\nunzip topomaps_all.zip\n```\n\n`data/topomaps_all.csv` is the extracted bulk metadata file. `data/readme.txt`\nhas helpful information about what fields are in the bulk metadata file.\n\n### Download list of COG files:\n\nOccasionally there are some files listed in the metadata that don't exist as\nGeoTIFF. I download a list of the `.tif` files on S3 so that I can\ncross-reference against the metadata and ensure that only files that exist are\nincluded in the MosaicJSON.\n\nThis step is optional, but recommended.\n\n```bash\nmkdir -p data/\nusgs-topo-tiler list-s3 \\\n    --bucket 'prd-tnm' \\\n    --prefix 'StagedProducts/Maps/HistoricalTopo/GeoTIFF/' \\\n    --ext '.tif' \\\n    \u003e data/geotiff_files.txt\n\n\u003e wc -l data/geotiff_files.txt\n    183112 data/geotiff_files.txt\n```\n\n_183112_ COG files!\n\n### Create MosaicJSON\n\nNow you're ready to start creating mosaics. This isn't entirely straightforward\nbecause maps were created at different times and at different scales in\ndifferent regions of the U.S. So it's not usually as simple as creating a mosaic\nof all maps of a single scale, unless you're ok with having gaps in the mosaic\nspatially, where tiles might return empty data.\n\n#### Footprints\n\nOne of the best way to visually see the results of a filtering query on the\nmetadata is to generate _footprints_ and then display them on a map.\n\nLets say we want to create a mosaic of mid-scale maps. We can use the\n`--filter-only` cli flag to export newline-delimited GeoJSON of the query.\n\n```bash\nusgs-topo-tiler mosaic-bulk \\\n    --meta-path data/topomaps_all.csv \\\n    --s3-list-path data/geotiff_files.txt \\\n    --min-scale 63360 \\\n    --max-scale 249000 \\\n    --filter-only \\\n    \u003e mid_scale_footprints.geojsonl\n```\n\nThen we can visualize this data, e.g. with [my CLI][keplergl_cli] for\n[kepler.gl][keplergl].\n\n[keplergl]: https://kepler.gl/\n[keplergl_cli]: https://github.com/kylebarron/keplergl_cli\n\n```bash\nkepler mid_scale_footprints.geojsonl\n```\n\nThis illustrates the core problem of these historical maps when making a\nMosaicJSON. Some areas have been mapped more than others, and some have never\nbeen mapped at this scale range. If you were to create a MosaicJSON from these\nparameters, you'd get empty images when requesting data over Northern Montana\nand Western Texas.\n\n![][mid_scale_footprints]\n\n[mid_scale_footprints]: https://github.com/kylebarron/usgs-topo-tiler/blob/master/assets/mid_scale_footprints.png?raw=true\n\n#### Generate MosaicJSON\n\nOnce you know the desired parameters of your query, remove the `--filter-only`\nflag to generate the MosaicJSON. For example:\n\n```bash\nusgs-topo-tiler mosaic-bulk \\\n    --meta-path data/topomaps_all.csv \\\n    --s3-list-path data/geotiff_files.txt \\\n    --min-scale 63360 \\\n    --max-scale 249000 \\\n    \u003e mid_scale_mosaic.json\n```\n\n`mid_scale_mosaic.json` is now a MosaicJSON file that can be used with\n`usgs-topo-mosaic` to render a web map. Note, however that this uses a custom\nasset string format, as described in [Removing Map\nCollars](#removing-map-collars), and won't necessarily work with all other\nMosaicJSON tools.\n\n#### Examples\n\n**Low zoom, newest available**\n\n```bash\nusgs-topo-tiler mosaic-bulk \\\n    --meta-path data/topomaps_all.csv \\\n    --s3-list-path data/geotiff_files.txt \\\n    --min-scale 250000 \\\n    \u003e mosaic_low.json\n```\n\n**Medium zoom, newest available, filling in with lower-resolution maps where necessary**\n\n```bash\nusgs-topo-tiler mosaic-bulk \\\n    --meta-path data/topomaps_all.csv \\\n    --s3-list-path data/geotiff_files.txt \\\n    --min-scale 63360 \\\n    \u003e mosaic_medium.json\n```\n\n**Medium zoom, oldest available, filling in with lower-resolution maps where necessary**\n\n```bash\nusgs-topo-tiler mosaic-bulk \\\n    --meta-path data/topomaps_all.csv \\\n    --s3-list-path data/geotiff_files.txt \\\n    --min-scale 63360 \\\n    --sort-preference oldest \\\n    \u003e mosaic_medium_oldest.json\n```\n\n**High zoom, newest available, continental U.S. only**\n\n```bash\nusgs-topo-tiler mosaic-bulk \\\n    --meta-path data/topomaps_all.csv \\\n    --s3-list-path data/geotiff_files.txt \\\n    --min-scale 24000 \\\n    --max-scale 63359 \\\n    `# Lower 48 states only` \\\n    --bounds '-161.96,12.85,-55.01,50.53' \\\n    \u003e mosaic_high.json\n```\n\n#### API\n\n```\nUsage: usgs-topo-tiler mosaic-bulk [OPTIONS]\n\n  Create MosaicJSON from CSV of bulk metadata\n\nOptions:\n  --meta-path PATH                Path to csv file of USGS bulk metadata dump\n                                  from S3  [required]\n\n  --s3-list-path PATH             Path to txt file of list of s3 GeoTIFF files\n  --min-scale FLOAT               Minimum map scale, inclusive\n  --max-scale FLOAT               Maximum map scale, inclusive\n  --min-year FLOAT                Minimum map year, inclusive\n  --max-year FLOAT                Maximum map year, inclusive\n  --woodland-tint / --no-woodland-tint\n                                  Filter on woodland tint or no woodland tint.\n                                  By default no filtering is applied.\n\n  --allow-orthophoto              Allow orthophoto  [default: False]\n  --bounds TEXT                   Bounding box for mosaic. Must be of format\n                                  \"minx,miny,maxx,maxy\"\n\n  -z, --minzoom INTEGER           Force mosaic minzoom\n  -Z, --maxzoom INTEGER           Force mosaic maxzoom\n  --quadkey-zoom INTEGER          Force mosaic quadkey zoom\n  --sort-preference [newest|oldest|closest-to-year]\n                                  Method for choosing assets within a given\n                                  mercator tile at the quadkey zoom.\n                                  [default: newest]\n\n  --closest-to-year INTEGER       Year used for comparisons when preference is\n                                  closest-to-year.\n\n  --filter-only                   Output filtered GeoJSON features, without\n                                  creating the MosaicJSON. Useful for\n                                  inspecting the footprints   [default: False]\n\n  --help                          Show this message and exit.\n```\n\n## Addendum\n\n### Removing map collars\n\nAll of the USGS historical maps have _collars_, regions of space around the map\nwhere metadata is printed. In order to create continuous map tiles from a\ncollection of these maps, these collars have to be clipped, so that only the map\nis showing.\n\n![Ruby, AK][ruby_ak]\n\n[ruby_ak]: https://github.com/kylebarron/usgs-topo-tiler/blob/master/assets/ruby_ak.jpg?raw=true\n\nThese maps are georeferenced, which means that it's straightforward to remove\nthe collar when you know the actual bounds contained in the map. However, I've\nfound that there's no reliable way to determine the bounds on the fly with just\nthe image and its filename.\n\nWhile building the mosaic ahead of time, you have access to this information,\nbut with the usual tiling setup, you'd only have access to the URL and image\nwhile tiling.\n\nTo get around this, I apply a \"hack\" to the MosaicJSON format. Instead of just\nencoding a URL string, I encode the url _and_ the bounds of the map as a JSON\nstring.\n\n**Summary**: when you build a mosaic using the cli in this library, it encodes a\nnon-standard MosaicJSON that works well with the `usgs-mosaic-tiler` tiler, but\nisn't necessarily readable by other MosaicJSON tools\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkylebarron%2Fusgs-topo-tiler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkylebarron%2Fusgs-topo-tiler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkylebarron%2Fusgs-topo-tiler/lists"}