{"id":18174888,"url":"https://github.com/kylebarron/landsat-mosaic-latest","last_synced_at":"2025-09-06T16:32:38.845Z","repository":{"id":93930182,"uuid":"256913378","full_name":"kylebarron/landsat-mosaic-latest","owner":"kylebarron","description":"Auto-updating global Landsat 8 mosaic of Cloud-Optimized GeoTIFFs from SNS notifications","archived":false,"fork":false,"pushed_at":"2020-05-25T20:38:21.000Z","size":6515,"stargazers_count":15,"open_issues_count":1,"forks_count":2,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-12-23T07:28:03.186Z","etag":null,"topics":["aerial-imagery","aws-lambda","cloud-optimized-geotiff","cogeo-mosaic","lambda","landsat","landsat-8","landsat-data","python","satellite-imagery","serverless"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kylebarron.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-04-19T04:25:37.000Z","updated_at":"2022-10-17T11:23:40.000Z","dependencies_parsed_at":"2023-08-28T03:03:14.948Z","dependency_job_id":null,"html_url":"https://github.com/kylebarron/landsat-mosaic-latest","commit_stats":{"total_commits":47,"total_committers":2,"mean_commits":23.5,"dds":"0.34042553191489366","last_synced_commit":"8623408c7416c35f8fbd434a5818124fa2842ca0"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylebarron%2Flandsat-mosaic-latest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylebarron%2Flandsat-mosaic-latest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylebarron%2Flandsat-mosaic-latest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylebarron%2Flandsat-mosaic-latest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kylebarron","download_url":"https://codeload.github.com/kylebarron/landsat-mosaic-latest/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232133704,"owners_count":18477293,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aerial-imagery","aws-lambda","cloud-optimized-geotiff","cogeo-mosaic","lambda","landsat","landsat-8","landsat-data","python","satellite-imagery","serverless"],"created_at":"2024-11-02T16:08:04.175Z","updated_at":"2025-01-01T22:34:16.182Z","avatar_url":"https://github.com/kylebarron.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# landsat-mosaic-latest\n\nAuto-updating Landsat 8 mosaics from AWS SNS notifications.\n\n## Overview\n\nAWS stores an [open, freely-accessible data\nset](https://registry.opendata.aws/landsat-8/) of [Landsat\n8](https://www.usgs.gov/land-resources/nli/landsat/landsat-8?qt-science_support_page_related_con=0#)\nimagery. Crucially, this data is stored in [_Cloud-Optimized\nGeoTIFF_](https://www.cogeo.org/) (COG), an extension of the GeoTIFF standard\nwhich specifies a smart internal layout for image overviews. By reading the\nimage's header, an application can understand the byte ranges of different parts\nof the image, and can read them using HTTP range requests without needing to\ndownload the entire file.\n\nThis file type allows for new cloud-native processing models. For example, you\ncan serve a basemap of Landsat 8 imagery using serverless AWS Lambda functions.\nThis is a huge advance in technology because it allows for **serving satellite\nimagery without needing to pregenerate and store any imagery**. This enables\nhuge cost savings, especially for hobby projects, where just storing hundreds of\nGB or TB of data would be cost-prohibitive.\n\nThere exists an AWS Simple Notification Service (SNS)\n[Topic](https://registry.opendata.aws/landsat-8/) that creates notifications\nwhen new Landsat data are added to the AWS open data set. This library defines\nan AWS Lambda function to run when those notifications are sent, and update two\nDynamoDB databases with identifiers for the most recent imagery per mercator\ntile.\n\nThis library does not provide on-the-fly image tiling. For that, look at\n[`awspds-mosaic`](https://github.com/kylebarron/awspds-mosaic).\n\nAlso note that this library creates a new DynamoDB table but does not populate\nit with initial values: it only updates the table as new imagery comes in. See\ninstructions below to create an initial Landsat mosaic.\n\n## Install\n\n```\ngit clone https://github.com/kylebarron/landsat-mosaic-latest\ncd landsat-mosaic-latest\npip install .\n```\n\n### Create quadkey index file (optional)\n\n**Unless you have specific requirements, you can skip this section.**\n\nLandsat images are produced in a [grid of _paths_ and\n_rows_](https://landsat.gsfc.nasa.gov/wp-content/uploads/2013/01/wrs2.gif). In\norder to keep things simple, eliminate geospatial dependencies, and create\nefficient an MosaicJSON, this package relies on a prebuilt index that associates\nthose path-row combinations to the mercator tile quadkeys used by the tiler.\n\nBy default, this library ships with a worldwide index at quadkey zoom level 8.\nIf you need a differing quadkey zoom or want to restrict your mosaic to a\ngeographic bounding box, you can build your own index.\n\nThe script to create an index is stored in\n[`landsat-cogeo-mosaic`][landsat-cogeo-mosaic]. For full instructions, see its\n[project docs][index_cli].\n\n[index_cli]: https://kylebarron.dev/landsat-cogeo-mosaic/cli/#index\n\nThe standard index bundled by default with `landsat-mosaic-latest` is created\nwith:\n\n```bash\nlandsat-cogeo-mosaic index \\\n    `# Path to Shapefile of path-row geometries` \\\n    --wrs-path data/WRS2_descending_0/WRS2_descending.shp \\\n    `# Path to CSV of scene metadata downloaded from AWS S3` \\\n    --scene-path data/scene_list.gz \\\n    `# Worldwide bounds` \\\n    --bounds '-180,-90,180,90' \\\n    `# Quadkey zoom` \\\n    --quadkey-zoom 8 \\\n    | gzip \\\n    \u003e landsat_mosaic_latest/data/index.json.gz\n```\n\nNote it's currently imperative to write the index to that exact location in\norder to be properly found during runtime.\n\n## Build\n\nIf you wish to change the quadkey index file, as described above, make sure you\ndo that before building, as that file will be included in the lambda bundle.\n\nThen building is simple: (requires Docker and Make)\n\n```bash\nmake package\n```\n\nThis creates a `package.zip` file in the current directory with this package's\ncode and any required dependencies. This will be uploaded to AWS in the next\nstep.\n\n## Deploy\n\nTo simplify deployment, this package uses the Serverless framework. [Refer to\ntheir docs](https://serverless.com/framework/docs/getting-started/) to install\nthe `sls` command line library and authorize it with your AWS credentials.\n\nBy default deployment creates _two_ DynamoDB tables, one for the absolute latest\nimagery, another for the latest _low-cloud_ imagery.\n\nThen it's simple to deploy this stack with a single line:\n\n```bash\nsls deploy \\\n    --table-name landsat-mosaic-latest \\\n    --cloudless-table-name landsat-mosaic-latest-cloudless \\\n    --max-cloud-cover 5\n```\n\n- `table-name` is the name given to the DynamoDB table without a cloud cover filter. You'll need to provide this information to the tiler when serving imagery. Default: `landsat-mosaic-latest-cloudless`.\n- `cloudless-table-name` is the name given to the DynamoDB table that uses the cloud cover filter below. Default `landsat-mosaic-latest`.\n- `max-cloud-cover` is an integer between 0 and 100 that defines the maximum percent cloud cover permitted for new imagery into the cloudless DynamoDB table. If a new Landsat scene has cloud cover greater than the given percent, it will only be added to the non-cloudless DynamoDB table. Default `5`.\n\n## Upload a base MosaicJSON\n\nThis library creates a new DynamoDB table but does not populate it with initial\nvalues: it only updates the table as new imagery comes in. To create an inital\nLandsat mosaic, we'll use [`landsat-cogeo-mosaic`][landsat-cogeo-mosaic].\n\nNote that when creating an initial MosaicJSON, you should use the same path-row\nindex as in the serverless function. The below commands point to the default,\nbundled `index.json.gz`.\n\n[landsat-cogeo-mosaic]: https://github.com/kylebarron/landsat-cogeo-mosaic\n\n### Setup\n\nInstall `cogeo-mosaic` and `landsat-cogeo-mosaic`:\n\n```\npip install \"cogeo-mosaic\u003e=3.0a3\" landsat-cogeo-mosaic\n```\n\n#### Create SQLite database of Landsat 8 metadata\n\nFor up-to-date instructions, see [`landsat-cogeo-mosaic` docs][landsat-cogeo-mosaic-docs]. But it's roughly:\n\n[landsat-cogeo-mosaic-docs]: https://kylebarron.dev/landsat-cogeo-mosaic/examples/global/\n\n```\ngit clone https://github.com/kylebarron/landsat-cogeo-mosaic/\ncd landsat-cogeo-mosaic\nmkdir -p data/\naws s3 cp s3://landsat-pds/c1/L8/scene_list.gz data/\ngunzip -c data/scene_list.gz \u003e data/scene_list\ncd data/\nsqlite3 scene_list.db \u003c ../scripts/csv_import.sql\ncd -\n```\n\nThen `scene_list.db` is the database to be used with the `--sqlite-path`\nargument below.\n\n### Latest Cloudless\n\nSome Landsat 8 path-row combinations have _never_ had a scene with cloud cover\n\u003c5%. This command will automatically relax the `--max-cloud` restriction until\nit finds a result for each path-row.\n\nAssuming you have cloned and are in the `landsat-mosaic-latest` repository:\n\n```bash\nlandsat-cogeo-mosaic create-from-db \\\n    `# Path to the sqlite database file` \\\n    --sqlite-path ../landsat-cogeo-mosaic/data/scene_list.db \\\n    `# Path to the path-row geometry file` \\\n    --pathrow-index landsat_mosaic_latest/data/index.json.gz \\\n    `# Min zoom of mosaic, 7 is a good default for Landsat` \\\n    --min-zoom 7 \\\n    `# Max zoom of mosaic, 12 is a good default for Landsat` \\\n    --max-zoom 12 \\\n    `# Maximum cloud cover. This means 5%` \\\n    --max-cloud 5 \\\n    `# Preference for choosing the asset for a tile` \\\n    --sort-preference newest \\\n    \u003e mosaic_cloudless_latest.json\n```\n\n### Latest\n\nThis is almost the same as the latest cloudless command, except that it removes\nthe `--max-cloud` argument.\n\nAssuming you have cloned and are in the `landsat-mosaic-latest` repository:\n\n```bash\nlandsat-cogeo-mosaic create-from-db \\\n    `# Path to the sqlite database file` \\\n    --sqlite-path ../landsat-cogeo-mosaic/data/scene_list.db \\\n    `# Path to the path-row geometry file` \\\n    --pathrow-index landsat_mosaic_latest/data/index.json.gz \\\n    `# Min zoom of mosaic, 7 is a good default for Landsat` \\\n    --min-zoom 7 \\\n    `# Max zoom of mosaic, 12 is a good default for Landsat` \\\n    --max-zoom 12 \\\n    `# Preference for choosing the asset for a tile` \\\n    --sort-preference newest \\\n    \u003e mosaic_latest.json\n```\n\n### Upload to DynamoDB\n\nThen upload these two generated MosaicJSON files to DynamoDB. The `--url`\nargument must match the names given to the DynamoDB tables in the `sls deploy`\nstep.\n\n**Note: This will overwrite any existing data in the DynamoDB table**.\n\n```bash\ncogeo-mosaic upload \\\n    --url 'dynamodb://us-west-2/landsat-mosaic-latest' \\\n    mosaic_latest.json\ncogeo-mosaic upload \\\n    --url 'dynamodb://us-west-2/landsat-mosaic-latest-cloudless' \\\n    mosaic_cloudless_latest.json\n```\n\n## Pricing\n\n**\\$2.64 per year** is a rough estimate of the cost to keep each DynamoDB table updated.\n\nNote that actually serving imagery using a\n[tiler](https://github.com/developmentseed/awspds-mosaic) is not included in\nthis estimate.\n\n### Lambda\n\n**Time**:\n\n- \\$ per 100ms: 0.0000016667 (when set to 1024mb memory. From a simple test, it looks like setting to lower memory doesn't reduce cost because it takes proportionally longer.)\n- Rough # of 100ms when the scene is not cloudy: 10\n- Percentage of time when scene is below max cloud cover: 0.3\n- Scenes per day: [~750](https://www.usgs.gov/faqs/what-are-acquisition-schedules-landsat-satellites?qt-news_science_products=0#qt-news_science_products)\n- Days per year: 365\n\nRoughly \\$1.36/year for the time cost with these estimates.\n\n**Requests**:\n\n- \\$ 0.20 per 1M requests\n- Scenes per day: 750\n- Days per year: 365\n\nRoughly \\$0.05/year.\n\n### DynamoDB\n\n**Reads**:\n\n- Scenes per day: 750\n- Percentage of time when scene is below max cloud cover: 0.3\n- Quadkeys per scene: ~10\n- 1 read per quadkey\n- Days per year: 365\n- \\$0.25 per million reads\n\nRoughly \\$0.21/year.\n\n**Writes**:\n\n- Scenes per day: 750\n- Percentage of time when scene is below max cloud cover: 0.3\n- Quadkeys per scene: ~10\n- 1 write per quadkey\n- Days per year: 365\n- \\$1.25 per million reads\n\nRoughly \\$1.02/year.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkylebarron%2Flandsat-mosaic-latest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkylebarron%2Flandsat-mosaic-latest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkylebarron%2Flandsat-mosaic-latest/lists"}