{"id":19558702,"url":"https://github.com/cholmes/duckdb-geoparquet-tutorials","last_synced_at":"2026-07-29T23:30:19.110Z","repository":{"id":180811012,"uuid":"665742825","full_name":"cholmes/duckdb-geoparquet-tutorials","owner":"cholmes","description":null,"archived":false,"fork":false,"pushed_at":"2023-10-18T20:30:08.000Z","size":45,"stargazers_count":115,"open_issues_count":2,"forks_count":8,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-08T22:07:40.031Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cholmes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-12T22:44:25.000Z","updated_at":"2024-12-28T14:05:59.000Z","dependencies_parsed_at":"2024-11-11T04:47:58.561Z","dependency_job_id":"d5a2611b-0a4a-429e-a967-e6d93e59bf8e","html_url":"https://github.com/cholmes/duckdb-geoparquet-tutorials","commit_stats":null,"previous_names":["cholmes/duckdb-geoparquet-tutorials"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cholmes%2Fduckdb-geoparquet-tutorials","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cholmes%2Fduckdb-geoparquet-tutorials/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cholmes%2Fduckdb-geoparquet-tutorials/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cholmes%2Fduckdb-geoparquet-tutorials/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cholmes","download_url":"https://codeload.github.com/cholmes/duckdb-geoparquet-tutorials/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240815209,"owners_count":19862019,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T04:47:53.312Z","updated_at":"2026-07-29T23:30:19.026Z","avatar_url":"https://github.com/cholmes.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":" # DuckDB and GeoParquet Tutorial\n\n This is a quick tutorial on how you can use DuckDB to easily access the [cloud-native version](https://beta.source.coop/cholmes/google-open-buildings) of the [Google Open Buildings](https://sites.research.google/open-buildings/) data set from [source.coop](https://beta.source.coop/) and transform it into your favorite GIS\n format. A big thanks to [Mark Litwintschik's post on DuckDB's Spatial Extension](https://tech.marksblogg.com/duckdb-gis-spatial-extension.html) for lots of the key information, it's highly recommended. \n\n## About DuckDB?\n\n[DuckDB](https://duckdb.org/) is an awesome new tool for working with data. In some ways it's a next generation 'SQLite' (which is behind GeoPackage in the geo world) - but fundamentally designed for analysis workflows. TODO: more explanation. \n\nTo install it just follow the instructions at: https://duckdb.org/docs/installation/index. This tutorial uses the command line version \n\nThere are a couple of awesome extensions that make it very easy to work with parquet files on the cloud. [httpfs](https://duckdb.org/docs/extensions/httpfs.html) enables you to pull in S3\nfiles directly within DuckDB, and [spatial](https://duckdb.org/docs/extensions/spatial) gives\nyou geometries with a number of operations, and lets you write out to over 50 different formats.\n\n## Setting up DuckDB\n\nOnce you've installed it then getting started is easy. Just type `duckdb` from the command-line. If you want to persist the tables you create you can supply a name, like `duckdb buildings.db`, but it's not necessary for this tutorial. After you're in the DuckDB interface you'll need to install and load the two extensions (you just need to install once, so can skip that in the future):\n\n```\nINSTALL spatial;\nLOAD spatial;\nINSTALL httpfs;\nLOAD httpfs;\n```\n\nThe DuckDB docs say that you should set your S3 region, but it doesn't seem to be necessary. For \nthis dataset it'd be:\n\n```\nSET s3_region='us-west-2';\n```\n\n## Full country into GeoParquet\n\nWe'll start with how to get a country-wide GeoParquet file from the [geoparquet-admin1](https://beta.source.coop/cholmes/google-open-buildings/browse/geoparquet-admin1) directory, which partitions the dataset into directories for each country and files for each admin level 1 region. \n\nFor this we don't actually even need the spatial extension - we'll just use DuckDB's great S3 selection interface to easily export to Parquet and then use another tool to turn it into official GeoParquet. \n\nThe following call selects all parquet files from the [`country=SSD`](https://beta.source.coop/cholmes/google-open-buildings/browse/geoparquet-admin1/country=SSD) (South Sudan) directory.\n\n```\nCOPY (SELECT * FROM \n    's3://us-west-2.opendata.source.coop/google-research-open-buildings/geoparquet-admin1/country=SSD/*.parquet') \n    TO 'south_sudan.parquet' (FORMAT PARQUET);\n```\n\nThese stream directly out to a local parquet file. Below we'll explore how to save it into DuckDB and work with it. The output will not be Geoparquet, but a Parquet file with Well Known Binary. Hopefully DuckDB will support native GeoParquet output so we won't need the conversion at the end. As long as you name the geometry column 'geometry' then you can use the [gpq](https://github.com/planetlabs/gpq) `convert` function to turn it from a parquet file with WKB to a valid GeoParquet file.\n\nTo set up gpq just use the [installation docs](https://github.com/planetlabs/gpq#installation). And when it's set up you can just say:\n\n```\ngpq convert south_sudan.parquet south_sudan-geo.parquet\n```\n\n\nThe `south_sudan-geo.parquet` file should be valid GeoParquet. You can then change it into any other format using GDAL's [`ogr2ogr`](https://gdal.org/programs/ogr2ogr.html):\n\n```\nogr2ogr laos.fgb laos-geo.parquet\n```\n\nNote that the above file is only about 3 megabytes. Larger countries can be hundreds of megabytes or gigabytes, so be sure you have a fast connections or patience. DuckDB should give some updates on progress as it works, but it doesn't seem to be super accurate with remote files.\n\n## Using DuckDB Spatial with GeoParquet\n\nNow we'll get into working with DuckDB a bit more, mostly to transform it into different output formats to start. We'll start small, which should work with most connections. But once we get to the bigger requests they may take awhile if you have a slower connection. DuckDB can still be useful, but you'll probably want to save the files as tables and persist locally.\n\n### Working with one file\n\nWe'll start with just working with a single parquet file.\n\n#### Count one file\n\nYou can get the count of a file, just put in the S3 URL to the parquet file.\n\n```\nSELECT count(*) FROM \n   's3://us-west-2.opendata.source.coop/google-research-open-buildings/geoparquet-admin1/country=LAO/Attapeu.parquet';\n```\n\nResults in:\n\n```\n┌──────────────┐\n│ count_star() │\n│    int64     │\n├──────────────┤\n│        98454 │\n└──────────────┘\n```\n\n#### Select all one file\n\nAnd you can see everything in it:\n\n```\nSELECT * FROM \n   's3://us-west-2.opendata.source.coop/google-research-open-buildings/geoparquet-admin1/country=LAO/Attapeu.parquet';\n```\n\nWhich should get you a response like:\n\n```\n┌────────────────┬────────────┬────────────────┬────────────────────────────────────────────────┬───────────┬─────────┬─────────┐\n│ area_in_meters │ confidence │ full_plus_code │                    geometry                    │    id     │ country │ admin_1 │\n│     double     │   double   │    varchar     │                      blob                      │   int64   │ varchar │ varchar │\n├────────────────┼────────────┼────────────────┼────────────────────────────────────────────────┼───────────┼─────────┼─────────┤\n│        86.0079 │     0.6857 │ 7P68VCWH+4962  │ \\x01\\x03\\x00\\x00\\x00\\x01\\x00\\x00\\x00\\x05\\x00…  │ 632004767 │ LAO     │ Attapeu │\n│        35.1024 │     0.6889 │ 7P68VCWH+4H65  │ \\x01\\x03\\x00\\x00\\x00\\x01\\x00\\x00\\x00\\x05\\x00…  │ 632004768 │ LAO     │ Attapeu │\n│        40.6071 │     0.6593 │ 7P68VCWH+53J5  │ \\x01\\x03\\x00\\x00\\x00\\x01\\x00\\x00\\x00\\x05\\x00…  │ 632004769 │ LAO     │ Attapeu │\n│           ·    │        ·   │       ·        │                       ·                        │     ·     │  ·      │    ·    │\n│           ·    │        ·   │       ·        │                       ·                        │     ·     │  ·      │    ·    │\n│           ·    │        ·   │       ·        │                       ·                        │     ·     │  ·      │    ·    │\n│ 641629546 │ LAO     │ Attapeu │\n│        59.2047 │     0.6885 │ 7P7976C9+J5X3  │ \\x01\\x03\\x00\\x00\\x00\\x01\\x00\\x00\\x00\\x05\\x00…  │ 641629547 │ LAO     │ Attapeu │\n│        13.8254 │     0.6183 │ 7P7976C9+M48G  │ \\x01\\x03\\x00\\x00\\x00\\x01\\x00\\x00\\x00\\x05\\x00…  │ 641629548 │ LAO     │ Attapeu │\n│       183.8289 │     0.7697 │ 7P7976H4+VHGV  │ \\x01\\x03\\x00\\x00\\x00\\x01\\x00\\x00\\x00\\x05\\x00…  │ 641629614 │ LAO     │ Attapeu │\n├────────────────┴────────────┴────────────────┴────────────────────────────────────────────────┴───────────┴─────────┴─────────┤\n│ 98454 rows (40 shown)                                                                                               7 columns │\n└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘\n```\n\n#### Filter one file\n\nFrom there you can easily filter, like only show the largest buildings:\n\n```\nSELECT * FROM \n  's3://us-west-2.opendata.source.coop/google-research-open-buildings/geoparquet-admin1/country=LAO/Attapeu.parquet'\n  WHERE area_in_meters \u003e 1000;\n```\n\n#### Get one file in your local duckdb\n\nIf you've got a fast connection you can easily just keep doing your sql queries against the parquet files that are sitting on S3. But you can also easily pull the data into a table and then work with it locally.\n\n```\nCREATE TABLE attapeu AS SELECT * EXCLUDE geometry, ST_GEOMFROMWKB(geometry) AS geometry FROM \n's3://us-west-2.opendata.source.coop/google-research-open-buildings/geoparquet-admin1/country=LAO/Attapeu.parquet';\n```\n\nThis creates a true 'geometry' type from the well known binary 'geometry' field, which \ncan then be used in spatial operations. Note this also shows one of Duck's [friendlier SQL](https://duckdb.org/2022/05/04/friendlier-sql.html) additions with `EXCLUDE`.\n\n#### Write out duckdb table\n\nYou can then write this out as common geospatial formats:\n\n```\nCOPY (SELECT * EXCLUDE geometry, ST_AsWKB(geometry) AS geometry from attapeu)\n      TO 'attapeu-1.fgb' WITH  (FORMAT GDAL, DRIVER 'FlatGeobuf');\n```\n\nThe DuckDB output does not seem to consistenly set the spatial reference system (hopefully someone will point out how to do this consistently or improve it in the future). You can clean this up with `ogr2ogr`:\n\n```\nogr2ogr -a_srs EPSG:4326 attapeu.fgb attapeu-1.fgb\n```\n\n#### Directly streaming output\n\nYou also don't have to instantiate the table in DuckDB if your connection is fast, you can just do:\n\n```\nCOPY (SELECT * EXCLUDE geometry, ST_AsWKB(ST_GEOMFROMWKB(geometry)) AS geometry FROM \n     's3://us-west-2.opendata.source.coop/google-research-open-buildings/geoparquet-admin1/country=LAO/Attapeu.parquet') \n     TO 'attapeu-2.fgb' WITH (FORMAT GDAL, DRIVER 'FlatGeobuf');\n```\n\nThis one also needs to cleaned up with a projection with ogr2ogr.\n\n### Working with a whole country\n\nUsing DuckDB with Parquet on S3 starts to really shine when you want to work with lots of data. There are lots of easy options to just download a file and then transform it. But bulk downloading from S3 and then getting the data formatted as you want can be a pain - configuring your S3 client, getting the names of everything to download, etc.\n\nWith DuckDB and these GeoParquet files you can just use various * patterns to select multiple files and treat them as a single one:\n\n```\nSELECT count(*) FROM 's3://us-west-2.opendata.source.coop/google-research-open-buildings/geoparquet-admin1/country=LAO/*.parquet';\n```\n\nThe above query gets you all the bulidings in Laos. If your connection is quite fast you can do all these calls directly on the parquet files. But for most it's easiest to load it locally:\n\n```\nCREATE TABLE laos AS SELECT * EXCLUDE geometry, ST_GEOMFROMWKB(geometry) AS geometry FROM \n's3://us-west-2.opendata.source.coop/google-research-open-buildings/geoparquet-admin1/country=LAO/*.parquet';\n```\n\nFrom there it's easy to add more data. Let's rename our table to `se_asia` and then pull down\nthe data from Thailand as well:\n\n```\nALTER TABLE laos RENAME TO se_asia;\nINSERT INTO se_asia from (SELECT * EXCLUDE geometry, ST_GEOMFROMWKB(geometry) AS geometry FROM \n's3://us-west-2.opendata.source.coop/google-research-open-buildings/geoparquet-admin1/country=THA/*.parquet');\n```\n\nThis will take a bit longer, as Thailand has about ten times the number of buildings of Laos. \n\nYou can continue to add other countries to the `se_asia` table, and then write it out as a number of gis formats just like above.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcholmes%2Fduckdb-geoparquet-tutorials","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcholmes%2Fduckdb-geoparquet-tutorials","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcholmes%2Fduckdb-geoparquet-tutorials/lists"}