{"id":24396918,"url":"https://github.com/cholmes/geoparquet-tools","last_synced_at":"2025-09-12T05:32:52.876Z","repository":{"id":273022683,"uuid":"918491294","full_name":"cholmes/geoparquet-tools","owner":"cholmes","description":"A collection of tools for GeoParquet, using PyArrow and DuckDB","archived":false,"fork":false,"pushed_at":"2025-02-04T06:22:37.000Z","size":79,"stargazers_count":28,"open_issues_count":2,"forks_count":1,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-29T18:36:32.038Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cholmes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-18T04:15:50.000Z","updated_at":"2025-03-29T16:50:57.000Z","dependencies_parsed_at":"2025-01-18T05:21:35.639Z","dependency_job_id":"654a14e9-679c-46a4-9177-a8603c813abe","html_url":"https://github.com/cholmes/geoparquet-tools","commit_stats":null,"previous_names":["cholmes/geoparquet-tools"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cholmes%2Fgeoparquet-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cholmes%2Fgeoparquet-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cholmes%2Fgeoparquet-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cholmes%2Fgeoparquet-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cholmes","download_url":"https://codeload.github.com/cholmes/geoparquet-tools/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250318299,"owners_count":21410911,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-19T21:59:57.700Z","updated_at":"2025-04-22T20:29:24.802Z","avatar_url":"https://github.com/cholmes.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# geoparquet-tools\n\nA collection of tools for GeoParquet, using PyArrow and DuckDB.\n\n## Installation\n\nFor now, just clone the repo and run `pip install -e .` from the root directory.\n\n## Usage\n\nThe `geoparquet-tools` package provides a command-line interface through the `gt` command. Here are the available commands:\n\n```\n$ gt --help\nUsage: gt [OPTIONS] COMMAND [ARGS]...\n\n  GeoParquet Tools CLI for working with GeoParquet files.\n\nOptions:\n  --help  Show this message and exit.\n\nCommands:\n  add        Commands for enhancing GeoParquet files in various ways.\n  check      Commands for checking GeoParquet files for best practices.\n  format     Commands for formatting GeoParquet files.\n  partition  Commands for partitioning GeoParquet files.\n  sort       Commands for sorting GeoParquet files.\n```\n\n### sort\n\nThe `sort` commands aims to provide different options to do spatial sorting of GeoParquet\nfiles. Right now it just does Hilbert sorting, using DuckDB's `ST_Hilbert` function. It\npreserves CRS information, so if you're working with projected data then it can be nicer\nthan using DuckDB directly. It outputs data according to recommended [GeoParquet\nbest practices](https://github.com/opengeospatial/geoparquet/pull/254/files) (except it doesn't yet right the bbox covering metadata).\n\n```\n$ gt sort hilbert --help\nUsage: gt sort hilbert [OPTIONS] INPUT_PARQUET OUTPUT_PARQUET\n\n  Reorder a GeoParquet file using Hilbert curve ordering.\n\n  Takes an input GeoParquet file and creates a new file with rows ordered by\n  their position along a Hilbert space-filling curve.\n\n  By default, applies optimal formatting (ZSTD compression, optimized row\n  groups, bbox metadata) while preserving the CRS.\n\nOptions:\n  -g, --geometry-column TEXT  Name of the geometry column (default: geometry)\n  -v, --verbose               Print verbose output\n  --help                      Show this message and exit.\n```\n\n### add\n\nThe `add` commands aim to enhance GeoParquet files in various ways, typically adding\nmore columns. Right now it just has a `country-codes` command, which adds a column with\nISO codes for countries based on spatial intersection, following the [administrative division extension](https://github.com/fiboa/administrative-division-extension) in [fiboa](https://github.com/fiboa). (adding bbox column is also on the todo list).\n\n```\n$ gt add admin-divisions --help\nUsage: gt add admin-divisions [OPTIONS] INPUT_PARQUET COUNTRIES_PARQUET\n                              OUTPUT_PARQUET\n\n  Add country ISO codes to a GeoParquet file based on spatial intersection.\n\nOptions:\n  --verbose  Print additional information.\n  --help     Show this message and exit.\n```\n\nThe `COUNTRIES_PARQUET` file that works will be available soon on [source cooperative](https://source.coop/cholmes/admin-boundaries). (Or you can easily make your own - it's just the Overture division, filtered by country, written out in GeoParquet). Future\nversions will aim to make this more automatic, and also enable different country file\ndefinnitions.\n\n### partition\n\nThe `partition` commands aim to provide different options to partition GeoParquet\nfiles. Right now it just has an `admin` command, which splits a GeoParquet file into\nseparate files by country code.\n\n```\n$ gt partition admin --help\nUsage: gt partition admin [OPTIONS] INPUT_PARQUET OUTPUT_FOLDER\n\n  Split a GeoParquet file into separate files by country code.\n\nOptions:\n  --verbose    Print additional information.\n  --overwrite  Overwrite existing country files.\n  --help       Show this message and exit.\n```\n\n### check\n\nThe `check` commands aim to provide different options to check GeoParquet files for\nadherence to [developing best practices](https://github.com/opengeospatial/geoparquet/pull/254/files). \n\n```\n$ gt check --help\nUsage: gt check [OPTIONS] COMMAND [ARGS]...\n\n  Commands for checking GeoParquet files for best practices.\n\nOptions:\n  --help  Show this message and exit.\n\nCommands:\n  all          Run all checks on a GeoParquet file.\n  bbox         Check GeoParquet metadata version and bbox structure.\n  compression  Check compression settings for geometry column.\n  row-group    Check row group optimization.\n  spatial      Check if a GeoParquet file is spatially ordered.\n```\n\n### format\n\nThe `format` command is still in development. It aims to enable formatting of GeoParquet\naccording to best practices, either all at once or by individual command, in sync with \nthe 'check'. So you could easily run check and then format. Right now it just has the\nability to add bbox metadata.\n\n\n## TODO's\n\nFirst todo is to convert all of these into proper github issues ;) \n\n - get hive partitioning working for country stuff\n - better handling of country file - download it automatically, test other country files, make a bit more generic. Param to set the country iso column (get rid of the table that guesses it)\n - spatial partitioning by kd-tree and s2\n - add tests\n - add better docs\n - call for country splitting to be all in one.\n - option for further partitioning within admin boundaries (like s2 / kd-tree when the files are above a threshold)\n - admin level 2 splitting\n - ticket to admin extension in fiboa about putting source boundary file.\n - tool to print country code stats - number of records in each country\n - better decompisition of checks - core call on its own, layer on the warning info\n - better handling of really small files - if only one row group don't print red on the size, and don't fail on spatial order analysis.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcholmes%2Fgeoparquet-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcholmes%2Fgeoparquet-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcholmes%2Fgeoparquet-tools/lists"}