{"id":48645576,"url":"https://github.com/shane98c/wiki-geoparquet","last_synced_at":"2026-04-19T05:01:40.362Z","repository":{"id":349071954,"uuid":"1193239720","full_name":"Shane98c/wiki-geoparquet","owner":"Shane98c","description":"geolocated english wikipedia articles in geoparquet and pmtiles","archived":false,"fork":false,"pushed_at":"2026-04-18T20:16:06.000Z","size":363,"stargazers_count":43,"open_issues_count":1,"forks_count":3,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-18T20:30:17.668Z","etag":null,"topics":["duckdb","geoparquet","geospatial","geotagged","parquet","pmtiles","wikidata","wikipedia"],"latest_commit_sha":null,"homepage":"https://shane98c.github.io/wiki-geoparquet/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Shane98c.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-27T02:35:41.000Z","updated_at":"2026-04-18T18:40:57.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Shane98c/wiki-geoparquet","commit_stats":null,"previous_names":["shane98c/wiki-geoparquet"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/Shane98c/wiki-geoparquet","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shane98c%2Fwiki-geoparquet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shane98c%2Fwiki-geoparquet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shane98c%2Fwiki-geoparquet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shane98c%2Fwiki-geoparquet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Shane98c","download_url":"https://codeload.github.com/Shane98c/wiki-geoparquet/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shane98c%2Fwiki-geoparquet/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31995167,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T20:23:30.271Z","status":"online","status_checked_at":"2026-04-19T02:00:07.110Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duckdb","geoparquet","geospatial","geotagged","parquet","pmtiles","wikidata","wikipedia"],"created_at":"2026-04-10T03:59:35.622Z","updated_at":"2026-04-19T05:01:40.356Z","avatar_url":"https://github.com/Shane98c.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# wiki-geoparquet\n\nEnglish Wikipedia main-namespace articles with Earth coordinates, as\n[GeoParquet](https://geoparquet.org/) +\n[PMTiles](https://docs.protomaps.com/pmtiles/).\n\nCoordinates from Wikidata P625, enriched with instance type, country, population,\nGeoNames IDs, inlink counts, and more. Updated monthly from Wikidata + Wikipedia dumps.\n\n**demo:**\n[shane98c.github.io/wiki-geoparquet](https://shane98c.github.io/wiki-geoparquet/)\n\n## Data\n\nAll files are on R2 with CORS enabled — query directly from the browser or any\nParquet-aware tool:\n\n| File                                                                                                                | Description                                                                                                           |\n| ------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |\n| [`wikipedia_geotagged.parquet`](https://pub-016504dd3a4d419a9c17a8939840935e.r2.dev/v2/wikipedia_geotagged.parquet) | GeoParquet, Hilbert-sorted with bbox covering                                                                         |\n| [`wikipedia_geotagged.pmtiles`](https://pub-016504dd3a4d419a9c17a8939840935e.r2.dev/v2/wikipedia_geotagged.pmtiles) | Vector tiles, auto-zoom with overzoom, drops by article length                                                        |\n| [`wikipedia_search.parquet`](https://pub-016504dd3a4d419a9c17a8939840935e.r2.dev/v2/wikipedia_search.parquet)       | Lightweight search index (lowercased label, coords, inlink count), sorted by label for prefix-range row-group pruning |\n\nPinned versions are also available on\n[GitHub Releases](https://github.com/Shane98c/wiki-geoparquet/releases/latest).\n\n## Schema\n\n| Column           | Type           | Description                                              |\n| ---------------- | -------------- | -------------------------------------------------------- |\n| `geometry`       | WKB Point      | WGS84 coordinates (from Wikidata P625)                   |\n| `page_id`        | int32          | Wikipedia page ID                                        |\n| `qid`            | string         | Wikidata QID (e.g. Q90)                                  |\n| `label`          | string         | Article title                                            |\n| `description`    | string         | Short description from `wikibase-shortdesc`              |\n| `instance_of`    | string         | Wikidata P31 type (e.g. \"city\", \"mountain\", \"museum\")    |\n| `country`        | string         | Country name from Wikidata P17                           |\n| `population`     | int64          | Population from Wikidata P1082 (nullable)                |\n| `geonames_id`    | string         | GeoNames ID for cross-referencing                        |\n| `elevation`      | int16          | Ground elevation in meters, sampled from Copernicus DEM  |\n| `gt_type`        | string         | Wikipedia geo classification (supplementary, from geo_tags) |\n| `page_len`       | int32          | Article length in bytes                                  |\n| `inlink_count`   | int32          | Number of namespace-0 pagelinks pointing here            |\n| `wikipedia_url`  | string         | Full article URL                                         |\n| `image_url`      | string         | Wikimedia Commons image URL                              |\n| `related_images` | list\\\u003cstring\\\u003e | Additional Wikidata images (P6802)                       |\n| `bbox`           | struct         | Covering bbox for spatial predicate pushdown             |\n\nThe PMTiles carry only `page_id`, `label`, and `inlink_count`. The demo\nenriches popups on click by querying the GeoParquet directly via DuckDB-WASM.\nReconstruct article URLs client-side: `https://en.wikipedia.org/?curid={page_id}`\n\n## Quick start\n\n### Query with DuckDB\n\n```sql\nINSTALL spatial; LOAD spatial;\n\n-- Find the most notable geotagged articles\nSELECT label, instance_of, country, inlink_count, ST_AsText(geometry)\nFROM 'https://pub-016504dd3a4d419a9c17a8939840935e.r2.dev/v2/wikipedia_geotagged.parquet'\nORDER BY inlink_count DESC\nLIMIT 20;\n\n-- Articles near Paris (48.8566, 2.3522) — ~50km bbox\nSELECT label, instance_of, country, inlink_count\nFROM 'https://pub-016504dd3a4d419a9c17a8939840935e.r2.dev/v2/wikipedia_geotagged.parquet'\nWHERE bbox.xmin BETWEEN 1.7 AND 3.0\n  AND bbox.ymin BETWEEN 48.4 AND 49.3\nORDER BY inlink_count DESC\nLIMIT 20;\n```\n\n### Browser search with duckdb-wasm\n\nThe `wikipedia_search.parquet` index has columns `label` (lowercased, used for\nboth sorting and lookup), `lon`, `lat`, and `inlink_count` for ranking. It's\nsorted by `label` so a lowercased prefix range lets DuckDB skip row groups and\nfetch ~2 MB per query instead of the full ~25 MB file.\n\n```js\n// In duckdb-wasm, INSTALL httpfs (or SET builtin_httpfs = false) — without\n// this the built-in HTTP handler downloads the whole file on every query.\n// See https://github.com/duckdb/duckdb-wasm/issues/2153.\nawait conn.query(\"SET builtin_httpfs = false;\");\n\nconst q = userInput.toLowerCase().replace(/'/g, \"''\");\nawait conn.query(`\n  SELECT label, lon, lat\n  FROM 'https://.../wikipedia_search.parquet'\n  WHERE label \u003e= '${q}' AND label \u003c '${q}~'\n  ORDER BY inlink_count DESC\n  LIMIT 10\n`);\n```\n\n### Use pmtiles in MapLibre\n\n```js\nimport { Protocol } from \"pmtiles\";\n\nlet protocol = new Protocol();\nmaplibregl.addProtocol(\"pmtiles\", protocol.tile);\n\nconst map = new maplibregl.Map({\n  style: {\n    sources: {\n      wikipedia: {\n        type: \"vector\",\n        url: \"pmtiles://https://pub-016504dd3a4d419a9c17a8939840935e.r2.dev/v2/wikipedia_geotagged.pmtiles\",\n      },\n    },\n    layers: [\n      {\n        id: \"articles\",\n        source: \"wikipedia\",\n        \"source-layer\": \"wikipedia\",\n        type: \"circle\",\n        paint: {\n          \"circle-radius\": [\n            \"interpolate\",\n            [\"linear\"],\n            [\"sqrt\", [\"get\", \"inlink_count\"]],\n            0,\n            2.5,\n            280,\n            20,\n          ],\n          \"circle-color\": \"#4264fb\",\n        },\n      },\n    ],\n  },\n});\n```\n\n## How it works\n\n1. Streams the Wikidata truthy N-Triples dump (~70 GB) to extract P625\n   coordinates and enrichment properties (instance type, country, population,\n   GeoNames ID, images)\n2. Resolves P31/P17 QIDs to human-readable labels via the Wikidata API\n3. Streams 5 Wikipedia SQL dump files (~12 GB) for article metadata, images,\n   descriptions, and inlink counts\n4. Writes Hilbert-sorted GeoParquet with bbox covering via DuckDB spatial\n5. Pipes DuckDB to [tippecanoe](https://github.com/felt/tippecanoe) for PMTiles\n   with attribute-based feature dropping\n\nSee the [Makefile](Makefile) for build steps.\n\n## Data source\n\nCoordinates and enrichment from\n[Wikidata entity dumps](https://dumps.wikimedia.org/wikidatawiki/entities/),\narticle metadata from\n[English Wikipedia SQL dumps](https://dumps.wikimedia.org/enwiki/latest/).\nBoth released under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/).\n\n## License\n\nCode: [MIT](LICENSE) Data outputs:\n[CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) (derived from\nWikipedia/Wikidata)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshane98c%2Fwiki-geoparquet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshane98c%2Fwiki-geoparquet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshane98c%2Fwiki-geoparquet/lists"}