{"id":51391092,"url":"https://github.com/marcosci/pelias-overture","last_synced_at":"2026-07-03T23:33:53.877Z","repository":{"id":353615442,"uuid":"1220182972","full_name":"marcosci/pelias-overture","owner":"marcosci","description":"Pelias importer for Overture Maps (addresses + places) via DuckDB GeoParquet.","archived":false,"fork":false,"pushed_at":"2026-04-24T18:25:27.000Z","size":148,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-24T18:26:01.766Z","etag":null,"topics":["duckdb","geocoder","geocoding","geoparquet","gers","importer","nodejs","overture-maps","overturemaps","pelias"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marcosci.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-24T16:24:25.000Z","updated_at":"2026-04-24T18:25:31.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/marcosci/pelias-overture","commit_stats":null,"previous_names":["marcosci/pelias-overture"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/marcosci/pelias-overture","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcosci%2Fpelias-overture","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcosci%2Fpelias-overture/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcosci%2Fpelias-overture/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcosci%2Fpelias-overture/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marcosci","download_url":"https://codeload.github.com/marcosci/pelias-overture/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcosci%2Fpelias-overture/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35105483,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-03T02:00:05.635Z","response_time":110,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duckdb","geocoder","geocoding","geoparquet","gers","importer","nodejs","overture-maps","overturemaps","pelias"],"created_at":"2026-07-03T23:33:53.224Z","updated_at":"2026-07-03T23:33:53.870Z","avatar_url":"https://github.com/marcosci.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pelias-overture\n\nA [Pelias](https://github.com/pelias/pelias) importer for [Overture Maps](https://overturemaps.org) data.\n\nStreams Overture GeoParquet directly into Elasticsearch via [`pelias-dbclient`](https://github.com/pelias/dbclient), emitting normalised Pelias `Document` records with stable [GERS](https://docs.overturemaps.org/gers/) source identifiers.\n\nStatus: **alpha.** Addresses and places themes work end-to-end against the public Overture S3 mirror and local partition mirrors. Tracks [pelias/pelias#954](https://github.com/pelias/pelias/issues/954).\n\n## Themes\n\n| Theme     | Status | Notes |\n|-----------|--------|-------|\n| addresses | ✓      | Point addresses, country filter and bbox push-down into DuckDB. |\n| places    | ✓      | POIs filtered by `minConfidence` (default 0.7) and optional category include/exclude. Multilingual names preserved. Runs additive to OSM venues by default. |\n| divisions | —      | Out of scope. Pelias already relies on Who's on First for admin hierarchy; see [pelias/pelias#954](https://github.com/pelias/pelias/issues/954). |\n| buildings, transportation, base | — | Out of scope for geocoding. |\n\n## Install\n\n```bash\ngit clone https://github.com/marcosci/pelias-overture.git\ncd pelias-overture\nnpm install\ncp pelias.json.example pelias.json\n$EDITOR pelias.json\n```\n\nRequires Node 20+. DuckDB native bindings are installed automatically; if they fail (e.g. arch without prebuilt binaries), see the Python fallback below.\n\n## Usage\n\n1. Create the Elasticsearch index + `pelias` alias using the official [`pelias-schema`](https://github.com/pelias/schema) mappings. This requires a Pelias-flavoured Elasticsearch with the ICU analysis plugin pre-installed — use the [`pelias/elasticsearch`](https://hub.docker.com/r/pelias/elasticsearch) image, **not** the stock `docker.elastic.co` one:\n   ```bash\n   npm run es:create-index\n   ```\n2. Run the importer:\n   ```bash\n   ./bin/start\n   ```\n\nOr via GitHub Container Registry:\n\n```bash\ndocker pull ghcr.io/marcosci/pelias-overture:latest\ndocker run --rm \\\n  -v $(pwd)/pelias.json:/app/pelias.json:ro \\\n  -v /data/overture:/data/overture:ro \\\n  -e PELIAS_CONFIG=/app/pelias.json \\\n  ghcr.io/marcosci/pelias-overture:latest\n```\n\nTags published: `latest` (main), `vX.Y.Z` + `vX.Y` + `vX` on release, `sha-\u003cshort\u003e` on every push.\n\nReads `pelias.json` via `pelias-config` and streams enabled themes through:\n\n```\nparquet reader → filter → transform → wof-admin-lookup → blacklist → dedupe → dbclient\n```\n\n## Configuration\n\nTop-level keys under `imports.overture`:\n\n| Key | Default | Description |\n|---|---|---|\n| `datapath` | — | Filesystem path to an Overture release, e.g. `/data/overture/2026-04-15.0`. |\n| `s3` | `{ enabled: false }` | `{ enabled, bucket, region, release }` to read directly from Overture S3. |\n| `reader` | `\"duckdb\"` | `\"duckdb\"` (primary) or `\"python\"` (spawn `overturemaps` CLI fallback). |\n| `themes.addresses.enabled` | `true` | |\n| `themes.places.enabled` | `true` | |\n| `themes.places.minConfidence` | `0.7` | Drops POIs below this Overture confidence score. |\n| `themes.places.categories.include` | `[]` | Whitelist of Overture category strings. |\n| `themes.places.categories.exclude` | `[]` | Blacklist of Overture category strings. |\n| `countryCode` | `[]` | ISO 3166-1 alpha-2 codes; empty = all. |\n| `bbox` | — | `[minLon, minLat, maxLon, maxLat]` filter. |\n| `adminLookup` | `true` | Attach WOF admin hierarchy via polygon PIP. |\n| `adminLookupConcurrency` | `4` | Max in-flight PIP calls. |\n| `deduplicate` | `true` | In-run dedupe on `source_id`. |\n| `batchSize` | `500` | Reader batch size. |\n\nSee [`pelias.json.example`](pelias.json.example) for the full shape.\n\n### Python fallback reader\n\nWhen DuckDB is unavailable:\n\n```bash\npipx install overturemaps\n```\n\nSet `imports.overture.reader: \"python\"` and `imports.overture.s3.release: \"2026-04-15.0\"`. The importer will spawn `overturemaps download` and consume the GeoJSON-seq stream instead of reading parquet directly. Expect ~2–3× slower throughput.\n\n## Identity \u0026 identifiers\n\n- `source` = `\"overture\"`\n- `layer` = `\"address\"` (addresses) or `\"venue\"` (places)\n- `source_id` = `\"overture:\u003ctheme\u003e:\u003cgers\u003e\"` — stable across Overture monthly releases\n- Raw GERS and upstream provenance kept at `addendum.overture` (gers, country, postal_city, sources, websites, phones, …)\n- Pelias `gid` is derived normally from `source:layer:source_id` — Overture GERS never collides with Pelias `gid`.\n\n## Dedup vs OSM / OpenAddresses\n\nOverture overlaps substantially with OSM venues and OpenAddresses. This importer ships in **additive** mode: it never silently disables other sources. Operator-side strategies live in [`docs/dedup.md`](docs/dedup.md).\n\n## Documentation\n\n- [`docs/parquet-reader-decision.md`](docs/parquet-reader-decision.md) — DuckDB vs Python CLI vs hyparquet.\n- [`docs/dedup.md`](docs/dedup.md) — cross-source dedup recommendations.\n- [`docs/category-map.md`](docs/category-map.md) — Overture → Pelias category taxonomy.\n- [`docs/performance.md`](docs/performance.md) — tuning for reader, admin lookup, and dbclient.\n- [`docs/benchmark-results.md`](docs/benchmark-results.md) — measured throughput snapshots.\n- [`docs/roadmap.md`](docs/roadmap.md) — upstream + ecosystem work beyond this repo.\n\n## Layout\n\n```\nbin/start                   CLI entry\nlib/pipeline.js             admin lookup → blacklist → dedupe → dbclient\nlib/sources/\n  index.js                  reader dispatch (duckdb | python)\n  parquetReader.js          DuckDB GeoParquet stream\n  cliFallbackReader.js      overturemaps CLI GeoJSON-seq stream\n  partitionPlanner.js       local + S3 partition enumeration\nlib/themes/\n  addresses/{index,transform,filter}.js\n  places/{index,transform,filter,categoryMap}.js\nlib/util/\n  gers.js                   source_id helpers\n  geometry.js               centroid + bbox helpers\nschema.js                   joi config schema\nscripts/benchmark-parquet-reader.js\ntest/unit, test/integration, test/e2e\n```\n\n## Tests\n\n```bash\nnpm test                    # unit + integration, jest\nnpm run test:unit\nnpm run test:integration    # generates local parquet fixture via DuckDB\n```\n\n## Release\n\nTag pushes (`vX.Y.Z`) trigger the `docker-publish` workflow, which pushes multi-tag images to [`ghcr.io/marcosci/pelias-overture`](https://github.com/marcosci/pelias-overture/pkgs/container/pelias-overture). Authentication uses the built-in `GITHUB_TOKEN` — no secrets to configure.\n\nImages publish for `linux/amd64` and `linux/arm64`. The DuckDB bindings ship prebuilt binaries for both architectures; buildx + QEMU cross-builds in CI.\n\n## Attribution\n\nOverture data is licensed under a mix of CDLA-Permissive-2.0 and ODbL depending on theme. See the [Overture data guide](https://docs.overturemaps.org/guides/) for attribution requirements. This importer preserves upstream `source_tags` at `addendum.overture.sources` so downstream consumers can honour them.\n\n## License\n\nMIT. See [`LICENSE`](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarcosci%2Fpelias-overture","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarcosci%2Fpelias-overture","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarcosci%2Fpelias-overture/lists"}