{"id":20705362,"url":"https://github.com/whosonfirst/go-whosonfirst-sqlite-features-index","last_synced_at":"2025-05-10T20:30:47.475Z","repository":{"id":97594584,"uuid":"190258044","full_name":"whosonfirst/go-whosonfirst-sqlite-features-index","owner":"whosonfirst","description":"Go package for indexing Who's On First features in SQLite databases.","archived":true,"fork":false,"pushed_at":"2024-12-13T18:31:33.000Z","size":26217,"stargazers_count":0,"open_issues_count":6,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-11T05:14:47.558Z","etag":null,"topics":["geojson","golang","sqlite","whosonfirst"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/whosonfirst.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-04T18:30:31.000Z","updated_at":"2024-12-13T18:31:47.000Z","dependencies_parsed_at":"2024-02-04T02:23:56.890Z","dependency_job_id":"e5bf9b4e-1e57-430b-bde8-f297d06f4a94","html_url":"https://github.com/whosonfirst/go-whosonfirst-sqlite-features-index","commit_stats":null,"previous_names":[],"tags_count":38,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whosonfirst%2Fgo-whosonfirst-sqlite-features-index","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whosonfirst%2Fgo-whosonfirst-sqlite-features-index/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whosonfirst%2Fgo-whosonfirst-sqlite-features-index/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whosonfirst%2Fgo-whosonfirst-sqlite-features-index/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/whosonfirst","download_url":"https://codeload.github.com/whosonfirst/go-whosonfirst-sqlite-features-index/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253480237,"owners_count":21915245,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["geojson","golang","sqlite","whosonfirst"],"created_at":"2024-11-17T01:17:46.957Z","updated_at":"2025-05-10T20:30:42.465Z","avatar_url":"https://github.com/whosonfirst.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# go-whosonfirst-sqlite-features-index\n\nGo package for indexing Who's On First features in SQLite databases using the `whosonfirst/go-whosonfirst-sqlite-index` and `whosonfirst/go-whosonfirst-sqlite-features` packages.\n\nThe `go-whosonfirst-sqlite-index` package indexes SQLite databases using table constucts defined in the `aaronland/go-sqlite` package and records defined by the `whosonfirst/go-whosonfirst-iterate/v2` package\n\n## Documentation\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/whosonfirst/go-whosonfirst-sqlite-features-index.svg)](https://pkg.go.dev/github.com/whosonfirst/go-whosonfirst-sqlite-features-index)\n\n## Tools\n\nTo build binary versions of these tools run the `cli` Makefile target. For example:\n\n```\n$\u003e make cli\ngo build -mod vendor -o bin/wof-sqlite-index-features cmd/wof-sqlite-index-features/main.go\n```\n\n### wof-sqlite-index-features\n\n```\n$\u003e ./bin/wof-sqlite-index-features -h\n  -all\n    \tIndex all tables (except the 'search' and 'geometries' tables which you need to specify explicitly)\n  -ancestors\n    \tIndex the 'ancestors' tables\n  -concordances\n    \tIndex the 'concordances' tables\n  -database-uri string\n    \t (default \"modernc://mem\")\n  -geojson\n    \tIndex the 'geojson' table\n  -geometries\n    \tIndex the 'geometries' table (requires that libspatialite already be installed)\n  -index-alt value\n    \tZero or more table names where alt geometry files should be indexed.\n  -index-alt-files\n    \tIndex alt geometries. This flag is deprecated, please use -index-alt=TABLE,TABLE,etc. instead. To index alt geometries in all the applicable tables use -index-alt=*\n  -index-relations\n    \tIndex the records related to a feature, specifically wof:belongsto, wof:depicts and wof:involves. Alt files for relations are not indexed at this time.\n  -index-relations-reader-uri string\n    \tA valid go-reader.Reader URI from which to read data for a relations candidate.\n  -iterator-uri string\n    \tA valid whosonfirst/go-whosonfirst-iterate/v2 URI. Supported emitter URI schemes are: directory://,featurecollection://,file://,filelist://,geojsonl://,git://,null://,repo:// (default \"repo://\")\n  -live-hard-die-fast\n    \tEnable various performance-related pragmas at the expense of possible (unlikely) database corruption (default true)\n  -names\n    \tIndex the 'names' table\n  -optimize\n    \tAttempt to optimize the database before closing connection (default true)\n  -processes int\n    \tThe number of concurrent processes to index data with (default 16)\n  -properties\n    \tIndex the 'properties' table\n  -rtree\n    \tIndex the 'rtree' table\n  -search\n    \tIndex the 'search' table (using SQLite FTS4 full-text indexer)\n  -spatial-tables\n    \tIf true then index the necessary tables for use with the whosonfirst/go-whosonfirst-spatial-sqlite package.\n  -spelunker-tables\n    \tIf true then index the necessary tables for use with the whosonfirst/go-whosonfirst-spelunker packages\n  -spr\n    \tIndex the 'spr' table\n  -strict-alt-files\n    \tBe strict when indexing alt geometries (default true)\n  -supersedes\n    \tIndex the 'supersedes' table\n  -timings\n    \tDisplay timings during and after indexing\n```\n\nFor example:\n\n```\n$\u003e ./bin/wof-sqlite-index-features \\\n\t-database-uri modernc://cwd/microhoods.db \\\n\t-all \\\n\t-iterator-uri 'repo://?include=properties.wof:placetype=microhood' \\\n\t/usr/local/data/whosonfirst-data-admin-us\n```\n\nOr creating databases for all the Who's On First repos:\n\n```\n#!/bin/sh\n\nfor REPO in $@\ndo\n\n    if [ ! -d ${REPO}/data ]\n    then\n\techo \"${REPO} has no data directory\"\n\tcontinue\n    fi\n    \n    FNAME=`basename ${REPO}`\n    echo \"make db for ${FNAME}\"\n\n    if [ -f \"/usr/local/data/whosonfirst-sqlite/${FNAME}.db\" ]\n    then\n\trm /usr/local/data/whosonfirst-sqlite/${FNAME}.db\n    fi\n\n    ./bin/wof-sqlite-index-features -timings -all -database-uri modernc:///usr/local/data/whosonfirst-sqlite/${FNAME}-latest.db -iterator-uri repo:// ${REPO} \n\ndone\n```    \n\n#### Inline queries\n\nYou can also specify inline queries by appending one or more `include` or `exclude` parameters to a `emitter.Emitter` URI, where the value is a string in the format of:\n\n```\n{PATH}={REGULAR EXPRESSION}\n```\n\nPaths follow the dot notation syntax used by the [tidwall/gjson](https://github.com/tidwall/gjson) package and regular expressions are any valid [Go language regular expression](https://golang.org/pkg/regexp/). Successful path lookups will be treated as a list of candidates and each candidate's string value will be tested against the regular expression's [MatchString](https://golang.org/pkg/regexp/#Regexp.MatchString) method.\n\nFor example:\n\n```\n$\u003e ./bin/wof-sqlite-index-features \\\n\t-all \\\n\t-database-uri modernc://cwd/ca-region.db \\\n\t-iterator-uri 'repo://?include=properties.wof:placetype=region' \\\t\n\t/usr/local/data/whosonfirst-data-admin-ca\n\n$\u003e sqlite3 ca-region.db\n\nSQLite version 3.28.0 2019-04-15 14:49:49\nEnter \".help\" for usage hints.\nsqlite\u003e SELECT id,name,placetype FROM spr;\n85682057|Ontario|region\n85682117|British Columbia|region\n85682065|New Brunswick|region\n85682123|Newfoundland and Labrador|region\n85682067|Northwest Territories|region\n85682075|Nova Scotia|region\n85682081|Prince Edward Island|region\n85682085|Manitoba|region\n85682091|Alberta|region\n85682095|Yukon|region\n85682113|Saskatchewan|region\n136251273|Quebec|region\n85682105|Nunavut|region\n```\n\nYou can pass multiple query parameters. For example:\n\n```\n$\u003e ./bin/wof-sqlite-index-features \\\n\t-all \\\n\t-database-uri modernc://cwd/ca-region.db \\\n\t-iterator-uri 'repo://?include=properties.wof:placetype=region\u0026include=properties.wof:name=(?i)new.*' \\\t\n\t/usr/local/data/whosonfirst-data-admin-ca\n\n$\u003e sqlite3 ca-region-new.db\n\nSQLite version 3.28.0 2019-04-15 14:49:49\nEnter \".help\" for usage hints.\nsqlite\u003e SELECT id,name,placetype FROM spr;\n85682065|New Brunswick|region\n85682123|Newfoundland and Labrador|region\n```\n\nThe default query mode is to ensure that all queries match but you can also specify that only one or more queries need to match by appending a `include_mode` or `exclude_mode` parameter where the value is either \"ANY\" or \"ALL\".\n\n#### SQLite performace-related PRAGMA\n\nNote that the `-live-hard-die-fast` flag is enabled by default. That is to enable a number of performace-related PRAGMA commands (described [here](https://blog.devart.com/increasing-sqlite-performance.html) and [here](https://www.gaia-gis.it/gaia-sins/spatialite-cookbook/html/system.html)) without which database index can be prohibitive and time-consuming. These is a small but unlikely chance of database corruptions when this flag is enabled.\n\nAlso note that the `-live-hard-die-fast` flag will cause the `PAGE_SIZE` and `CACHE_SIZE` PRAGMAs to be set to `4096` and `1000000` respectively so the eventual cache size will require 4GB of memory. This is probably fine on most systems where you'll be indexing data but I am open to the idea that we may need to revisit those numbers or at least make them configurable.\n\n## Spatial indexes\n\n### RTree\n\nRTree indexes are available if SQLite has been compiled with the [R*Tree module](https://www.sqlite.org/rtree.html) and you have indexed the [rtree](https://github.com/whosonfirst/go-whosonfirst-sqlite-features#rtree), [spr](https://github.com/whosonfirst/go-whosonfirst-sqlite-features#spr) and [properties](https://github.com/whosonfirst/go-whosonfirst-sqlite-features#properties) tables. For example:\n\n```\n$\u003e ./bin/wof-sqlite-index-features \\\n\t-index-alt-files \\\n\t-rtree \\\n\t-spr \\\n\t-properties \\\n\t-timings \\\n\t-database-uri modernc:///usr/local/ca-alt.db \\\n\t/usr/local/data/whosonfirst-data-admin-ca/\n```\n\n## Indexing \n\nIndexing time will vary depending on the specifics of your hardware (available RAM, CPU, disk I/O) but as a rule building indexes with the `geometries` table will take longer, and create a larger database, than doing so without. For example indexing the [whosonfirst-data](https://github.com/whosonfirst-data/whosonfirst-data) repository with spatial indexes:\n\n```\n$\u003e ./bin/wof-sqlite-index-features \\\n\t-driver spatialite \\\n\t-all \\\n\t-geometries \\\n\t-database-uri modernc:///usr/local/data/dist/sqlite/whosonfirst-data-latest.db \\\n\t-timings \\\n\t/usr/local/data/whosonfirst-data\n\n...time passes...\n06:12:51.274132 [wof-sqlite-index-features] STATUS time to index geojson (951541) : 13m41.994217581s\n06:12:51.274158 [wof-sqlite-index-features] STATUS time to index spr (951541) : 13m0.21007633s\n06:12:51.274173 [wof-sqlite-index-features] STATUS time to index names (951541) : 17m50.759093941s\n06:12:51.274178 [wof-sqlite-index-features] STATUS time to index ancestors (951541) : 3m37.431723948s\n06:12:51.274182 [wof-sqlite-index-features] STATUS time to index concordances (951541) : 2m36.737857568s\n06:12:51.274187 [wof-sqlite-index-features] STATUS time to index geometries (951541) : 43m48.39054903s\n06:12:51.274192 [wof-sqlite-index-features] STATUS time to index all (951541) : 4h41m45.492361401s\n\n\u003e du -h /usr/local/data/dist/sqlite/whosonfirst-data-latest.db\n15G     /usr/local/data/dist/sqlite/whosonfirst-data-latest.db\n```\n\nAnd without:\n\n```\n$\u003e ./bin/wof-sqlite-index-features \\\n\t-all \\\n\t-database-uri modernc:///usr/local/data/dist/sqlite/whosonfirst-data-latest-nospatial.db \\\n\t-timings \\\n\t/usr/local/data/whosonfirst-data\n...time passes...\n10:06:13.226187 [wof-sqlite-index-features] STATUS time to index names (951541) : 12m32.359733539s\n10:06:13.226206 [wof-sqlite-index-features] STATUS time to index ancestors (951541) : 3m27.294843778s\n10:06:13.226212 [wof-sqlite-index-features] STATUS time to index concordances (951541) : 2m5.947968206s\n10:06:13.226220 [wof-sqlite-index-features] STATUS time to index geojson (951541) : 10m11.355455209s\n10:06:13.226226 [wof-sqlite-index-features] STATUS time to index spr (951541) : 11m32.687081163s\n10:06:13.226233 [wof-sqlite-index-features] STATUS time to index all (951541) : 3h43m20.687783762s\n\n\u003e du -h /usr/local/data/dist/sqlite/whosonfirst-data-latest-nospatial.db \n12G     /usr/local/data/dist/sqlite/whosonfirst-data-latest-nospatial.db\n```\n\nAs of this writing individual tables are indexed atomically. There may be some improvements to be made indexing tables in separate Go routines but my hunch is this will make SQLite sad and cause a lot of table lock errors. I don't need to be right about that, though...\n\n## See also\n\n* https://github.com/aaronland/go-sqlite\n* https://github.com/aaronland/go-sqlite-modernc\n* https://github.com/whosonfirst/go-whosonfirst-sqlite-features\n* https://github.com/whosonfirst/go-whosonfirst-sqlite-index\n* https://github.com/whosonfirst/go-whosonfirst-iterate","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwhosonfirst%2Fgo-whosonfirst-sqlite-features-index","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwhosonfirst%2Fgo-whosonfirst-sqlite-features-index","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwhosonfirst%2Fgo-whosonfirst-sqlite-features-index/lists"}