{"id":23188121,"url":"https://github.com/borisgontar/osmtags-js-sqlite","last_synced_at":"2026-01-04T14:08:40.553Z","repository":{"id":226594344,"uuid":"172269216","full_name":"borisgontar/osmtags-js-sqlite","owner":"borisgontar","description":"A Node.js application to collect tags from OpenStreetMap osm.pbf files into a SQLite3 database.","archived":false,"fork":false,"pushed_at":"2024-09-16T11:04:47.000Z","size":11909,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-09-16T12:42:28.137Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/borisgontar.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2019-02-23T22:05:31.000Z","updated_at":"2024-09-16T11:04:00.000Z","dependencies_parsed_at":"2024-03-09T23:45:29.313Z","dependency_job_id":null,"html_url":"https://github.com/borisgontar/osmtags-js-sqlite","commit_stats":null,"previous_names":["borisgontar/osmtags-js-sqlite"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borisgontar%2Fosmtags-js-sqlite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borisgontar%2Fosmtags-js-sqlite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borisgontar%2Fosmtags-js-sqlite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borisgontar%2Fosmtags-js-sqlite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/borisgontar","download_url":"https://codeload.github.com/borisgontar/osmtags-js-sqlite/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230260727,"owners_count":18198591,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-18T11:12:58.444Z","updated_at":"2026-01-04T14:08:40.514Z","avatar_url":"https://github.com/borisgontar.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# osmtags-js-sqlite\n\nThis is a simple Node.js application to collect tags from OpenStreetMap `osm.pbf` files\ninto a SQLite3 database. It extracts all tags from the input files and creates\na table in the target database with the following data:\n* All distinct keys present in the tags.\n* For each key, all (if not too many) its distinct values.\n* For each {key, value} pair, numbers of times this pair was found in nodes, ways and relations.\n\nThe table is created as:\n```sql\ncreate table osmtags (\n  key text not null,   -- tag's key, e.g. 'amenity'\n  value text not null, -- tag's value, e.g. 'fuel'\n  n int default 0,     -- number of times 'amenity=fuel' seen in nodes\n  w int default 0,     --    same for ways\n  r int default 0,     --    same for relations\n  primary key(key, value)\n);\n```\nSome keys, like `name` can have too many values, so there is a limit for number\nof such values. When this limit is reached all {key, value} table rows for such\nkey are coalesced into a single row with value \"~\". In this case the counts\nrepresent numbers of times the key was found in nodes, ways and relations,\nregardless of its value.\n\nSome keys can have multiple 'subkeys', like `addr:street`, `addr:housenumber`, etc.\nThe application allows to coalesce such keys into a single one (`addr` in this example).\n\n## Installation\n1. Clone the repository.\n2. Run `npm install` to install the dependencies.\n\n## Usage\nRun the application as:\n```bash\n$ node osmtags.js options path-to-osm.pbf [another-path ...]\n```\n\nwhere _options_ are:\n```\n$ node osmtags.js -h\n\nExtracts all keys and their values from an OSM pbf file into a SQLite database\n\nOptions:\n  -d, --database    Path to the database.                           [required]\n  -c, --coalesce    Coalesce keys like name:xx into single name.\n  -l, --limit       Represent key with more than l values as single row\n                    with value \"~\".                              [default 256]\n  -q, --quiet       Run silently.\n  -h, --help        Show this help and exit.\n      --version     Show version number and exit.\n\nPositional args: paths to the input files.\n```\nNote that:\n\n* If the target database does not exists it will be created.\n  If the table `osmtags` does not exists it will be created,\n  otherwise the new data will be merged with existing data.\n\n* If the `-c` option is present, all keys like `addr` and `addr:whatever`\nare treated as the same key `addr`.\n\n* The `-l` option sets the limit of stored distinct values for a key\n to the specified number. The default is 256.\n\nThe application is simple and fast. Counting tags in\n`planet-latest.osm.pbf` (dated Oct. 2022, 67Gb) took 3 hours 10 min.\non my PC (Intel i7-2600 at 3.40GHz, memory DDR3 at 800MHz, SATA SSD).\n```\n$ node osmtags.js -d planet-tags.sqlite -c -l 1024 planet-latest.osm.pbf\nreading planet-latest.osm.pbf\nscanned: 7961992696 nodes, 892558311 ways, 10291367 relations\n         778282 items/sec.\nno tags in 7761372553 nodes, 14830012 ways, 182 relations\nelapsed: 3:09:54.805 (h:mm:ss.mmm)\n```\n\nOn my NUC (Intel i0-12900, memory DDR4 at 3200 MHz, NVMe SSD) the process\ntook 1 hour 14 min., 1995181 items/sec.\n\nThe resuling database is about 36Mb in size. It's included into the `example`\nsubdirectory just in case you want to play with it.\n\n## Examples\n\nTo run queries against the database you need the `sqlite3` utility.\nDownload the 'sqlite-tools' binaries from https://sqlite.org/download.html and\ncopy the executable files into a directory in your $PATH. To make query results\nlook better, set output mode like this:\n```bash\n$ sqlite3 planet-tags.sqlite\nSQLite version 3.40.0 2022-11-16 12:10:08\nEnter \".help\" for usage hints.\nsqlite\u003e .mode col\nsqlite\u003e .header on\n```\n\nThe application does not trim leading and trailing white space from\nkeys and values. Let's see if there are such:\n```sql\nsqlite\u003e select count(*) from osmtags where key != trim(key);\n14\nselect count(*) from osmtags where value != trim(value);\n564\n```\nOpenStreetMap allows but\n[does not recommend](https://wiki.openstreetmap.org/wiki/Semi-colon_value_separator)\nusing semicolons as separators for multiple values of the same key. Let's see:\n```sql\nsqlite\u003e select count(*) from osmtags where instr(value, ';') != 0;\n21682\n```\nWhich keys with too many values are most frequenly used?\n```sql\nsqlite\u003e select key, (n+w+r) total from osmtags where value='~' order by total desc limit 10;\nkey         total\n----------  ----------\nbuilding    575692021\naddr        559712241\nsource      301573005\nhighway     229231539\nname        111103790\ntiger       75717248\nnatural     55921276\nsurface     48147704\nref         47618702\nlanduse     37465609\n```\nAnd what about keys with not too many values?\n```sql\nselect key, value, (n+w+r) total from osmtags where value!='~' order by total desc limit 10;\nkey         value       total\n----------  ----------  ----------\npower       tower       14762730\nwall        no          12164943\npower       pole        10497260\nlanes       2           9149008\nlayer       1           6741169\nlit         yes         6404340\ncreated_by  JOSM        4855805\nlanes       1           4548328\nintermitte  yes         4425470\nleaf_type   broadleave  4273000\n```\nWhich tags are used only in ways?\n```sql\nselect key,value,w from osmtags where n=0 and r=0 order by w desc limit 10;\nkey         value       w\n----------  ----------  ----------\nLINZ        cliff_edge  54373\nHFCS        Urban Mino  47301\nmapper      mspray11    36623\nHFCS        Urban Coll  34776\nid_origin   ~           34293\nmapper      mspray12    32129\nzoning      grass       29480\nfootway     asphalt     27913\nmapper      mspray13    26951\nSource      Akros       25816\n```\n\n## P.S.\nOf course the [taginfo](https://taginfo.openstreetmap.org/) site already has\nall kinds of statistics about tags in the OpenStreetMap database.\nI just wanted to have a tool, something small and simple and, most important,\neasily customizable to my needs. After all, it's about 200 lines of code.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fborisgontar%2Fosmtags-js-sqlite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fborisgontar%2Fosmtags-js-sqlite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fborisgontar%2Fosmtags-js-sqlite/lists"}