{"id":22398113,"url":"https://github.com/borisgontar/osm-pbf-parser-node","last_synced_at":"2025-07-31T13:31:41.530Z","repository":{"id":64644425,"uuid":"576635800","full_name":"borisgontar/osm-pbf-parser-node","owner":"borisgontar","description":"Streaming OpenStreetMap PBF parser for Node.js","archived":false,"fork":false,"pushed_at":"2024-07-14T10:35:30.000Z","size":53,"stargazers_count":14,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-03T17:43:41.921Z","etag":null,"topics":["node-js","openstreetmap","parser","pbf"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/borisgontar.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-10T13:38:16.000Z","updated_at":"2024-08-29T19:15:10.000Z","dependencies_parsed_at":"2024-07-14T11:43:01.884Z","dependency_job_id":null,"html_url":"https://github.com/borisgontar/osm-pbf-parser-node","commit_stats":{"total_commits":16,"total_committers":1,"mean_commits":16.0,"dds":0.0,"last_synced_commit":"bc5d21471b5148d4ba17d4dcf9f0125526ffe2ba"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borisgontar%2Fosm-pbf-parser-node","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borisgontar%2Fosm-pbf-parser-node/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borisgontar%2Fosm-pbf-parser-node/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/borisgontar%2Fosm-pbf-parser-node/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/borisgontar","download_url":"https://codeload.github.com/borisgontar/osm-pbf-parser-node/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227990203,"owners_count":17852200,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["node-js","openstreetmap","parser","pbf"],"created_at":"2024-12-05T07:08:53.477Z","updated_at":"2024-12-05T07:08:54.058Z","avatar_url":"https://github.com/borisgontar.png","language":"JavaScript","funding_links":[],"categories":["Libraries"],"sub_categories":["JavaScript"],"readme":"# osm-pbf-parser-node\n\nStreaming [OpenStreetMap PBF](https://wiki.openstreetmap.org/wiki/PBF_Format) parser\n\nThis Node.js module reads a stream in osm.pbf format and transforms it\ninto a readable stream of OSM entities (header, nodes, ways and relations).\n\nThe module uses (more or less) recent Javascript features, like\nthe nullish coalescing (??) operator, so it will not work with\nolder versions of Node.js. Works for me on version 16.13.\n\n## Example\n\n```javascript\nimport { createOSMStream } from 'osm-pbf-parser-node';\nfor await (let item of createOSMStream('path-to-file.osm.pbf'))\n    console.log(item);\n```\nThe output will look like the following.\n\nThe OSM Header block:\n```javascript\n{\n  bbox: { left: -95159650000, right: -74309980000, top: 57508260000, bottom: 41637700000 },\n  required_features: [ 'OsmSchema-V0.6', 'DenseNodes' ],\n  optional_features: [],\n  writingprogram: 'osmium/1.14.0',\n  source: '',\n  osmosis_replication_timestamp: 1658434914,\n  osmosis_replication_sequence_number: 3403,\n  osmosis_replication_base_url: 'http://download.geofabrik.de/north-america/canada/ontario-updates'\n}\n```\nFor every node:\n```javascript\n{\n    \"type\":\"node\",\n    \"id\":3406566,\n    \"lat\":35.3320358,\n    \"lon\":33.330649,\n    \"tags\": {\n        \"highway\":\"traffic_signals\",\n        \"is_in\":\"Ozanköy; Girne; Kuzey Kıbrıs Türk Cumhuriyeti\"\n    },\n    \"info\": {\n        \"version\":30,\n        \"timestamp\":\"2021-11-29T20:15:45Z\",\n        \"changeset\":nnnnn,\n        \"uid\":nnnnn,\n        \"user\":\"xxxxx\"\n    }\n}\n```\n\nFor every way:\n```javascript\n{\n    \"type\":\"way\",\n    \"id\":3990794,\n    \"refs\":[20883417,5028923737,5028923736, ...],\n    \"tags\":{\n        \"man_made\":\"breakwater\"\n    },\n    \"info\":{ ... }\n}\n```\n\nFor every relation:\n```javascript\n{\n    \"type\":\"relation\",\n    \"id\":69898,\n    \"members\":[\n        {\"type\":\"node\",\"ref\":268094888,\"role\":\"via\"},\n        {\"type\":\"way\",\"ref\":24665096,\"role\":\"to\"},\n        {\"type\":\"way\",\"ref\":708939399,\"role\":\"from\"}\n    ],\n    \"tags\":{\n        \"restriction\":\"no_right_turn\",\"type\":\"restriction\"\n    }\n    \"info\":{ ... }\n}\n```\nThe properties `tags` and `info` are optional.\nThe `info` fields are similar for all three object types.\nAlso, according to https://download.geofabrik.de/technical.html, the metadata\nfields `info.user`, `info.uid` and `info.changeset`\nare removed from public osm.pbf downloads since May 2018.\n\n## Installation\n\n```bash\nnpm install osm-pbf-parser-node\n```\n\n## Usage\n\nThe module exports an async generator function:\n```javascript\nexport async function* createOSMStream(file: string, opts?: OSMOptions):\n    AsyncGenerator\u003cobject, void, unknown\u003e;\n```\nThe arguments are path to the input file in the osm.pbf format\nand an object with the following properties:\n\n* `withTags` - whether to include (and which) tags into the output.\nCan be a boolean, or an object {node: _what_, way: _what_, relation: _what_},\nwhere each _what_ is in turn either `true` (the default) or `false` or an array\nof tag keys to include. In the latter case all other tags are\nnot included, so `withTags.node == []` is the same as `withTags.node = false`.\n\n* `withInfo` - whether to include metadata information into output.\n\n* `writeRaw` - if `true`, send raw OSMData block to the output, see an example below.\n\nThe defaults are:\n```javascript\n{ withTags: true, withInfo: false, writeRaw: false }\n```\n\nThe module also exports the OSMTransform class:\n```javascript\nimport { Transform } from 'node:stream';\nexport class OSMTransform extends Transform {\n    constructor(osmopts?: OSMOptions, opts?: TransformOptions);\n}\n```\nThis class can be used in a chain of pipes like this:\n```javascript\nnew Promise(resolve =\u003e {\n    createReadStream('path-to-file.osm.pbf')\n        .pipe(new OSMTransform(osmopts))\n        .pipe(consume)\n        .on('finish', resolve);\n});\n\n```\nwhere `consume` is the next Writable. For example,\nthe following code just prints out all received objects:\n```javascript\nconst consume = new Transform.PassThrough({\n    objectMode: true,\n    transform: (items, enc, next) =\u003e {\n        for (let item of items)\n            console.log(item);\n        next();\n    }\n});\n```\nNote that the Writable side always receives arrays of items.\nThe length of such arrays can vary from 1 (e.g. for OSMHeader)\nto several thousands. The order of items in the output is always\nthe same as the order in the input stream.\n\nThe following example shows how to use OSMTransform for reading directly from an URL:\n```javascript\nimport { get as http_get } from 'node:http';\nnew Promise((resolve, reject) =\u003e {\n    http_get(url, res =\u003e {\n        if (res.statusCode != 200) {\n            console.log(`got status code ${res.statusCode} ${res.statusMessage}`);\n            return reject('request failed');\n        }\n        res.pipe(new OSMTransform(osmopts))\n            .pipe(consume)\n            .on('finish', resolve);\n    });\n});\n```\nSee file `test.js` for a complete example.\n\n## Raw output\n\nIf `writeRaw` is `true`, OSMTransform pushes compressed OSMData blocks\ninto output. In this case the next Writable in the pipeline should\ninflate the data blocks and call `parse` to convert them into an array\nof nodes, etc. The package export this function as:\n```javascript\nexport function parse(osmdata: Buffer, options: OSMTransform|OSMOptions): Array\u003cobject\u003e;\n```\n\n For example:\n```javascript\nnew Promise(resolve =\u003e {\n    createReadStream(file)\n        .pipe(new OSMTransform({writeRaw: true}))\n        .pipe(rawWritable)\n        .on('finish', resolve)\n        .on('error', e =\u003e console.error(e));\n});\n```\nwhere the RawWritable class does the job:\n```javascript\nconst rawWritable = new Writable({\n    objectMode: true,\n    write(chunk, enc, next) {\n        if (chunk instanceof Buffer) {\n            let buf = inflateSync(chunk);\n            let batch = parse(buf, {withTags: true, withInfo: false});\n            // ... do something with batch\n        } else\n            // chunk[0] contains OSM Header\n        next();\n    }\n});\n```\n\n## Performance\n\nThe script `test.js` does nothing but counts nodes, ways and relations\nin the input stream. Here is the speed of parsing canada-latest.osm.pbf\nas of Nov. 2022, about 2.75 GB in size, using OSMTransform with\nwithTags=true, withInfo=false:\n\n* on ASUS StudioBook (i7-9750H, DDR4-2666): 2m50s, about 2.37 millions items per second.\n\n* on Intel NUC-12 (i9-12900, DDR4-3200, NVMe SSD): 1m55s, about 3.5\nmillions items per second.\n\nFor some reason parsing of really big files is slower. Parsing 67GB of\nplanet-latest.osm.pbf took 1h22m (1.8 millions items per second) on NUC-12.\n\nThe speed of createOSMStream is about 1.6 times lower, apparently because it executes\n`yield` millions of times.\n\n\n## Limitations\n\nThe OSMData blocks are supposed to be inflatable by `inflate` from `node:zlib`,\nthe compressed data in `zlib_data`. Other compression methods are not\nimplemented.\n\n## Notes\n\nThis module uses the synchronous `inflateSync` from `node:zlib`.\nThe asynchronous `inflate` may result in a better speed, but\nI haven't seen more that 10% faster. On the other hand it\nuses considerably more memory. To my opinion, using the `writeRaw` mode\nand worker threads on the Writable side leads to much better results.\n\nThe proto files have been updated from\nhttps://github.com/openstreetmap/OSM-binary/tree/master/osmpbf\nand compiled by the [Mapbox pbf compiler](https://github.com/mapbox/pbf).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fborisgontar%2Fosm-pbf-parser-node","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fborisgontar%2Fosm-pbf-parser-node","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fborisgontar%2Fosm-pbf-parser-node/lists"}