{"id":23505064,"url":"https://github.com/woltapp/parallelpbf","last_synced_at":"2025-04-15T23:50:44.121Z","repository":{"id":39002461,"uuid":"216575743","full_name":"woltapp/parallelpbf","owner":"woltapp","description":"OpenStreetMap PBF format multithreaded reader","archived":false,"fork":false,"pushed_at":"2023-12-05T22:24:42.000Z","size":237,"stargazers_count":58,"open_issues_count":5,"forks_count":17,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-29T02:51:31.946Z","etag":null,"topics":["openstreetmap","pbf"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/woltapp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-10-21T13:31:00.000Z","updated_at":"2025-03-09T21:47:05.000Z","dependencies_parsed_at":"2023-01-19T06:00:46.174Z","dependency_job_id":null,"html_url":"https://github.com/woltapp/parallelpbf","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/woltapp%2Fparallelpbf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/woltapp%2Fparallelpbf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/woltapp%2Fparallelpbf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/woltapp%2Fparallelpbf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/woltapp","download_url":"https://codeload.github.com/woltapp/parallelpbf/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249173061,"owners_count":21224481,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["openstreetmap","pbf"],"created_at":"2024-12-25T09:18:14.223Z","updated_at":"2025-04-15T23:50:44.103Z","avatar_url":"https://github.com/woltapp.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Parallel OSM PBF parser\n\n[OSM PBF format](https://wiki.openstreetmap.org/wiki/PBF_Format) multithreaded reader/writer written in Java. Supports all \ncurrent OSM PBF features and options (only for reading)\n\n## Rationale\n\n\nThe OSMPBF format consists of sequence of independent blobs, containing actual OSM data. All existing Java readers\nof OSMPBF format read that file sequentially, processing each blob one by one using just a single thread. \nParsing single blob usually involves decompressing it and calculating OSM entity values from a delta-packed\ndata ([check wiki](https://wiki.openstreetmap.org/wiki/PBF_Format) for details). Obviously it is more\nCPU bound task, than IO bound task, so loading CPU up should speed up the processing. The Simplest way to do that\nis to distribute the work on all the cores. And here we go...\n\n## Download\n\n### Maven \n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.wolt.osm\u003c/groupId\u003e\n    \u003cartifactId\u003eparallelpbf\u003c/artifactId\u003e\n    \u003cversion\u003e0.3.1\u003c/version\u003e\n\u003c/dependency\u003e\n```\n        \n### Gradle\n\n```gradle\ncompile group: 'com.wolt.osm', name: 'parallelpbf', version: '0.3.1'\n```\n        \n### SBT \n\n```sbt\nlibraryDependencies += \"com.wolt.osm\" % \"parallelpbf\" % \"0.3.1\"\n```\n        \n### GitHub release\n\n→ https://github.com/woltapp/parallelpbf/releases/tag/v0.3.1\n        \n## Reading                \n        \nAs parsing is asynchronous, it heavily relies on the callbacks. There are 7 callbacks defined:\n\n* `Consumer\u003cNode\u003e onNode` - is called for each Node in the OSM PBF file. This callback must be reenterable as it will be \ncalled simultaneously from the different parallel executing threads.\n\n* `Consumer\u003cWay\u003e onWay` - is called for each Way in the OSM PBF file. This callback must be reenterable as it will be \ncalled simultaneously from the different parallel executing threads.\n\n* `Consumer\u003cRelation\u003e onRelation` - is called for each Relation in the OSM PBF file. This callback must be reenterable as it will be \ncalled simultaneously from the different parallel executing threads.\n\n* `Consumer\u003cChangeset\u003e onChangeSet` - is called for each ChangeSet in the OSM PBF file. This callback must be reenterable as it will be \ncalled simultaneously from the different parallel executing threads.\n\n* `Consumer\u003cHeader\u003e onHeader` - is called for the Header object of the OSM PBF file. Each OSM PBF file should have just a single Header object,\nso it is safe to assume, that this callback will be called just once.\n\n* `Consumer\u003cBoundBox\u003e onBoundBox` - is called for the Bounding box object of the OSM PBF file. OSM file may not have BoundBox object written,\nso this callback may never be called. As BoundBox is a part of OSMPBF Header object, it is safe to assume, that this callback will be called just once.\n\n* `Runnable onComplete` - called only in case of successful parse completion. All the other callbacks are guaranteed \nto finish before calling `onComplete` and no other callbacks will happen after `onComplete` call.\n\nCallbacks can be attached to the parser using appropriate calls:\n\n```java\nInputStream input = Thread.currentThread().getContextClassLoader()\n        .getResourceAsStream(\"sample.pbf\");\n\nnew ParallelBinaryParser(input, 1)\n        .onHeader(this::processHeader)\n        .onBoundBox(this::processBoundingBox)\n        .onComplete(this::printOnCompletions)\n        .onNode(this::processNodes)\n        .onWay(this::processWays)\n        .onRelation(this::processRelations)\n        .onChangeset(this::processChangesets)\n        .parse();\n```\n\nAll callbacks are optional, if you do not set some callback, nothing will break. Parsing of data for missing callback \nwill be skipped. So, for example, if you need just relations data, you should not set other callbacks and data blocks carrying\nother types of OSM data will be skipped completely, thus saving processing time. \nThere is an exception from that rule - Header data block is always parsed, even if no callback is set.\nEven more, if no Node/Way/Relation/Changeset callbacks will be set,  actual processing of data will be skipped \nafter finding first Header block. \n\n`ParallelBinaryParser` constructor accepts two mandatory arguments:\n\n* `InputStream input` - InputStream pointing to the beginning of the OSMPBF data. \n* `int threads` - Number of threads for parallel processing. Parser will automatically throttle and stop input \nreading if all threads are busy. Each thread keeps blob data in memory, so memory usage will be at least\n64MB per thread, but probably couple of hundreds megabytes per thread, depending on a block content.\n\nThere are also two optional arguments for partitioning support:\n\n* `noPartitions` - Total number of partitions processed file should be divided.\n* `myShard` - Number of partition, associated with the this instance of the parser.\n\nThe partitioning is added to support multi-host parallel loading, when each hosts reads it's own amount of data \nindependently and then somehow combines that data or continues processing it on each hosts separately. The whole idea\nof partitioning here is that we split up the file to some number of partitions or shard and only process OSMData blocks  \nfrom our 'own' shard, skipping all data blocks belonging to the other shard. Even with partitioning enabled, the whole\nInputStream will be processed and all OSMHeader blocks will be read and analyzed.\n\nTo start actually processing the input stream, you should call `parse()` function. It will create all required threads\nand start data reading from the input and parsing it. That function is intentionally blocking, but it is safe to \nwrap it to some other thread and wait for completion using `onComplete` callback.  \n\n### Warning on order instability\n\nOSM PBF file can be sorted and stored in a ordered way. Unfortunately, due to parallel nature of the parser, that \nordering will be broken during parsing and several consequent parse runs may return data in a different order for \neach run. In case order is important for you, you can either sort after parse or switch back to the single threaded\nparsers. \n\n\n### Performance comparision\n\n| Region         | Size in GB | Single thread read time in seconds | 24 threads read time in seconds |\n|----------------|------------|------------------------------------|---------------------------------|\n| Czech republic |  0.7       |  133                               | 40                              |\n| Asia           |  7.3       |  2381                              | 405                             |\n| Europe         |  21        |  3545                              | 953                             |\n| Planet         |  47        |  8204                              | 3203                            |\n\n## Writing\n\nWrite API differs from the Reading API, as it makes no sense to use callbacks here. The writer object provides three\nmethods to start writing, feed the writer with data and close writer. Writing function itself is thread-safe and reenterable,\nso can be used from parallel threads.\n\nSo the correct workflow will be:\n\n```java\nwriter = new ParallelBinaryWriter(output,1, bbox);\nwriter.start();\nwriter.write(node);\nwriter.close();\n```\n\n`ParallelBinaryWriter` accepts two mandatory arguments and one optional:\n\n* `OutputStream output` - OutputStream that will hold OSM PBF data.\n* `int threads` - Number of threads for parallel processing. Writer will automatically throttle and block on `.write()` call if all threads are busy. Each thread keeps blob data in memory, so memory usage will be at least 16MB per thread, but may be more, depending on block content.\n* `BoundBox boundBox` - Optional BoundBox of the data to be written, can be `null`\n\nOSM PBF header will be written to the OutputStream during construction.\n\n`.start()` call actually spawns writing threads and allows to make `.write()` calls. The `.start()` call is not thread-safe.\n\n`.write(OsmEntity)` call sends specified entity to one of the writing threads. This call is thread safe and calling it in parallel\nis recommended. In case of writing threads overload, the `.write()` call will block and wait for an empty writing thread to handle request.\n\n`.close()` will flush block to the output stream and terminate writing threads. Writer should not be used after calling `.close()`\non it. \n\n## Versioning\n\nWe use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://github.com/akashihi/parallelpbf/tags). \n\n## Authors\n\n* **Denis Chaplygin** - *Initial work* - [akashihi](https://github.com/akashihi)\n* **Scott Crosby** - *.proto definition files* - [scrosby](https://github.com/openstreetmap/OSM-binary) \n\n## License\n\nThis project is licensed under the GPLv3 License - see the LICENSE file for details.\nThe .proto definition files are licensed under the MIT license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwoltapp%2Fparallelpbf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwoltapp%2Fparallelpbf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwoltapp%2Fparallelpbf/lists"}