{"id":15203185,"url":"https://github.com/woltapp/spark-osm-datasource","last_synced_at":"2025-10-29T03:30:58.533Z","repository":{"id":48922950,"uuid":"217699985","full_name":"woltapp/spark-osm-datasource","owner":"woltapp","description":"Native Spark OSM PBF data source","archived":false,"fork":false,"pushed_at":"2024-04-03T11:22:20.000Z","size":69,"stargazers_count":18,"open_issues_count":2,"forks_count":6,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-02-01T21:12:11.971Z","etag":null,"topics":["osm","pbf","spark"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/woltapp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-26T11:38:30.000Z","updated_at":"2024-10-23T08:56:27.000Z","dependencies_parsed_at":"2024-09-24T05:30:53.687Z","dependency_job_id":"fbe5eec1-cbca-462b-91e1-cf84df292d6b","html_url":"https://github.com/woltapp/spark-osm-datasource","commit_stats":{"total_commits":33,"total_committers":2,"mean_commits":16.5,"dds":"0.030303030303030276","last_synced_commit":"6cc34a40ccfc12c23a73bc6bb4e07f4c9969b6a6"},"previous_names":["akashihi/spark-osm-datasource"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/woltapp%2Fspark-osm-datasource","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/woltapp%2Fspark-osm-datasource/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/woltapp%2Fspark-osm-datasource/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/woltapp%2Fspark-osm-datasource/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/woltapp","download_url":"https://codeload.github.com/woltapp/spark-osm-datasource/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238762726,"owners_count":19526460,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["osm","pbf","spark"],"created_at":"2024-09-28T04:41:51.921Z","updated_at":"2025-10-29T03:30:53.210Z","avatar_url":"https://github.com/woltapp.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark OSM PBF datasource\n\n[OSM PBF format](https://wiki.openstreetmap.org/wiki/PBF_Format) spark datasource.\n\n## Rationale\n\nOpenStreetMap is a literally planet scale database storing features found on a Earth surfaces. Commonly used OSM processing software\neither uses single host processing or converts and stores OSM data to some database engine and uses it as a source of truth for distributed\nprocessing. With Spark abilities to automatically distribute huge dataset over hundreds of nodes and process more data, that a\nsingle node could fit, it sounds pretty logical to implement native Spark OSM PBF data source  \n## Download\n\nSupported scala versions are 2.11 and 2.12\n\n### Maven \n        \n         \u003cdependency\u003e\n                \u003cgroupId\u003ecom.wolt.osm\u003c/groupId\u003e\n                \u003cartifactId\u003espark-osm-datasource_2.11\u003c/artifactId\u003e\n                \u003cversion\u003e0.3.0\u003c/version\u003e\n         \u003c/dependency\u003e\n\n         \u003cdependency\u003e\n                \u003cgroupId\u003ecom.wolt.osm\u003c/groupId\u003e\n                \u003cartifactId\u003espark-osm-datasource_2.12\u003c/artifactId\u003e\n                \u003cversion\u003e0.3.0\u003c/version\u003e\n         \u003c/dependency\u003e\n        \n### Gradle\n\n         compile group: 'com.wolt.osm', name: 'spark-osm-datasource_2.11', version: '0.3.0'\n         compile group: 'com.wolt.osm', name: 'spark-osm-datasource_2.12', version: '0.3.0'\n        \n### SBT \n                        \n        libraryDependencies += \"com.wolt.osm\" % \"spark-osm-datasource\" % \"0.3.0\"\n        \n### Github release\n\n        https://github.com/woltapp/spark-osm-datasource/releases/tag/v0.31.0\n        \n## Usage                \n        \nOSM PBF data source uses [parallel OSM PBF parser](https://github.com/woltapp/parallelpbf) internally, therefore it needs to provide\nit with a input stream. OSM DataSource internally uses HDFS driver, so it will accept any HDFS compatible path, like local\nfile, http link, s3 link etc and convert it to the inputstream. The stream will be opened and read as many times, as many\npartitions are requested:\n\n    val spark = SparkSession\n      .builder()\n      .appName(\"OsmReader\")\n      .config(\"spark.master\", \"local[4]\")\n      .getOrCreate()\n\n    val osm = spark.read.option(\"partitions\", 256).format(OsmSource.OSM_SOURCE_NAME).load(\"s3://maps/extract-cesko-brno.osm.pbf\").drop(\"INFO\").persist(StorageLevel.MEMORY_AND_DISK)\n    \nThere are two options for the reader:\n\n* `partitions` - Number of partitions to split OSM PBF file. Pay attentions, that full planet file is about 1.2TB of uncompressed data, so plan\nyour partitioning accordingly.\n* `threads` - Number of threads to use by [parallel OSM PBF parser](https://github.com/woltapp/parallelpbf). Keep it\n`1` for a local deployments or set to number of cores per worker node.\n* `useLocalFile` - Enables Spark file distribution mechanics, see below.\n\n### Using with Spark file distirbution \n\nAnother one option is to use Sparks ability to cache file on the nodes and read it locally. With that approach under some circumstances (repeated re-reading\nof same set of big files in different steps) a better performance can be achieved.\n\n        spark.sparkContext.addFile(\"s3://maps/extract-cesko-brno.osm.pbf\")\n        val osm = spark.read.option(\"partitions\", 256).option(\"useLocalFile\", \"true\").format(OsmSource.OSM_SOURCE_NAME).load(\"extract-cesko-brno.osm.pbf\")\n        \nPlease keep in mind, that you should specify just a file name as the `load` parameter, not a full path.\n        \n## Schema\n\nDue to Spark processing paradigm, all types of the OSM entities are mapped to the same dataframe of the following structure:\n\n    StructField(\"ID\", LongType, nullable = false),\n    StructField(\"TAG\", MapType(StringType, StringType, valueContainsNull = false), nullable = false),\n    StructField(\"INFO\", StructType(    StructField(\"UID\", IntegerType, nullable = true),\n                                       StructField(\"USERNAME\", StringType, nullable = true),\n                                       StructField(\"VERSION\", IntegerType, nullable = true),\n                                       StructField(\"TIMESTAMP\", LongType, nullable = true),\n                                       StructField(\"CHANGESET\", LongType, nullable = true),\n                                       StructField(\"VISIBLE\", BooleanType, nullable = false)\n                                        ), nullable = true),\n    StructField(\"TYPE\", IntegerType, nullable = false),\n    StructField(\"LAT\", DoubleType, nullable = true),\n    StructField(\"LON\", DoubleType, nullable = true),\n    StructField(\"WAY\", ArrayType(LongType, containsNull = false), nullable = true),\n    StructField(\"RELATION\", ArrayType(StructType(    StructField(\"ID\", LongType, nullable = false),\n                                                     StructField(\"ROLE\", StringType, nullable = true),\n                                                     StructField(\"TYPE\", IntegerType, nullable = false)\n                                                 ), containsNull = false), nullable = true)\n\nOSM nodes will have `LAT` and `LON` fields set and `WAY` and `RELATION` fields will be null.\nOSM nodes will have `WAY` fields set and `LAT`/`LON` and `RELATION` fields will be null.\nOSM relations will have `RELATION` fields set and `LAT`/`LON` and `WAYS` fields will be null.\n\n`TYPE` field in both `RELATION` structure and in dataframe structure indicates type of the OSM entity. It is an\ninteger field with following values:\n\n* 0 - Item is a OSM Node entity\n* 1 - Item is a OSM Way entity\n* 2 - Item is a OSM Relation entity\n\nReader supports column pruning during read, so dropped columnts will not be loaded from the OSM PBF file. Even more,\nif both `LAT` and `LON` columns will be dropped, no node processing will be started at all. Same applies for\n`WAY` and `RELATION` columnn.\n\n### Warning on order instability\n\nOSM PBF file can be sorted and stored in a ordered way. Unfortunately, due to parallel nature of the Spark, that \nordering will be broken during parsing and several consequent dataframe loads may return data in a different order for \neach run. In case order is important for you, you can sort the dataframe after loading. \n                          |\n\n## Versioning\n\nWe use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://github.com/woltapp/spark-osm-reader/tags). \n\n## Authors\n\n* **Denis Chaplygin** - *Initial work* - [akashihi](https://github.com/akashihi)\n\n## License\n\nThis project is licensed under the GPLv3 License - see the LICENSE file for details.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwoltapp%2Fspark-osm-datasource","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwoltapp%2Fspark-osm-datasource","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwoltapp%2Fspark-osm-datasource/lists"}