{"id":20455432,"url":"https://github.com/simplexspatial/simplexspatial-data-distribution-analysis","last_synced_at":"2026-05-10T15:43:04.004Z","repository":{"id":144563154,"uuid":"262382548","full_name":"simplexspatial/simplexspatial-data-distribution-analysis","owner":"simplexspatial","description":"Analisys of data distribution of OSM dataset.","archived":false,"fork":false,"pushed_at":"2020-05-11T15:33:00.000Z","size":568,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-15T23:28:22.422Z","etag":null,"topics":["osm4scala","scala","spark"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simplexspatial.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"code_of_conduct.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-08T17:10:40.000Z","updated_at":"2020-05-11T15:33:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"4c20bf10-1904-4f38-b2e3-efd7a6e10e58","html_url":"https://github.com/simplexspatial/simplexspatial-data-distribution-analysis","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simplexspatial%2Fsimplexspatial-data-distribution-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simplexspatial%2Fsimplexspatial-data-distribution-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simplexspatial%2Fsimplexspatial-data-distribution-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simplexspatial%2Fsimplexspatial-data-distribution-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simplexspatial","download_url":"https://codeload.github.com/simplexspatial/simplexspatial-data-distribution-analysis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242006161,"owners_count":20056515,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["osm4scala","scala","spark"],"created_at":"2024-11-15T11:18:52.243Z","updated_at":"2025-12-02T15:04:57.021Z","avatar_url":"https://github.com/simplexspatial.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-v2.0%20adopted-ff69b4.svg)](code_of_conduct.md)\n\n# Data Distribution.\n\nThis repository contains different data analysis of data distribution in the OSM dataset.\n\n## Spark locally\nIf you don't have access to a Spark cluster, it is possible to execute it locally.\nA laptop with 16Gb memory and 8 cores should be enough.\nIn my case, I'm using a Desktop with 16cores and 32Gb RAM. Full specs at the very bottom.\n\nTo start Spark in local mode, after download and uncompress:\n```shell script\nsbin/start-all.sh\n```\n\nTo access to the UI: [http://localhost:8080/](http://localhost:8080/)\n\nTo stop Spark in local mode:\n```shell script\nsbin/stop-all.sh\n```\n\n\n## Extract blocks\nTo be able to parallelize, lets extract all blocks. Full universe will take 4 minutes:\n```shell script\nspark-submit \\\n  --class com.simplexportal.simplexspatial.analysis.Driver \\\n  --master \"local[*]\" \\\n  target/scala-2.11/simplexspatial-data-distribution-analysis-assembly-0.1.jar \\\n  extract \\\n  -i file:///home/angelcc/Downloads/osm/planet/planet-200309.osm.pbf \\\n  -o file:///home/angelcc/Downloads/osm/planet/blobs\n```\n\n## Node IDs distribution\nFollowing, example of how to report for 100 \"partitions\", locally, using 5 cores and 4Gb per core.\nIt will take around 30 minutes.\n\n```shell script\n/home/angelcc/apps/spark-2.4.5-bin-hadoop2.7/bin/spark-submit \\\n  --class com.simplexportal.simplexspatial.analysis.Driver \\\n  --master \"spark://angelcc-B450-AORUS-ELITE:7077\" \\\n  --deploy-mode cluster \\\n  --executor-memory 4G \\\n  --total-executor-cores 5 \\\n  --num-executors 1 \\\n  target/scala-2.11/simplexspatial-data-distribution-analysis-assembly-0.1.jar \\\n  mod \\\n  -p 100 \\\n  -i file:///home/angelcc/Downloads/osm/planet/blobs \\\n  -o file:///home/angelcc/Downloads/osm/planet/distribution/nodeId/100\n```\n\n## Tile distribution\nFollowing, example of how to distribution report for tiles of 10000x10000, locally,\nusing 5 cores and 4Gb per core. It will take around 30 minutes.\n\n```shell script\n/home/angelcc/apps/spark-2.4.5-bin-hadoop2.7/bin/spark-submit \\\n  --class com.simplexportal.simplexspatial.analysis.Driver \\\n  --master \"spark://angelcc-B450-AORUS-ELITE:7077\" \\\n  --deploy-mode cluster \\\n  --executor-memory 4G \\\n  --total-executor-cores 5 \\\n  --num-executors 1 \\\n  target/scala-2.11/simplexspatial-data-distribution-analysis-assembly-0.1.jar \\\n  tile \\\n  --latPartitions 10000 \\\n  --lonPartitions 10000 \\\n  -i file:///home/angelcc/Downloads/osm/planet/blobs \\\n  -o file:///home/angelcc/Downloads/osm/planet/distribution/tile/10000x10000\n```\n\n## Zeppelin\nTo start the notebook, from a temporal folder:\n```shell script\nmkdir logs notebook\ndocker run -p 8081:8080 --rm \\\n   -v $PWD/logs:/logs \\\n   -v $PWD/notebook:/notebook \\\n   -v /home/angelcc/Downloads/osm/planet/distribution/nodeId/100:/zeppelin/data/nodeId \\\n   -v /home/angelcc/Downloads/osm/planet/distribution/tile/10000x10000:/zeppelin/data/tile \\\n   -e ZEPPELIN_LOG_DIR='/logs' \\\n   -e ZEPPELIN_NOTEBOOK_DIR='/notebook' \\\n   --name zeppelin \\\n   apache/zeppelin:0.9.0\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimplexspatial%2Fsimplexspatial-data-distribution-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimplexspatial%2Fsimplexspatial-data-distribution-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimplexspatial%2Fsimplexspatial-data-distribution-analysis/lists"}