{"id":29369981,"url":"https://github.com/codesoap/pbf-reblob","last_synced_at":"2025-07-26T12:03:25.665Z","repository":{"id":259064801,"uuid":"876234265","full_name":"codesoap/pbf-reblob","owner":"codesoap","description":"Reduce the size of OSM's PBF files by increasing blob sizes","archived":false,"fork":false,"pushed_at":"2024-10-21T16:49:07.000Z","size":38,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-17T09:11:22.148Z","etag":null,"topics":["compression","open-street-map","osm","pbf"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codesoap.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-21T16:12:09.000Z","updated_at":"2024-10-21T16:49:10.000Z","dependencies_parsed_at":"2024-10-22T06:04:23.881Z","dependency_job_id":null,"html_url":"https://github.com/codesoap/pbf-reblob","commit_stats":null,"previous_names":["codesoap/pbf-reblob"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/codesoap/pbf-reblob","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codesoap%2Fpbf-reblob","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codesoap%2Fpbf-reblob/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codesoap%2Fpbf-reblob/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codesoap%2Fpbf-reblob/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codesoap","download_url":"https://codeload.github.com/codesoap/pbf-reblob/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codesoap%2Fpbf-reblob/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264465641,"owners_count":23612576,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","open-street-map","osm","pbf"],"created_at":"2025-07-09T13:01:38.129Z","updated_at":"2025-07-09T13:05:21.793Z","avatar_url":"https://github.com/codesoap.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"pbf-reblob allows you to easily increase the blob size within open\nstreetmap's PBF files. This reduces the size of PBF files. It works best\nwith small extracts.\n\n# Demo\nHere are the results of file sizes reduced with pbf-reblob; zstd\ncompression seems to work better than the default zlib compression with\nlarger blobs:\n\n![graph of achieved compression ratios](https://github.com/codesoap/pbf-reblob/releases/download/v0.1.0/compare.png)\n\n# Installation\nYou can download the tool from the releases page:\nhttps://github.com/codesoap/pbf-reblob/releases\n\nIf you have the Go toolchain installed and prefer to build pbf-reblob\nyourself, you can get it by running this:\n\n```bash\ngo install github.com/codesoap/pbf-reblob@latest\n# The binary is now at ~/go/bin/pbf-reblob.\n```\n\n# Usage\n```console\n$ # Change blob size to roughly 16MiB (while not surpassing this blob size):\n$ pbf-reblob serbia-latest.osm.pbf serbia-latest-16M.osm.pbf\n\n$ # Change blob size to roughly 32MiB, with zstd compression:\n$ pbf-reblob -s 32M -c zstd serbia-latest.osm.pbf serbia-latest-32M.zstd.osm.pbf\n\n$ # Let's see the results:\n$ du -h serbia-latest*\n190M    serbia-latest-16M.osm.pbf\n186M    serbia-latest-32M.zstd.osm.pbf\n194M    serbia-latest.osm.pbf\n\n$ pbf-reblob -h\nUsage:\n  pbf-reblob [-v] [-s \u003csize\u003e] [-c \u003ccompression\u003e] \u003cIN_FILE\u003e \u003cOUT_FILE\u003e\nOptions:\n  -c string\n        output compression; either 'raw', 'zlib' or 'zstd' (default \"zlib\")\n  -s string\n        uncompressed blob size limit; suffixes 'k' and 'M' allowed (default \"16M\")\n  -v    verbose\n```\n\n# How It Works\nPBF files contain numerous blobs of OSM entities. The popular tool\n[osmium](https://osmcode.org/osmium-tool/) usually puts one group of\n~8000 OSM entities into each blob. However the file format allows for\nuncompressed blob sizes of up to 32MiB, while recommending sizes up to\n16MiB. These sizes are not reached with ~8000 OSM entities.\n\nBy moving multiple entity groups into a single blob, the amount of blobs\nin a PBF file can be significantly reduced. This reduces file size, because\neach blob contains a list of used strings for the entities in the blob. If\nthere are multiple small blobs, the same strings will often be stored\nmultiple times (once for each block that uses it). By reducing the\namount of blobs, the amount of duplicate strings can be reduced.\n\nThis seems to be most effective with small PBF files. I assume this is\nbecause within a smaller area, there is a higher chance for the same\nstrings to be reused.\n\n# Side Effects\nWhile no data is lost with this method of compression, the changed blob\nsize might affect the tools working with PBF files. Most prominently,\nmany tools will likely use more memory when working with larger blobs.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodesoap%2Fpbf-reblob","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodesoap%2Fpbf-reblob","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodesoap%2Fpbf-reblob/lists"}