{"id":15058941,"url":"https://github.com/bdon/osmexpress","last_synced_at":"2025-04-12T15:40:37.930Z","repository":{"id":46629408,"uuid":"205719348","full_name":"bdon/OSMExpress","owner":"bdon","description":"Fast database file format for OpenStreetMap","archived":false,"fork":false,"pushed_at":"2025-01-14T07:06:10.000Z","size":1083,"stargazers_count":253,"open_issues_count":9,"forks_count":21,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-10T19:37:57.291Z","etag":null,"topics":["database","openstreetmap","spatial"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bdon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"protomaps"}},"created_at":"2019-09-01T18:55:55.000Z","updated_at":"2025-03-15T19:21:31.000Z","dependencies_parsed_at":"2023-12-15T00:27:27.808Z","dependency_job_id":"d3ed9a79-deb3-4902-a8f4-52a0c21b48d5","html_url":"https://github.com/bdon/OSMExpress","commit_stats":{"total_commits":132,"total_committers":9,"mean_commits":"14.666666666666666","dds":"0.19696969696969702","last_synced_commit":"0a5672e98c662db3fac99029d509c8480a09ef93"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bdon%2FOSMExpress","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bdon%2FOSMExpress/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bdon%2FOSMExpress/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bdon%2FOSMExpress/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bdon","download_url":"https://codeload.github.com/bdon/OSMExpress/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248590771,"owners_count":21129887,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","openstreetmap","spatial"],"created_at":"2024-09-24T22:33:53.514Z","updated_at":"2025-04-12T15:40:37.908Z","avatar_url":"https://github.com/bdon.png","language":"C++","readme":"# OSM Express\n\n![Screenshot](examples/screenshot.png)\n\n[Manual](docs/MANUAL.md), [Programming Guide](docs/PROGRAMMING_GUIDE.md)\n\nOSM Express is a fast storage format for OpenStreetMap that powers [SliceOSM](https://github.com/SliceOSM). It's designed as a low level building block specific to the OSM data model; common access patterns such as random lookups by ID, in-place minutely updates, and spatial queries are efficient and simple to manage in production applications. \n\n## Features\n\n* **Random access:** Look up nodes, ways and relations and their metadata by ID; fetch member elements of ways and relations to construct geometries.\n* **Spatial indexing:** Nodes are bucketed into [S2 Geometry](http://s2geometry.io) cells. Access a region by providing a cell covering; works for nonrectangular regions.\n* **Scalable:** OSM Express works the same way for OSM data of any size, from a small city to the entire planet. The entire planet can be worked with efficiently on typical hardware such as a laptop computer.\n* **In-place updates:** Included are scripts to download minutely changesets from [planet.openstreetmap.org](https://planet.openstreetmap.org) and apply them to an .osmx database.\n* **Concurrent access:** Multiple processes can open the database file for reading simultaneously. No running server process is required. Writing minutely updates doesn't block reader access. Reads and writes are transactional. \n* **Portable:** An .osmx file can be read and written to from either C++ or Python.\n\n## Details\n\nOSM Express is a compact 1,500 LOC, and really a cobbling together of a few low-level libraries:\n\n* [Libosmium](https://osmcode.org/libosmium/index.html) for the reading and writing of .osm.pbf files.\n* [LMDB](https://symas.com/lmdb) for a memory-mapped ACID key-value store with fast cursor iteration.\n* [Cap'n Proto](https://capnproto.org) for in-memory and on-disk representation of OSM elements.\n* [CRoaring](https://roaringbitmap.org) for in-memory representation of ID sets as compressed bitmaps.\n* [S2 Geometry](http://s2geometry.io) for indexing of geographic coordinates.\n\n## Installation\n\n[See the manual for instructions on building from source](/docs/PROGRAMMING_GUIDE.md). \n\n## Usage\n\nOSM Express is being used in production for [SliceOSM](https://slice.openstreetmap.us) and the file format is stable.\n\n* Use the `osmx` command line tool to expand a .osm.pbf to an .osmx database and perform basic tasks such as extracting regions or querying by ID. No programming required.\n* Use the [Python library](python/) library via `pip install osmx` to access an .osmx database programatically. See the [Python Examples](python/examples) for how to create command line tools, webservers or detailed diffs based on minutely data. \n* Use the C++ library to access an .osmx database programatically. \n\n### Command line\n\n```bash\nosmx expand planet.osm.pbf planet.osmx # converts a pbf or xml to osmx. Takes 5-10 hours for the planet, resulting in a ~600GB file.\nosmx extract planet.osmx extract.osm.pbf --bbox 40.7411\\,-73.9937\\,40.7486\\,-73.9821 # extract a new pbf for the given bounding box.\nosmx update planet.osmx 3648548.osc 3648548 2019-08-29T17:50:02Z --commit # applies an OsmChange diff.\nosmx query planet.osmx # Print statistics, seqnum and timestamp.\nosmx query planet.osmx way 34633854 # look up an element by ID.\n```\n\n`osmx extract` has a flag `--noUserData` intended for public facing instances which will remove the user, uid and changeset fields to comply with [GDPR guidelines](https://wiki.openstreetmap.org/wiki/GDPR).\n\nDetailed command line usage can be found in the [Manual](docs/MANUAL.md).\n\n### Headers\n\nThe C++ API is very rough with minimal abstraction. [examples/way_wkt.cpp](examples/way_wkt.cpp) is a short, commented C++ program that uses the headers to read a way from a .osmx file and outputs its [Well-Known Text](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry) LineString geometry.\n\n```bash\n./way_wkt ../ny.osmx 34633854\nEmpire State Building\tLINESTRING (-73.9864855 40.7484833,-73.9851554 40.7479226,-73.9848259 40.7483735,-73.9861526 40.7489422,-73.9863111 40.7487242,-73.9863282 40.7487007,-73.9864684 40.7485078,-73.9864855 40.7484833)\n```\n\n[examples/bbox_wkt.cpp](examples/bbox_wkt.cpp) is a more complex example that takes a bounding box as input, and returns WKT LineStrings for ways that overlap the bbox. This overlap is an approximation based on cells and may include ways outside the bounding box.\n\nDetailed C++ usage can be found in the [Programming Guide](docs/PROGRAMMING_GUIDE.md).\n\n### Docker (experimental)\n\nA `Dockerfile` is provided but users will need to build their own container. To do so, run:\n```\ndocker build -t osmx .\n```\n\n## License and Development\n\n2-Clause BSD, see [LICENSE.md](LICENSE.md).","funding_links":["https://github.com/sponsors/protomaps"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbdon%2Fosmexpress","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbdon%2Fosmexpress","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbdon%2Fosmexpress/lists"}