{"id":13548166,"url":"https://github.com/osmlab/osm-wayback","last_synced_at":"2025-04-02T20:31:55.843Z","repository":{"id":137437209,"uuid":"125133698","full_name":"osmlab/osm-wayback","owner":"osmlab","description":"Embedding OSM object histories from planet-history into GeoJSON objects for historical OSM-QA-Tiles with libosmium and RocksDB","archived":false,"fork":false,"pushed_at":"2018-12-12T03:57:13.000Z","size":27713,"stargazers_count":10,"open_issues_count":8,"forks_count":1,"subscribers_count":33,"default_branch":"master","last_synced_at":"2024-11-03T16:38:11.214Z","etag":null,"topics":["geojson","openstreetmap","openstreetmap-data","osmium","rocksdb"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/osmlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-03-14T00:48:37.000Z","updated_at":"2022-10-16T23:56:27.000Z","dependencies_parsed_at":"2023-07-21T08:17:22.881Z","dependency_job_id":null,"html_url":"https://github.com/osmlab/osm-wayback","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osmlab%2Fosm-wayback","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osmlab%2Fosm-wayback/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osmlab%2Fosm-wayback/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osmlab%2Fosm-wayback/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/osmlab","download_url":"https://codeload.github.com/osmlab/osm-wayback/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246888199,"owners_count":20850210,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["geojson","openstreetmap","openstreetmap-data","osmium","rocksdb"],"created_at":"2024-08-01T12:01:06.607Z","updated_at":"2025-04-02T20:31:50.834Z","avatar_url":"https://github.com/osmlab.png","language":"C++","funding_links":[],"categories":["C++","others"],"sub_categories":[],"readme":"# osm-wayback\n\u003c!-- [![Build Status](https://travis-ci.org/mapbox/osm-wayback.svg?branch=master)](https://travis-ci.org/mapbox/osm-tag-history) --\u003e\n\nCreates a [RocksDB](//rocksdb.org) key-value store of each version of OSM objects found in OSM history files. This history index can then be used to augment GeoJSON files of OSM objects to add a `@history` property that includes a record of all _previous_ edits.\n\nosm-wayback is currently designed to support large(ish)-scale historical analysis of OpenStreetMap edits, specifically focused on how objects change overtime (and who is editing them).\n\n#### Current Development Notes:\n\n1. The history is index is keyed by `osm-id`+\"!\"+`version` (with separate column families for nodes, ways, and relations).\n\n2. `add_history` will lookup every previous version of an object passed into it. If an object is passed in at version 3, it will look up versions 1,2, and 3. This is necessary for the tag comparisons. In the event there exists a version 4 in the index, it will not be included because version 3 was fed into `add_history`.\n\n3. Since `add_history` is driven by a stream of (current, valid) GeoJSON objects, deleted objects are not yet supported.\n\n## Build\n\nInstall mason to manage dependencies\n\n\n\tgit submodule init\n\tgit submodule update\n\t\nThen build with `cmake`:\n\n\tmkdir build\n\tcd build\n\tcmake ..\n\tmake\n\t\nTo use the `run.sh` script, also run the following:\n\n\t.mason/mason install osmium 1.9.1\n\t.mason/mason link osmium 1.9.1\n\t\n\t.mason/mason install tippecanoe 1.31.0\n\t.mason/mason link tippecanoe 1.31.0\n\t\n\tcd geometry-reconstruction\n\tnpm install\n\n\n## Running\n\n#### A canned workflow: `run.sh`\nThe `run.sh` script automates all of the steps to turn OSM history files into historical vector tiles with only 2 inputs: `OSM_HISTORY_FILE` and `ROOT_FOR_OUTPUT`.\n\t\nFor example, to run generate historical vector tiles from the albany example file included in `example/history_of_albandy.osh.pbf`: \n\t\n\t$ ./run.sh example/history_of_albany.osh.pbf example/albany\n\nThis will create the following files in the `example` directory (in the following order): \n\n| File | Description|\n|------------------------------|------------------------------|\n| **albany.osm.pbf** | Latest version of (all) objects in `history_of_albany.osh.pbf`|\n| **albany.geojsonseq** | GeoJSON sequence of objects exported by _osmium export_ with the `example/osmiumconfig` configuration. _(Not ALL OSM objects, only what osmium understands)_| \n| **albany_INDEX** | The RocksDB Index of `history_of_albany.osh.pbf` |\n| **albany.history**| Each OSM object from `albany.geojsonseq` with an additional `@history` property that contains each previous (major) version (see [`HISTORICAL_SCHEMA.md`](https://github.com/osmlab/osm-wayback/blob/master/HISTORICAL_SCHEMA.md) for more on this schema) | \n| **albany.history.geometries** | Each feature from `albany.history` enriched with an additional `nodeLocations` attribute storing the location of every version of every node ever associated with each object.  |\n| **albany\\_historical\\_geometries_\u003cbr\u003etopojson.geojsonseq** | Each feature from `albany.history.geometries` with a TopoJSON encoded `@history` attribute that describes each historical version (including minor versions) with geometries|\n| **`albany_historical.mbtiles`** | Historical vector tiles rendered at zoom 15 for albany! |\n\n_Note that once run, each of these files are standalone and can be deleted in the order they are generated. Each file is used only as the input to the next function. This workflow is the result of each utility here relyong on standalone input. For example, you could build a North-America INDEX and then lookup history for just new\\_york.geojsonseq. Looking up node locations will always require a second pass after histories are built. Separating these files and steps adds negligible time cost and allows tag-only history analysis._\n\n\n#### The complete workflow\n\nFirst build up a historic lookup index.\n**Note:** For large files (Country / Planet), increase `ulimit` so that RocksDB can have many files open at once (\u003e4000 for the full planet history file).\n\n\tbuild_lookup_index INDEX_DIR OSM_HISTORY_FILE\n\nSecond, pass a stream of GeoJSON features as produced by [osmium-export](http://docs.osmcode.org/osmium/latest/osmium-export.html) to the `add_history` function\n\n\n\tcat features.geojsonseq | add_history INDEX_DIR\n\nThe output is a stream of augmented GeoJSON features with an additional `@history` array (see [HISTORICAL_SCHEMA.md](https://github.com/osmlab/osm-wayback/blob/master/HISTORICAL_SCHEMA.md)) for more on the schema of `@history`. Note: If a feature is not in the input file, it's history will not be in the output file.\n\n\n## Historic Geometries\nA fourth column family storing node locations can be created during `build_lookup_index`, depending on the value of the variable, `LOC` in `build_lookup_index.cpp`.\n\nIf the node location column family exists, the `HISTORY GEOJSONSEQ` may be passed to `add_geometry`. This function looks up every version of every node in each historical version of the object. It adds `nodeLocations` as a top-level dictionary, keyed by `node ID` and then `changeset ID` for each node.\n\n\tcat \u003cHISTORY GEOJSONSEQ\u003e | add_geometry \u003cROCKSDB\u003e \n\t\nWill create a line-delimited stream of GeoJSON OSM objects with the `nodeLocations` attribute.\n\nReconstructing historical geometries (available for nodes \u0026 ways) is then done in a separate process in `geometry-reconstruction`:\n\n\tnode geometry-reconstruction/index.js \u003cHISTORY GEOJSONSEQ with Node Locations\u003e  \n\t\nCurrently, multiple output types are supported, see [`geometry-reconstruction/README.md`](https://github.com/osmlab/osm-wayback/blob/master/geometry-reconstruction/README.md) for more information about the following output types:\n\n1. Every major and minor version are independent objects (Best for rendering historical geometries)\n2. Entries in the `@history` object include `geometry` attribute (Best for historical analysis)\n3. The `@history` object is a TopoJSON object, storing every version of the object. (More efficient than 2.)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosmlab%2Fosm-wayback","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fosmlab%2Fosm-wayback","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosmlab%2Fosm-wayback/lists"}