{"id":19881441,"url":"https://github.com/giscience/osm-realtime-update","last_synced_at":"2025-05-02T14:30:37.255Z","repository":{"id":26367019,"uuid":"108410786","full_name":"GIScience/OSM-realtime-update","owner":"GIScience","description":":clock2: OpenStreetMap Data Extracts - updated on the fly!","archived":false,"fork":false,"pushed_at":"2022-12-10T16:08:06.000Z","size":10283,"stargazers_count":41,"open_issues_count":10,"forks_count":6,"subscribers_count":16,"default_branch":"master","last_synced_at":"2024-04-09T15:12:04.403Z","etag":null,"topics":["javascript","mapping","nodejs","openstreetmap","pbf"],"latest_commit_sha":null,"homepage":"https://giscience.github.io/OSM-realtime-update/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GIScience.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-10-26T12:49:53.000Z","updated_at":"2024-04-09T15:12:04.404Z","dependencies_parsed_at":"2023-01-14T04:30:20.598Z","dependency_job_id":null,"html_url":"https://github.com/GIScience/OSM-realtime-update","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GIScience%2FOSM-realtime-update","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GIScience%2FOSM-realtime-update/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GIScience%2FOSM-realtime-update/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GIScience%2FOSM-realtime-update/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GIScience","download_url":"https://codeload.github.com/GIScience/OSM-realtime-update/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224315203,"owners_count":17290992,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["javascript","mapping","nodejs","openstreetmap","pbf"],"created_at":"2024-11-12T17:14:17.021Z","updated_at":"2024-11-12T17:14:17.640Z","avatar_url":"https://github.com/GIScience.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Real-time OSM\n\nA service providing real-time OSM data.\n\nThree components:\n\n1. The API adds, deletes and requests information about tasks that provide\n   region-bound real-time OSM data in PBF file format via a permanent URL\n\n2. The backend runs, manages and gathers statistics for tasks\n\n3. The frontend allows manipulating tasks and serves the OSM data produced by\n   the tasks.\n\n\n## Installation\n\n### Manual\n\n1. `git clone https://github.com/GIScience/OSM-realtime-update`\n\n2. `cd OSM-realtime-update/server`\n\n3. `npm install`\n\n4. `npm start`\n\n### Docker (caution: not fully tested)\n\nUse the Dockerfile provided.\n\n1. `docker build -t realtimeosm .`\n\n2. `docker run -v /path/to/data:/OSM-realtime-update/server/data -p 1234:1234 -d realtimeosm` (NOTE: data folder must not be existent, will be created by container)\n\n\n### Config file\n\nYou will find a config.js file in the server directory controlling server and api global\nsettings. For instance this will let you determine the number of workers to be run in \nparallel, specific storage paths, ports and update settings. \n\n## Usage\n\nReal-time OSM provides a web-frontend, that is accessible via port 1234.\n\nPoint your browser to `http://localhost:1234/`.\n\n## API Specification\n\nBase Url: `/api`\n\nMain collection: `/api/tasks`\n\nIndividual resource: `/api/tasks/:id`\n\n\n#### Add task\n\n* URL: `/api/tasks`\n\n* Method: POST: x-www-form-unencoded or raw/JSON\n\n* Request Parameters:\n\n\t- *name:* character string [a-zA-Z]\n\t- *coverage:* a polygon in WKT / GeoJSON format or a Geofabrik region string (eg. 'togo')\n\t\t- maximum nodes: 10000\n\t- *expirationDate (optional):* timestamp in ISO 8601 format [YYYY-MM-DDTHH:MM:SS+HH:MM]\n    - *updateInterval (optional):* time in seconds\n\n* Response:\n\n\t- status code [int]\n\t- tasks properties\n\t\t- id [int]\n\t\t- name [string]\n\t\t- coverage [WKT polygon]\n\t\t- URL to final product [string]\n\t\t- expiration date [YYYY-MM-DDTHH:MM:SS+HH:MM]\n        - updateInterval [int]\n        - lastUpdated [YYYY-MM-DDTHH:MM:SS+HH:MM]\n        - creationDate [YYYY-MM-DDTHH:MM:SS+HH:MM]\n        - averageRuntime [int]\n\n* Example WKT/x-www-form-unencoded:\n\n`curl --data \"name=test_region\u0026coverage=Polygon ((-0.61 53.05, -0.22 51.92, -3.04 52.20, -0.61 53.05))\u0026expirationDate=2020-05-01\" http://localhost:1234/api/tasks`\n\n* Example GeoJSON/x-www-form-unencoded:\n      \n`curl --data 'name=test_region\u0026coverage={\"type\":\"Feature\",\"geometry\":{\"type\":\"Polygon\",\"coordinates\":[[[-0.61,53.05],[-3.04,52.20],[-0.22,51.92],[-0.61,53.05]]]}, \"properties\": {\"name\": \"GB\"}}\u0026expirationDate=2020-05-01' http://localhost:1234/api/tasks`\n\n* Example Geofabrik region string/x-www-form-unencoded:\n\n`curl --data 'name=Togo\u0026coverage=togo\u0026updateInterval=10' http://localhost:1234/api/tasks`\n\n\n#### Delete task\n\n* URL: `/api/tasks`\n\n* Method: DELETE\n\n* Request Parameters\n\n\t- *id* [int]\n\n* Response:\n\n\t- status code [int]\n\t- status message [string]\n\n* Example:\n\n\tcurl --data \"id=5\" -X DELETE http://localhost:1234/api/tasks\n\n\u0026nbsp;\n\n\n#### Get information about tasks\n\n* URL: `/api/tasks`\n\n\n* Method: GET\n\n* Filter Parameters in URL:\n\n\t- *id* [int]: `/api/tasks?id=:id`\n\t- *name* [string]: `/api/tasks?name=:name`\n\t\t- multiple matches prints multiple tasks\n\t\n\tIf no parameter is given, all tasks are returned.\n\n    Individual tasks can also be accessed via their resource URL `/api/tasks/:id`.\n\n* Response:\n\n\t- status code [int]\n\t- status message [string]\n\t- list of tasks found with properties:\n\t\t- name [string]\n\t\t- id [int]\n\t\t- coverage [WKT polygon]\n\t\t- URL to final product [string]\n\t\t- runtime statistics [string]\n\n* Example:\n\n\t* curl http://localhost:1234/api/tasks\n\n\t* curl http://localhost:1234/api/tasks/2\n\n\t* curl http://localhost:1234/api/tasks/id=2\n\n\t* curl http://localhost:1234/api/tasks/name=test1\n\n\n\n\n\n\u0026nbsp;\n\n\n\n## Backend Specification\n\n#### Data storage \n\n\nThe OSM data files are stored in a subfolder that is served by\nthe web app. File names are generated using name and id.\n\nA main sqlite3 database keeps track of the application state, it \ncontains two tables:\n\n1. tasks\n    - stores all information about tasks\n\t- header: id, name, url, coverage, lastUpdated, averageRuntime,\n\t  addedDate, expirationDate\n\n2. taskstats\n    - stores all benchmarks\n    - header: timestamp, taskID, timing\n\nFormat:\n- all dates in ISO 8601 format.\n- coverage: a polygon in WKT-format\n- timing: milliseconds\n\n\n\n#### Update strategy\n\nFor each task, initialise data by downloading a suitable .pbf from Geofabrik,\nclip it to the task's boundary and update it using osmupdate.\n\nAlgorithm:\n\n1. Is an update process running for this task?\n   - Yes: Abort\n   - No: go to step 2 and start update\n\n2. Is there an initial .pbf-file?\n   - Yes: Update the file using osmupdate (+timing)\n   - No: Generate initial .pbf-file:\n\t  - download smallest Geofabrik extract that covers the task \n[(sample code)](https://github.com/BikeCitizens/geofabrik-extracts)\n      - clip extract using the task polygon/bbox\n      - go to step 2\n\n\u0026nbsp;\n\n\n#### Task management\n\nWorker objects care for a specific task including scheduled updates. A master object\nmanages all workers and checks regularly whether a) workers became obsolete\nbecause tasks have been removed or b) a new worker should be spawned because a\nnew task has been added.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgiscience%2Fosm-realtime-update","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgiscience%2Fosm-realtime-update","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgiscience%2Fosm-realtime-update/lists"}