Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/giscience/osm-realtime-update

:clock2: OpenStreetMap Data Extracts - updated on the fly!
https://github.com/giscience/osm-realtime-update

javascript mapping nodejs openstreetmap pbf

Last synced: 3 months ago
JSON representation

:clock2: OpenStreetMap Data Extracts - updated on the fly!

Awesome Lists containing this project

README 

        
# Real-time OSM



A service providing real-time OSM data.



Three components:



1. The API adds, deletes and requests information about tasks that provide

   region-bound real-time OSM data in PBF file format via a permanent URL



2. The backend runs, manages and gathers statistics for tasks



3. The frontend allows manipulating tasks and serves the OSM data produced by

   the tasks.



## Installation



### Manual



1. `git clone https://github.com/GIScience/OSM-realtime-update`



2. `cd OSM-realtime-update/server`



3. `npm install`



4. `npm start`



### Docker (caution: not fully tested)



Use the Dockerfile provided.



1. `docker build -t realtimeosm .`



2. `docker run -v /path/to/data:/OSM-realtime-update/server/data -p 1234:1234 -d realtimeosm` (NOTE: data folder must not be existent, will be created by container)



### Config file



You will find a config.js file in the server directory controlling server and api global

settings. For instance this will let you determine the number of workers to be run in 

parallel, specific storage paths, ports and update settings. 



## Usage



Real-time OSM provides a web-frontend, that is accessible via port 1234.



Point your browser to `http://localhost:1234/`.



## API Specification



Base Url: `/api`



Main collection: `/api/tasks`



Individual resource: `/api/tasks/:id`



#### Add task



* URL: `/api/tasks`



* Method: POST: x-www-form-unencoded or raw/JSON



* Request Parameters:



	- *name:* character string [a-zA-Z]

	- *coverage:* a polygon in WKT / GeoJSON format or a Geofabrik region string (eg. 'togo')

		- maximum nodes: 10000

	- *expirationDate (optional):* timestamp in ISO 8601 format [YYYY-MM-DDTHH:MM:SS+HH:MM]

    - *updateInterval (optional):* time in seconds



* Response:



	- status code [int]

	- tasks properties

		- id [int]

		- name [string]

		- coverage [WKT polygon]

		- URL to final product [string]

		- expiration date [YYYY-MM-DDTHH:MM:SS+HH:MM]

        - updateInterval [int]

        - lastUpdated [YYYY-MM-DDTHH:MM:SS+HH:MM]

        - creationDate [YYYY-MM-DDTHH:MM:SS+HH:MM]

        - averageRuntime [int]



* Example WKT/x-www-form-unencoded:



`curl --data "name=test_region&coverage=Polygon ((-0.61 53.05, -0.22 51.92, -3.04 52.20, -0.61 53.05))&expirationDate=2020-05-01" http://localhost:1234/api/tasks`



* Example GeoJSON/x-www-form-unencoded:

      

`curl --data 'name=test_region&coverage={"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[-0.61,53.05],[-3.04,52.20],[-0.22,51.92],[-0.61,53.05]]]}, "properties": {"name": "GB"}}&expirationDate=2020-05-01' http://localhost:1234/api/tasks`



* Example Geofabrik region string/x-www-form-unencoded:



`curl --data 'name=Togo&coverage=togo&updateInterval=10' http://localhost:1234/api/tasks`



#### Delete task



* URL: `/api/tasks`



* Method: DELETE



* Request Parameters



	- *id* [int]



* Response:



	- status code [int]

	- status message [string]



* Example:



	curl --data "id=5" -X DELETE http://localhost:1234/api/tasks



 



#### Get information about tasks



* URL: `/api/tasks`



* Method: GET



* Filter Parameters in URL:



	- *id* [int]: `/api/tasks?id=:id`

	- *name* [string]: `/api/tasks?name=:name`

		- multiple matches prints multiple tasks

	

	If no parameter is given, all tasks are returned.



    Individual tasks can also be accessed via their resource URL `/api/tasks/:id`.



* Response:



	- status code [int]

	- status message [string]

	- list of tasks found with properties:

		- name [string]

		- id [int]

		- coverage [WKT polygon]

		- URL to final product [string]

		- runtime statistics [string]



* Example:



	* curl http://localhost:1234/api/tasks



	* curl http://localhost:1234/api/tasks/2



	* curl http://localhost:1234/api/tasks/id=2



	* curl http://localhost:1234/api/tasks/name=test1



 



## Backend Specification



#### Data storage 



The OSM data files are stored in a subfolder that is served by

the web app. File names are generated using name and id.



A main sqlite3 database keeps track of the application state, it 

contains two tables:



1. tasks

    - stores all information about tasks

	- header: id, name, url, coverage, lastUpdated, averageRuntime,

	  addedDate, expirationDate



2. taskstats

    - stores all benchmarks

    - header: timestamp, taskID, timing



Format:

- all dates in ISO 8601 format.

- coverage: a polygon in WKT-format

- timing: milliseconds



#### Update strategy



For each task, initialise data by downloading a suitable .pbf from Geofabrik,

clip it to the task's boundary and update it using osmupdate.



Algorithm:



1. Is an update process running for this task?

   - Yes: Abort

   - No: go to step 2 and start update



2. Is there an initial .pbf-file?

   - Yes: Update the file using osmupdate (+timing)

   - No: Generate initial .pbf-file:

	  - download smallest Geofabrik extract that covers the task 

[(sample code)](https://github.com/BikeCitizens/geofabrik-extracts)

      - clip extract using the task polygon/bbox

      - go to step 2



 



#### Task management



Worker objects care for a specific task including scheduled updates. A master object

manages all workers and checks regularly whether a) workers became obsolete

because tasks have been removed or b) a new worker should be spawned because a

new task has been added.