{"id":19816985,"url":"https://github.com/geoadmin/routing-graph-packager","last_synced_at":"2025-05-01T10:33:20.643Z","repository":{"id":39802732,"uuid":"366621504","full_name":"geoadmin/routing-graph-packager","owner":"geoadmin","description":"Valhalla routing network packager for OSM data.","archived":false,"fork":false,"pushed_at":"2025-01-20T13:08:58.000Z","size":5474,"stargazers_count":3,"open_issues_count":0,"forks_count":2,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-06T12:51:43.635Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/geoadmin.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-12T06:55:50.000Z","updated_at":"2025-01-20T13:08:58.000Z","dependencies_parsed_at":"2024-01-08T10:42:25.541Z","dependency_job_id":"89d46350-01b1-4290-86d9-7960ea1cb6b8","html_url":"https://github.com/geoadmin/routing-graph-packager","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geoadmin%2Frouting-graph-packager","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geoadmin%2Frouting-graph-packager/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geoadmin%2Frouting-graph-packager/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geoadmin%2Frouting-graph-packager/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/geoadmin","download_url":"https://codeload.github.com/geoadmin/routing-graph-packager/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251860593,"owners_count":21655772,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T10:11:22.766Z","updated_at":"2025-05-01T10:33:20.636Z","avatar_url":"https://github.com/geoadmin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Routing Graph Packager HTTP API\n\n[![tests](https://github.com/geoadmin/routing-graph-packager/actions/workflows/test-ubuntu.yml/badge.svg)](https://github.com/geoadmin/routing-graph-packager/actions/workflows/test-ubuntu.yml)\n[![Coverage Status](https://coveralls.io/repos/github/gis-ops/routing-graph-packager/badge.svg)](https://coveralls.io/github/gis-ops/routing-graph-packager)\n\nA [FastAPI](https://github.com/tiangolo/fastpi) app to dynamically deliver ZIPped [Valhalla](https://github.com/valhalla/valhalla) graphs from a variety of data sources.\n\nThe default road dataset is the [OSM](openstreetmap.org) planet PBF. If available, it also supports road datasets of commercial vendors, such as TomTom and HERE, assuming they are provided in the [OSM PBF format](https://wiki.openstreetmap.org/wiki/PBF_Format#).\n\n## Features\n\n- **user store**: with basic authentication for `POST` and `DELETE` endpoints\n- **bbox extracts**: generate routing packages within a bounding box\n- **data updater**: includes a daily OSM updater\n- **asynchronous API**: graph generation is outsourced to a [`ARQ`](https://github.com/samuelcolvin/arq) worker\n- **email notifications**: notifies the requesting user if the job succeeded/failed\n- **logs API** read the logs for the worker, the app and the graph builder via the API\n\n## \"Quick Start\"\n\nThe following will download\n\nFirst you need to clone the project:\n\n```\ngit clone https://github.com/gis-ops/routing-graph-packager.git\n```\n\nSince the graph generation takes place in docker containers, you'll also need to pull the relevant image: `docker pull ghcr.io/gis-ops/docker-valhalla/valhalla:latest`.\n\nThe easiest way to quickly start the project is to use `docker-compose`:\n\n```\ndocker-compose up -d\n```\n\nWith the project defaults, you can now make a `POST` request which will generate a graph package in `DATA_DIR` (default `./data`) from Andorra:\n\n```\ncurl --location -XPOST 'http://localhost:5000/api/v1/jobs' \\\n--header 'Authorization: Basic YWRtaW5AZXhhbXBsZS5vcmc6YWRtaW4=' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n\t\"name\": \"test\",  # name needs to be unique for a specific router \u0026 provider\n\t\"description\": \"test descr\",\n\t\"bbox\": \"1.531906,42.559908,1.6325,42.577608\",  # the bbox as minx,miny,maxx,maxy\n\t\"provider\": \"osm\",  # the dataset provider, needs to be registered in ENABLED_PROVIDERS\n\t\"update\": \"true\"  # whether this package should be updated on every planet build\n}'\n```\n\nAfter a minute you should have the graph package available in `./data/output/osm_test/`. If not, check the logs of the worker process or the Flask app.\n\nThe `routing-packager-app` container running the HTTP API has a `supervisor` process running in a loop, which:\n\n- downloads a planet PBF (if it doesn't exist) or updates the planet PBF (if it does exist)\n- builds a planet Valhalla graph\n- then updates all graph extracts with a fresh copy\n\nBy default, also a fake SMTP server is started, and you can see incoming messages on `http://localhost:1080`.\n\n## Concepts\n\n### Graph \u0026 OSM updates\n\nUnder the hood we're running a `supervisor` instance to control the graph builds.\n\nTwo instances of the [Valhalla docker image](https://github.com/gis-ops/docker-valhalla) take turns building a new graph from an updated OSM file. Those two graphs are physically separated from each other in subdirectories `$TMP_DATA_DIR/osm/8002` \u0026 `$TMP_DATA_DIR/osm/8003`.\n\nAfter each graph build finished, the OSM file is updated for the next graph build.\n\n### Data sources\n\nThis service tries to be flexible in terms of data sources and routing engines. Consequently, we support proprietary dataset such as from TomTom or HERE.\n\nHowever, all data sources **must be** in the OSM PBF format and follow the OSM tagging model. There is [commercial support](https://github.com/gis-ops/prop2osm) in case of interest.\n\n### `POST` new job\n\nThe app is listening on `/api/v1/jobs` for new `POST` requests to generate some graph according to the passed arguments. The lifecycle is as follows:\n\n1. Request is parsed, inserted into the Postgres database and the new entry is immediately returned with a few job details as blank fields.\n2. Before returning the response, the graph generation function is queued with `ARQ` in a Redis database to dispatch to a worker.\n3. If the worker is currently\n   - **idle**, the queue will immediately start the graph generation:\n     - Pull the job entry from the Postgres database\n     - Update the job's `status` database field along the processing to indicate the current stage\n     - Zip graph tiles from disk according to the request's bounding box and put the package to `$DATA_DIR/output/\u003cJOB_NAME\u003e`, along with a metadata JSON\n   - **busy**, the current job will be put in the queue and will be processed once it reaches the queue's head\n4. Send an email to the requesting user with success or failure notice (including the error message)\n\n### Logs\n\nThe app exposes logs via the route `/api/v1/logs/{log_type}`. Available log types are `worker`, `app` and `builder`. An optional query parameter `?lines={n}` limits the output to the last `n` lines. Authentication is required.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeoadmin%2Frouting-graph-packager","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeoadmin%2Frouting-graph-packager","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeoadmin%2Frouting-graph-packager/lists"}