Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mjaschen/brouter-routingdata-builder
One-stop solution to create routing data files (.rd5) for BRouter.
https://github.com/mjaschen/brouter-routingdata-builder
brouter
Last synced: about 2 months ago
JSON representation
One-stop solution to create routing data files (.rd5) for BRouter.
- Host: GitHub
- URL: https://github.com/mjaschen/brouter-routingdata-builder
- Owner: mjaschen
- Created: 2021-02-23T07:27:51.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2024-09-07T09:16:54.000Z (4 months ago)
- Last Synced: 2024-11-01T02:42:34.532Z (2 months ago)
- Topics: brouter
- Language: Shell
- Homepage:
- Size: 53.7 KB
- Stars: 8
- Watchers: 3
- Forks: 4
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Funding: .github/FUNDING.yml
Awesome Lists containing this project
README
# BRouter Routing Data Builder
[](https://github.com/mjaschen/brouter-routingdata-builder/actions/workflows/ci.yml)
This Docker image allows to build the routing data files (`.rd5`) for BRouter.
## Requirements
At least 250 GiB of free disk space is required to build the routing data for the whole world.
At least 8 GiB of RAM is required to build the routing data for the whole world, at least 16 GiB is recommended.
## Building Routing Data
Building the routing data for the first time consists of four steps:
1. Pull the Docker image
2. Download a planet file
3. Download SRTM data
4. Build the routing data
For all subsequent builds running the last step is sufficient, as the planet data is updated automatically to the latest version in the build process and SRTM data doesn't change at all.
## Pull the Docker image
```shell
docker pull mjaschen/brouter-routingdata-builder
```
or
```shell
docker pull ghcr.io/mjaschen/brouter-routingdata-builder
```
## Downloading a Planet File
[Planet OSM](https://planet.openstreetmap.org/) is the canonical source of planet files for the whole world. For building global routing data, download the *Latest Weekly Planet File* **in PBF format**. Warning: the planet file has a size of roughly 75 GiB (May 2024).
For smaller regions and/or testing purposes, a smaller extract of the planet can be used, e.g. for Europe, a single country or an even smaller region. Geofabrik offers [extracts for all regions and countries](https://download.geofabrik.de/index.html). Download a file in **.osm.pbf** format.
Another option for getting custom planet files, especially for very small extracts (town and city level, smaller regions) is the [BBBike Extract Service](https://extract.bbbike.org/). Choose “Protocolbuffer (PBF)” as export format.
Store the downloaded planet file into a directory on the host system.
## Downloading SRTM Elevation Data
### Default SRTM Data
SRTM data with 3 arc seconds resolution (~90 m) can be downloaded here:
Just select the wanted tiles and put them into a directory. Needed format:
- Tile size: 5 deg x 5 deg
- Format: “Esri ASCII”
It's also possible to download the whole set at this [Google Drive Folder](https://drive.google.com/folderview?id=0B_J08t5spvd8RWRmYmtFa2puZEE&usp=drive_web#list). Unpack the archive and mount the directory `SRTMv4.1/5_5x5_ascii` into the container.
### High Resolution SRTM Data (optional)
[Sonny](https://sonny.4lima.de) provides curated high resolution (1 arc second) SRTM data for Europe in HGT format.
These files can be converted to BEF format using the `ElevationRasterTileConverter` tool which is part of BRouter.
## Building the Routing Data
``` shell
docker run --rm \
--user "$(id -u):$(id -g)" \
--env PLANET=slovenia-latest.osm.pbf \
--env JAVA_OPTS="-Xmx2048M -Xms2048M -Xmn256M" \
--volume /tank/brouter/tmp:/brouter-tmp \
--volume /tank/brouter/planet:/planet \
--volume /tank/brouter/srtm1:/srtm1:ro \
--volume /tank/brouter/srtm3:/srtm3:ro \
--volume /tank/brouter/segments:/segments \
ghcr.io/mjaschen/brouter-routingdata-builder
```
Let's take a closer look on what happens here:
- `--rm` delete the container after the processing is done
- `--user "$(id -u):$(id -g)"`: the processing script can and should be run as a non-privileged user, the user ID and group ID for that user can be provided with the `--user` option; in the example the current user ID and its primary group ID is used. Ensure that the mounted directories (`tmp`, `planet`, `segments`) are writable by the given user.
- `--env PLANET=slovenia-latest.osm.pbf`: the downloaded planet file is located in `/tank/brouter/planet/` (see the step further down where that directory is mounted into the container) and its name is `slovenia-latest.osm.pbf` (**important:** the planet file *must* be located in the directory which is mounted at `/planet` in the container; that means for this example that `/tank/brouter/planet/slovenia-latest.osm.pbf` must exist)
- `--env JAVA_OPTS="-Xmx2048M -Xms2048M -Xmn256M"`: the memory related options for the Java VM should be set according to the specs of the docker container; the example shows values for smaller planet files – when processing larger regions (europe, world, …) the memory requirements are higher (the default value is `-Xmx6144M -Xms6144M -Xmn256M`)
- `--volume /tank/brouter/tmp:/brouter-tmp`: the process of creating the routing data uses a lot of temporary files which are written in this mounted directory
- `--volume /tank/brouter/planet:/planet`: the directory where the planet file is located; write access is needed because the planet file will be updated by the processing script before the routing data is created
- `--volume /tank/brouter/srtm1:/srtm1:ro`: the directory where the high resolution (1 arc second) SRTM data is located, in BEF (`*.bef`) format; it's mounted read-only as there are no changes applied to the SRTM data. If no high resolution SRTM data is available, just omit this option.
- `--volume /tank/brouter/srtm3:/srtm3:ro`: the directory where the default (3 arc seconds resolution) SRTM data is located, either in zipped ASCII format (`*.zip`) or in BEF (`*.bef`) format; it's mounted read-only as there are no changes applied to the SRTM data
- `--volume /tank/brouter/segments:/segments`: that's the target directory where the created routing data files (.rd5) will be stored
If updating the planet file isn't desired, just provide `--env PLANET_UPDATE=0` as option. In this case the directory which contains the planet file can be mounted read-only.
## Benchmarks
### Planet
Updating the planet file and building routing data for the whole world takes several hours depending on the available system ressources:
| Build Time | CPU | RAM | Disk | OS | Remarks |
| --------------: | ------------------------------------------------- | ------: | ----------------------------------------------- | -------- | -------------------------------------------------------------------------------- |
| 3:45 h | Xeon E3-1270 v3 @ 3.50GHz (8 cores) | 32 GiB | 4x HGST HUH721010ALE600 10 TB, ZFS RAID-Z | Linux | `JAVA_OPTS=-Xmx15360M -Xms15360M -Xmn512M` |
| 2:49 h | AMD Ryzen 9 5950X (16 cores) | 128 GiB | 2x SAMSUNG MZQLB3T8HALS-00007 4 TB NVME, RAID 1 | Linux | `JAVA_OPTS="-Xmx30720M -Xms30720M -Xmn1024M"` |
| 3:33 h | AMD Ryzen 5 3600 (6 cores) | 64 GiB | 4x TOSHIBA MG08ACA16TEY 16 TB, ZFS RAID-Z | Linux | `JAVA_OPTS="-Xmx6G -Xms6G"` |
| 3:26 h | AMD Ryzen 5 3600 (6 cores) | 64 GiB | 4x TOSHIBA MG08ACA16TEY 16 TB, ZFS RAID-Z | Linux | `JAVA_OPTS="-Xmx8G -Xms8G"` |
| 3:20 h | AMD Ryzen 5 3600 (6 cores) | 64 GiB | 4x TOSHIBA MG08ACA16TEY 16 TB, ZFS RAID-Z | Linux | `JAVA_OPTS="-Xmx9G -Xms9G"` |
| 3:21 h | AMD Ryzen 5 3600 (6 cores) | 64 GiB | 4x TOSHIBA MG08ACA16TEY 16 TB, ZFS RAID-Z | Linux | `JAVA_OPTS="-Xmx10G -Xms10G"` |
| 3:21 h | AMD Ryzen 5 3600 (6 cores) | 64 GiB | 4x TOSHIBA MG08ACA16TEY 16 TB, ZFS RAID-Z | Linux | `JAVA_OPTS="-Xmx20G -Xms20G"` |
| 4:58 h - 5:13 h | Ampere Altra (16 shared ARM cores, Hetzner CAX41) | 32 GiB | shared NVMe | Linux | `JAVA_OPTS="-Xmx16G -Xms16G"` |
| 3:56 h | Apple Silicon M1 Max (10 cores) | 64 GiB | Apple NVMe | macOS 14 | `JAVA_OPTS="-Xmx20G -Xms20G"` |
| 2:34 h | Apple Silicon M1 Max (10 cores) | 64 GiB | Apple NVMe | macOS 14 | `JAVA_OPTS="-Xmx12G -Xms12G"`; BRouter 1.7.7 with high resolution elevation data |
It seems that no more than 10 GB of heap memory is required to create routing data for the whole planet in a minimum of time.
### Extracts/Regions
| Region | Build Time | Machine | CPU | RAM | Disk | OS | Remarks |
| ------- | -------------: | --------------------- | ----------------------- | -----: | ---------- | -------- | ---------------------------------------------------------------------------------------------------------- |
| Germany | 10:05 min | Apple Mac Studio 2022 | Apple M1 Max (10 cores) | 64 GiB | Apple NVMe | macOS 14 | `JAVA_OPTS=-Xmx20G -Xms20G`; skipped planet file update |
| Europe | 57:24 min | Apple Mac Studio 2022 | Apple M1 Max (10 cores) | 64 GiB | Apple NVMe | macOS 14 | `JAVA_OPTS=-Xmx12G -Xms12G`; skipped planet file update; BRouter 1.7.7 with high resolution elevation data |
## Development
### Building the Docker Image
```shell
docker build --pull -t brouter-routingdata-builder .
```
The image is built automatically with Github Actions when either a tag or the *main* branch is pushed. After a successful build the image is deployed to [Github Container Registry](https://github.com/users/mjaschen/packages/container/package/brouter-routingdata-builder) and [Docker Hub](https://hub.docker.com/r/mjaschen/brouter-routingdata-builder).