{"id":15058933,"url":"https://github.com/kartoza/docker-osm","last_synced_at":"2025-05-16T00:06:58.818Z","repository":{"id":35315631,"uuid":"39577209","full_name":"kartoza/docker-osm","owner":"kartoza","description":"A docker compose project to setup an OSM PostGIS database with automatic updates from OSM periodically ","archived":false,"fork":false,"pushed_at":"2025-02-20T11:15:01.000Z","size":2680,"stargazers_count":297,"open_issues_count":17,"forks_count":108,"subscribers_count":32,"default_branch":"develop","last_synced_at":"2025-04-08T11:11:10.620Z","etag":null,"topics":["diff","docker","imposm","openstreetmap","osm","osm-data","osm-pbf","osm-replication","postgis","qgis"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kartoza.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["kartoza"]}},"created_at":"2015-07-23T16:03:44.000Z","updated_at":"2025-03-20T07:28:13.000Z","dependencies_parsed_at":"2025-03-04T00:28:20.778Z","dependency_job_id":"d3029e92-c2fb-4f70-85e1-d93c14e9cf81","html_url":"https://github.com/kartoza/docker-osm","commit_stats":{"total_commits":120,"total_committers":18,"mean_commits":6.666666666666667,"dds":"0.42500000000000004","last_synced_commit":"b9f798897060daa039ff3128e7c605125af09a2d"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kartoza%2Fdocker-osm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kartoza%2Fdocker-osm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kartoza%2Fdocker-osm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kartoza%2Fdocker-osm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kartoza","download_url":"https://codeload.github.com/kartoza/docker-osm/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254442854,"owners_count":22071878,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diff","docker","imposm","openstreetmap","osm","osm-data","osm-pbf","osm-replication","postgis","qgis"],"created_at":"2024-09-24T22:33:43.712Z","updated_at":"2025-05-16T00:06:53.788Z","avatar_url":"https://github.com/kartoza.png","language":"C","funding_links":["https://github.com/sponsors/kartoza"],"categories":[],"sub_categories":[],"readme":"# Docker-OSM\n\nA docker compose project to setup an OSM PostGIS database with automatic\nupdates from OSM periodically.\nThe only files you need is a PBF file, geojson (if you intend to restrict data download to\na smaller extent than the one specified by the PBF) and run the docker compose project.\n\n## General architecture\n\n![Alt text](/docs/architecture.png?raw=true \"Optional Title\")\n\n## Quick setup\n\nAs a quick example, we are going to setup Docker-OSM with default values everywhere:\n* Download a PBF file from http://download.geofabrik.de/\n* Put the file in the `settings` folder and rename it `country.pbf`. \n\nAlternatively you can execute the `settings_downloader.sh` script to download the pbf and the clip file\n```bash\nbash ./settings_downloader.sh GEOJSON_URL CONTINENT COUNTRY ie\nbash ./settings_downloader.sh https://github.com/kartoza/docker-osm/raw/develop/settings/clip.geojson africa south-africa\n```\nFor a full list of allowed file names read json file `countries.json`\n\nAlternatively you can use the python script `pbf_downloader.py`\n\nFor local usage, the containers are set up using docker-compose configuration. The configuration files consists of \ntwo sets of config file. The first one is `.env` which contains lists of environment variables.\nCopy the `.env` file from the `.example.env` in this repo.\n\n```bash\ncp .example.env .env\n```\n\nFor subsequent configuration, edit the `.env` files to tweak your options.\n\nThe second set of configuration is using a `docker-compose.yml` files, which is a compose files docker-compose is using.\n\nFor minimum set of production environment, the configuration file is described in `docker-compose.yml` file.\nThe other YAML files with prefix `docker-compose` is a configuration file that you can merge with the basic `docker-compose.yml` file. To use more than one configuration file, you edit `.env` file and change the `COMPOSE_FILE` variable to include all \nthe compose file you desired, separated by a colon `:` for each file.\n\nFor example, by default, the example file is using both `docker-compose.yml` and `docker-compose.develop.yml` because\nwe expect you to provide the necessary settings in `settings` folder. Thus the `COMPOSE_FILE` variable looks like this:\n\n```\nCOMPOSE_FILE=docker-compose.yml:docker-compose.develop.yml\n```\n\nIn production environment, normally all the persistent data is stored in a volume, instead of bind mounted.\nIn this case only the `docker-compose.yml` is enough, and you need to provide the settings inside the volume itself.\n\nTo build the image yourself, include the `docker-compose.build.yml` file.\n\nTo use helper services such as pgadmin, include `docker-compose.pgadmin.yml`, and `docker-compose.web.yml` for web demo.\n\nIf you are familiar with how docker-compose work, you can also use a standard convention by putting `docker-compose.override.yml`\nfile and include it in the `COMPOSE_FILE` variable.\n\nTo store the configuration for long-term use (for archiving or diffing of different configuration). You can interpolate the \ncurrent variables in `.env` and generate a full config files:\n\n```bash\ndocker-compose config \u003e docker-compose.production.yml\n```\n\nThe above command will include and merged all your config files specified in `COMPOSE_FILE` variable and also fill out the \nvariables parameterized in the docker-compose file.\n\n* If you want to connect from your local QGIS Desktop:\n  * In the file `docker-compose.yml`, uncomment the block:\n\n```yml\n# Uncomment to use the postgis database from outside the docker network\nports:\n - \"35432:5432\"\n```\n* Do `make run` in the build directory. This will download and execute the docker-osm project. \nIt might be very long depending of your bandwidth and the PBF you are importing.\n* In QGIS, add a new PostGIS connection: `localhost`, database `gis`, port `35432`, `docker` for both username and password.\n* That's it! You have an OSM database, up and running. The update is done every 2 minutes from the main OSM website.\n\nFor further reading and customizations, read below.\n\n## Docker cloud\n\nDockerfiles are executed on [Docker Cloud kartoza/docker-osm](https://cloud.docker.com/swarm/kartoza/repository/docker/kartoza/docker-osm/general)\n\n```bash\ndocker pull kartoza/docker-osm:imposm-latest\ndocker pull kartoza/docker-osm:osmupdate-latest\n```\n\nTo run you can use the provided docker-compose project and use the images\nhosted on the internet. This is useful if you want to integrate Docker-OSM in\nyour existing docker-compose project.\n\n## Usage\n\n### PBF File\nIn this example we will set up an OSM database for South Africa that \nwill pull for updates every 2 minutes.\n\nSpecify a PBF file for your area in the environment variables for `osm_downloader` container.\nYou can download some PBF files on these URLS for instance :\n* http://download.geofabrik.de/\n* http://download.openstreetmap.fr/extracts/\n\n\nYou must put only one PBF file in the settings folder. Only the last one will be read.\n\n### OSM Features\n\nIn `settings`, you can edit the `mapping.yml` to customize the PostGIS schema.\nYou can find the documentation about the mapping configuration on the imposm\nwebsite: https://imposm.org/docs/imposm3/latest/mapping.html\nThe default file in Docker-OSM is coming from\nhttps://raw.githubusercontent.com/omniscale/imposm3/master/example-mapping.yml\n\n**Note** Imposm is designed for spatial analysis, not for OSM contribution analysis.\nIf you need such a feature, you need to use another database schema supporting OSM Metadata. \nYou can check the [OSM Wiki](https://wiki.openstreetmap.org/wiki/Databases_and_data_access_APIs#Database_Schemas) for \"Lossless\" schemas.\n\n### Updates\n\nYou can configure the time interval in the docker-compose file. By default,\nit's two minutes. If you set the TIME variable to 0, no diff files will be\nimported.\n\nThe default update stream is worldwide. So even if you imported a local PBF, if\nyou don't set a clipping area, you will end with data from all over the world.\n\n### Clipping\n\nDuring the initial import or post update imposm uses the flag `-limito` which allows \nyou to define a smaller area that you can work with.  \nThis is always desirable to limit the features being imported into the database rather \nthan clipping them.\n\n**NB:** Ensure you add a geojson covering the area you intend to clip into the `settings` folder.\nThe geojson can be the same extent of the administrative area for your country, or it can be a \nsmaller extent. The CRS of the geojson should always be EPSG:4326.\n\n\n**NB:** It is encouraged to simplify the geometry for the `clip.geojson` as\na simplified geometry is easier to process during the import. \nRather use the minimum bounding box for the area you intend to clip your dataset with.\n\n### QGIS project\n\nThere is a default QGIS project provided in the `web/` folder, named `osm_mirror_qgis_project.qgz`. To be able to load the layers in this project correctly in QGIS, first run `make materialized_views` and `make  elevation` then set up your connection service file with the following parameters. The host parameter is where you have set up `docker-osm`:\n\n```\n[docker-osm]\ndbname=gis\nport=35432\nuser=docker\npassword=docker\nhost=\u003chost\u003e\nsslmode=disable\n```\n\n### QGIS Styles\n\nThe database is provided with some default styles. These styles will be loaded\nautomatically when loaded in QGIS. It's following the default OSM mapping from\nImpOSM.\n\n```bash\nmake import_styles\nmake remove_styles\nmake backup_styles\n```\n\n### SQL Trigger, functions, views...\n\nYou can add PostGIS functions, triggers, materialized views into an SQL file called `post-pbf-import.sql`. \nIt will be imported automatically in the database.\n\n### Build and run\n\nNow build the docker images needed to run the application:\n\n```bash\ndocker-compose build\ndocker-compose up \n```\n\nIn production, you should daemonize the services when bringing them up:\n\n```bash\ndocker-compose up -d\n```\n\nYou can check the timestamp of your database by reading the file :\n``settings/timestamp.txt`` or you can use : \n\n```bash\nmake timestamp\n```\n\n### Display\n\nIn the makefile, you can switch to another docker compose project.\nThe other one includes QGIS Server. When it's running, you should be able to\nopen, on the host(not in docker), the `index.html` file and see OSM and QGIS\nServer showing PostGIS tables. The webpage is using Leaflet.\n\nIf you want to tweak the QGIS Project, you need to add a host in your in `/etc/hosts`:\n```\n127.0.0.1       db\n```\nBecause in the docker-compose file, the link is made with the PostGIS database using the alias `db`.\n\n## In the background\n\n![architecture](https://raw.githubusercontent.com/kartoza/docker-osm/develop/docs/docker-compose.png)\n\n### Docker OSM Update\n\nThis docker image, when run will regularly fetch any new diff file for all the\nchanges that have happened in the world over the update interval.\n\nYou can also specify a custom url for fetching the diff if you wish to retrieve\nregional diffs rather than the global one.\n\nYou can specify a polygonal area for the diff so that it will only apply features\nfrom the diff that fall within that area. For example providing a polygon of the\nborders of Malawi will result in only Malawi features being extracted from the diff.\n\n**Note:** the diff retrieved and options specified here are not related to the\ninitial base map used - so for example if your initial base map is for Malawi and\nyou specify a diff area in Botswana, updated features in Botswana will be applied\nto your base map which only includes features from Malawi. For this reason, take\ncare to ensure that your diff area coincides with the region covered by your\noriginal base map.\n\nOnce the diff has been downloaded, it is placed into /home/import_queue where\nit will be picked up by the long running imposm3 container, which will apply\nthe diff to the database.\n\nYou should have 3 folders : osm_pbf, import_queue, import_done\n\nPut a state file in base-pbf like this one :\nhttp://download.openstreetmap.fr/extracts/africa/south_africa.state.txt\n\n```bash\ndocker build -t osmupdate .\ndocker run -v $('pwd')import-queue/:/home/import-queue -v $('pwd')base-pbf/:/home/base-pbf -v $('pwd')import-done/:/home/import-done -d osmupdate\n```\n\nWith -e, you can add some settings :\n\n```bash\n - MAX_DAYS = 100, the maximum time range to assemble a cumulated changefile.\n - DIFF = sporadic, osmupdate uses a combination of minutely, hourly and daily changefiles. This value can be minute, hour, day or sporadic.\n - MAX_MERGE = 7, argument to determine the maximum number of parallely processed changefiles.\n - COMPRESSION_LEVEL = 1, define level for gzip compression. values between 1 (low compression but fast) and 9 (high compression but slow)\n - BASE_URL = http://planet.openstreetmap.org/replication/, change the URL to use a custom URL to fetch regional file updates.\n - IMPORT_QUEUE = import_queue\n - IMPORT_DONE = import_done\n - OSM_PBF = osm_pbf\n - TIME = 120, seconds between two executions of the script\n```\n\nIf you are using docker-compose, you can use these settings within the \n```docker-compose.yml``` file.\n\n### Docker ImpOSM3\n\nThis image will take care of doing the initial load for the selected region\n(e.g. planet, or a country such as Malawi) into your database. It will then\napply, at a regular interval (default is 2 minutes), any diff that arrives\nin the /home/import_queue folder to the postgis OSM database. The diffs\nare fetched by a separate container (see osm_update container).\n\nThe container will look for an OSM file (*.pbf) and its state file\n(*.state.txt) in BASE_PBF.\n\nWith -e, you can add some settings :\n\n```bash\n - TIME = 120, seconds between 2 executions of the script\n - POSTGRES_USER = docker, default user\n - POSTGRES_PASS = docker, default password\n - POSTGRES_HOST = db\n - POSTGRES_PORT = 5432\n - SETTINGS = settings, folder for settings (with *.json and *.sql)\n - CACHE = cache, folder for caching\n - BASE_PBF = base_pbf, folder the OSM file\n - IMPORT_DONE = import_done, folder for diff which has been imported\n - IMPORT_QUEUE = import_queue, folder for diff which hasn't been imported yet\n - SRID = 4326, it can be 3857\n - OPTIMIZE = false, check (Imposm)[http://imposm.org/docs/imposm3/latest/tutorial.html#optimize]\n - DBSCHEMA_PRODUCTION = public, check (Imposm)[http://imposm.org/docs/imposm3/latest/tutorial.html#deploy-production-tables]\n - DBSCHEMA_IMPORT = import, check (Imposm)[http://imposm.org/docs/imposm3/latest/tutorial.html#deploy-production-tables]\n - DBSCHEMA_BACKUP = backup, check (Imposm)[http://imposm.org/docs/imposm3/latest/tutorial.html#deploy-production-tables]\n```\n\nYou can adjust these preferences in the ```docker-compose.yml``` file provided\nin this repository.\n\n# PostGIS\n\nFor environment variables associated with `docker-postgis` refer to [docker postgis repository](https://github.com/kartoza/docker-postgis)\n\n### Support\n\nIf you require more substantial assistance from [kartoza](https://kartoza.com)  (because our work and interaction on docker-osm is pro bono),\nplease consider taking out a [Support Level Agreeement](https://kartoza.com/en/shop/product/support) \n\n\n# Credits\n\nThis application was designed and implemented by:\n\n* Etienne Trimaille (etienne.trimaille@gmail.com)\n* Tim Sutton (tim@kartoza.com)\n\nWith some important design ideas provided by Ariel Nunez (ingenieroariel@gmail.com).\n\nParts of this project are built on the existing work of others.\n\nJuly 2015\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkartoza%2Fdocker-osm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkartoza%2Fdocker-osm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkartoza%2Fdocker-osm/lists"}