{"id":18256928,"url":"https://github.com/mediacomem/vtmarkersgenerator","last_synced_at":"2025-04-08T22:28:55.817Z","repository":{"id":73832609,"uuid":"286981493","full_name":"MediaComem/vtmarkersgenerator","owner":"MediaComem","description":"Tasks pipeline to generate MBtiles (vector tiles) from a PostGIS server using channel notification, og2ogr and tippecanoe. ","archived":false,"fork":false,"pushed_at":"2021-01-28T15:23:51.000Z","size":189,"stargazers_count":1,"open_issues_count":0,"forks_count":2,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-02-14T17:43:25.996Z","etag":null,"topics":["clustering","map","vector-tiles"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MediaComem.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-12T10:13:35.000Z","updated_at":"2024-01-07T03:18:29.000Z","dependencies_parsed_at":"2023-09-21T08:02:57.551Z","dependency_job_id":null,"html_url":"https://github.com/MediaComem/vtmarkersgenerator","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MediaComem%2Fvtmarkersgenerator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MediaComem%2Fvtmarkersgenerator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MediaComem%2Fvtmarkersgenerator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MediaComem%2Fvtmarkersgenerator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MediaComem","download_url":"https://codeload.github.com/MediaComem/vtmarkersgenerator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247938525,"owners_count":21021538,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clustering","map","vector-tiles"],"created_at":"2024-11-05T10:24:11.225Z","updated_at":"2025-04-08T22:28:55.812Z","avatar_url":"https://github.com/MediaComem.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Vector Tiles Markers Generator\n\nTasks pipeline to generate MBtiles (vector tiles) from a PostGIS server using channel notification, og2ogr and tippecanoe. This was especially made for simple tasks like creating a layer with a lot of markers/points.\n\nWhen a new notification from `PG_notify` is received, a GeoJSON is created with a specific SQL query for the given task and then processed with Tippecanoe to create a MBTiles file. The file can be then served for example with [maptiler/tileserver-gl](https://github.com/maptiler/tileserver-gl) or [consbio/mbtileserver](https://github.com/consbio/mbtileserver).\n\n## Getting started with Node.js\n\nTo run the application on your machine, you will need:\n\n* ogr2ogr (from GDAL)\n* tippecanoe\n\n### Installation of ogr2ogr on ubuntu\n\n```bash\nsudo add-apt-repository ppa:ubuntugis/ppa\nsudo apt-get update\nsudo apt-get install gdal-bin\n```\n\n### Installation of Tippecanoe on ubuntu\n\n```bash\ngit clone https://github.com/mapbox/tippecanoe.git\ncd tippecanoe\nmake -j\nmake install\n\n### Running the application\n\n```bash\n# Install dependencies.\nnpm ci\n\n# Run the application.\nnpm run start\n```\n\n## Getting started with Docker\n\nTo run the development environment in Docker containers, you will need:\n\n* Docker 19+\n* Docker Compose 1.25+\n\nThen simply build and run the application:\n\n```bash\ndocker build . --tag smapshot-points-vt-generate:X.X\n```\n\n## Using the vector tiles\n\nBe aware that current Tippecanoe configuration guess what the best max zoom is. Vector tiles use the principle of \"overzoom\". When max zoom is reached, the displayed vector tiles are still shown at greater zoom. If you load the map at a higher zoom of the max zoom, you'll see the tiles only if the max zoom parameter is the same as the one set by Tippecanoe on the configuration. So it means you should avoid loading tiles with `.pbf` directly in your front-end library, but use the TileJSON URL (generated by your vector tiling server) instead which contains the generated max zoom value.\n\n## Configuration\n\nTilesets are stored at `/usr/src/app/output` use a volume to share it with other containers using docker-compose:\n\n```yaml\nvolumes:\n    - 'vt_tilesets:/usr/src/app/output'\n```\n\n## Tasks file\n\nTasks are defined in a yaml file as `./tasks.yml`. Copy a local file to the container within a DockerFile or share a volume to allow the container to access it:\n\n```yaml\nvolumes:\n    - yourLocalRelativePath/tasks.yml:/usr/src/app/tasks.yml\n```\n\nEach task is defined with the name of the task under the tasks attributes:\n\n```yaml\ntasks:\n    nameOfTheTask:\n        channelName: \"myChannelName\" # Channel name is use in Postgresl to trigger a new task with for example 'NOTIFY myChannelName;'\n        sql: \"SELECT id, location FROM images\" # SQL query used to export data to GeoJSON. Geometry is automagically discovered. Other attributes are stored in the properties of each feature\n        sqlColumNameRef: \"images.id\" # Column reference which are used in the payload and the update query. Tips: avoid ambiguity by proving table name\n        vtParams: # Command parameters to generate vector tiles with Tippecanoe. Default are '--force', '--quiet' and export-input paths.\n        - \"-z9\" # -zg auto is not recommended. Tile-join support only merging mbtiles with same max zoom level.\n        - \"--drop-densest-as-needed\"\n        - \"--extend-zooms-if-still-dropping\"\n```\n\n## Trigger and Notify PSQL function\n\nExample of a trigger and a way to notify for change on a table:\n\n```sql\nCREATE OR REPLACE FUNCTION update_points_vt() RETURNS trigger AS $$\nBEGIN\n -- When state changed, update vt\n IF old.state \u003c\u003e new.state\n THEN\n    -- To update an item (single mode)\n    PERFORM pg_notify(\n        'myChannelName'::text,\n        json_build_object(\n            'action', 'add', -- add/remove are supported\n            'ref', new.id\n        )::text\n    );\n    -- To update all items (bulk mode)\n    PERFORM pg_notify('myChannelName'::text, json_build_object()::text);\n END IF;\n\n RETURN NULL;\nEND;\n$$ LANGUAGE plpgsql;\n\nDROP TRIGGER IF EXISTS update_points_vt_trigger ON images;\n\nCREATE TRIGGER update_points_vt_trigger AFTER UPDATE on images\nFOR EACH ROW EXECUTE PROCEDURE update_points_vt();\n```\n\n## Environment\n\nMoreover, the following environment variables need to be set at build time:\n\nVariable                         | Default value                                | Description\n:---                             | :---                                         | :---\n`DB_USER`                        | -                                            | The username to connect to the database\n`DB_PASS`                        | -                                            | The password to connect to the database\n`DB_HOST`                        | -                                            | The hostname to connect to the database\n`DB_PORT`                        | -                                            | The port to connect to the database\n`DB_NAME`                        | -                                            | The database name to connect to the database\n\nOptional variables can also be set for extra features:\n\nVariable                         | Default value                                | Description\n:---                             | :---                                         | :---\n`KILL_IMAGE_NAME`                | -                                            | Image name corresponding to a running container to send kill signal (for example to gracefully restart it when new tilesets are generated)\n`KILL_SIGNAL`                    | -                                            | Type of kill signal send to KILL_IMAGE_NAME container\n`TMP_PATH`                       | -                                            | Temporary working folder inside the container. Be aware that TMP_PATH and OUTPUT_PATH need to be on the same filesystem.\n`OUTPUT_PATH`                    | -                                            | Output mbtiles folder inside the container. Be aware that TMP_PATH and OUTPUT_PATH need to be on the same filesystem.\n\n### Reload the vector tiles server\n\nA fork of `consbio/mbtileserver` has been created which add a file watcher functionality. Every minute a cron job is initiated to check if mbtiles inside a given folder have been changed and send a kill signal inside the container to gracefully restart the tilesets generation of mbtileserver. If you need a faster solution, you should check the next chapter.\n\n#### Using kill signal\n\n⚠ Be aware that sharing the docker socket could comprise the security of your docker instance and especially your host\n\nThe pipeline can send a kill signal to another Docker container sharing the docker socket. It can be useful to gracefully restart the vector tiles server when new tilesets are generated.\n\nTo use this feature, the docker socket needs to be mounted as a volume when using docker-compose:\n\n```yaml\nvolumes:\n    - '/var/run/docker.sock:/var/run/docker.sock'\n```\n\nAnd docker container user `smapshot_vt` need to be granted write permission on the `docker.sock` file of the host. This might be done by giving access to `docker` user group.\n\n#### Using mediacomem/mbtileserver fork\n\nA safer option is to use a tileserver with a cron job like the mediacomem/mbtileserver fork.\n\nThe setup is imple: the volumes containing the tilesets need to be shared with the container.\nThe disadvantage is that the cron job is watching only every minute for changes.\n\n## Code improvements\n\nSome directions of how the project could be enhanced:\n\n* Queue seem to be limited currenlty. After a certain amount of PSQL Notification, new notification aren't added to the queue. This might be a limitation of better-queue which use by default in-memory management.\n* Functions use too much arguments. It could be better to use object to share arguments across function in index.ts.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmediacomem%2Fvtmarkersgenerator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmediacomem%2Fvtmarkersgenerator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmediacomem%2Fvtmarkersgenerator/lists"}