{"id":13425704,"url":"https://github.com/ondrap/mapbox-filter","last_synced_at":"2025-07-08T19:33:27.860Z","repository":{"id":66322737,"uuid":"138433172","full_name":"ondrap/mapbox-filter","owner":"ondrap","description":"Utilities for filtering vector mbitles files according to Mapbox GL styles","archived":false,"fork":false,"pushed_at":"2024-02-12T16:26:51.000Z","size":124,"stargazers_count":27,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-28T04:23:32.947Z","etag":null,"topics":["mapbox-gl","mapbox-vector-tile","mbtiles"],"latest_commit_sha":null,"homepage":null,"language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ondrap.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-06-23T20:59:55.000Z","updated_at":"2024-07-11T09:45:19.000Z","dependencies_parsed_at":"2024-01-17T03:15:34.022Z","dependency_job_id":"102b113c-cfa8-40ad-b05d-8939c256a21a","html_url":"https://github.com/ondrap/mapbox-filter","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondrap%2Fmapbox-filter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondrap%2Fmapbox-filter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondrap%2Fmapbox-filter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondrap%2Fmapbox-filter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ondrap","download_url":"https://codeload.github.com/ondrap/mapbox-filter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225458213,"owners_count":17477410,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mapbox-gl","mapbox-vector-tile","mbtiles"],"created_at":"2024-07-31T00:01:17.076Z","updated_at":"2024-11-20T02:34:29.752Z","avatar_url":"https://github.com/ondrap.png","language":"Haskell","funding_links":[],"categories":["Haskell","CLI Utilities"],"sub_categories":[],"readme":"# Mapbox-filter - filtering mbtiles file according to Mapbox GL JS styles\n\nA library that can interpret a subset of the Mapbox style epxression, a very simplified\nparser for the Mapbox GL JS style and an executable that can:\n\n- Dump the tile (.mvt, .pbf files) and show which features will be included given the style file at a particular zoom level.\n- Iterate through the `mbtiles` file and filter the tile contents according to the MapBox style, thus making the `mbtiles` file smaller.\n- Preprocess attributes with right-to-left (arabic etc.) text fields; as a result the\n  mapbox-gl rtl plugin can be omitted.\n- Run a webserver for\n  * serving the tiles from the `mbtile` file\n  * serving the real-time filtered tiles\n  * after serving a tile saving the compressed tile back to the database (Planetiler database only is currently supported in this mode)\n- Publish tiles to S3 so that you don't need to run a webserver at all. As this can\n  take a very long time, incremental, differential and parallel upload is supported.\n\nThis library supports only a subset of the expression language (https://www.mapbox.com/mapbox-gl-js/style-spec/#expressions-types).\nIt's because I don't need that and most of the language isn't going to be used in the filter expression anyway. If you need\nfeatures that are not implemented yet, create an issue.\n\nThe filtering first executes the filtering expression and removes features that will not\nbe displayed. Then it removes metadata that is not used in the styles. The removal\nprocess is currently somewhat crude (it retains all metadata used at the particular layer),\nbut it should be enough for most usecases.\n\nCurrently only the Planetiler mbtile files are supported for `filter` and `publish` commands.\nThe `web` command should be compatibile with any mbtile file.\n\n## How to compile\n\n1. Install stack - https://docs.haskellstack.org/en/stable/README/\n2. `stack setup`\n3. `stack build`\n4. `stack install` - installs binary `mapbox-filter` to ~/.local/bin\n5. or you can run `stack exec -- mapbox-filter` instead without installing\n\nI have not tested it but it will probably work on Windows as well.\n\nA special version of `text-icu` library is required. Everything should work correctly\nwith stack, cabal users need to look into the `stack.yaml` file and install the library\nmanually.\n\n## Moving data from a zoom-level down\n\nSometimes it might be desirable to move some data from a higher zoom-level to a lower zoom-level.\nThe `copy-down` function replaces all data that is matched by the filter from the \ndestination zoom level with data form one level up. In the following example,\nthe river data is moved from zoom 9 to zoom 8. \n\n```\n{\n  \"dst-zoom\": 8,\n  \"source-layer\": \"waterway\", \n  \"filter\": [\n    \"all\",\n    [\"==\", [\"geometry-type\"], \"LineString\"],\n    [\"!=\", [\"string\", [\"get\", \"class\"]], \"stream\"],\n    [\n      \"match\",\n      [\"string\", [\"get\", \"brunnel\"], \"\"],\n      [\"tunnel\", \"bridge\"],\n      false,\n      true\n    ]\n  ]\n}\n```\n\n## Examples\n\nShow CLI help:\n```\n$ mapbox-filter -h\n$ mapbox-filter publish -h\n```\n\nApply the style on all the tiles in the `cz.mbtiles`. The process uses all available CPUs.\nYou can you use multiple `-j` options to create one file containing data for all styles.\n```\n$ mapbox-filter filter -j mapboxstyle.json cz.mbtiles\n```\n\nServe the mbtiles file. The endpoint for MapBox is: http://server_name:3000/tiles/metadata.json\n```\n$ mapbox-filter web -p 3000 cz.mbtiles\n```\n\nServe the mbtiles file while doing online filtering according to the mapboxstyle.json file.\nPre-process the right-to-left metadata text fields.\n```\n$ mapbox-filter web -p 3000 --rtl-convert -j mapboxstyle.json cz.mbtiles\n```\n\nPublish filtered mbtiles to S3. Higher parallelism might be desirable, use the `-p`\nparameter to facilitate more parallel uploads to S3.\n```\n$ mapbox-filter publish \n  -j mapboxstyle.json\n  -u https://s3.eu-central-1.amazonaws.com/my-test-bucket/styled-map\n  -t s3://my-test-bucket/styled-map -p 10 cz.mbtiles\n```\n\n## Incremental job\n\nUnless given the `-f` option, the filtering/publishing remembers roughly the last position\nand when restarted, the job starts from the last position. The information is retained in a file\n`\u003cname\u003e.mbtiles.SOME_NUMBERS`. When the mbtile file is replaced or the style is changed,\nthe `SOME_NUMBERS` change and a new full job is forced.\n\n## Differential upload\n\nThe S3 is billed by a access request; in order to minimize access costs, the program\nautomatically creates a file `\u003cname\u003e.mbtile.hashes`. When the publishing is complete, copy\nthe file manually to S3 to have the information available later. \nUpon next job restart (regardless if with or without the `-f` option), you can specify \nthe hash database with the `--hashes-db` parameter; only the changed tiles will be uploaded or deleted.\nA new hashes file will be created.\nThis should minimize costs upon country updates, when only a minority of the tiles is changed.\n\n## Performance considerations\n\n### Parallelism and RTS tuning\n\nThe `filter` and `publish` commands by default use as many cores as is available on the computer.\nHowever, sometimes this does not lead to better performance. You can limit the number of cores\nwith a special RTS (runtime system) command `-N`. It might be also beneficial to tune garbage \ncollector with the `-A` parameter; you may need to experiment with the settings.\n\nWhen publishing directly to S3, the bottleneck is usually the network; in such case it may be\nbetter to use higher parallelism to achieve higher throughput. The following command\nwill use 16 cores, 80 parallel threads and has an allocation unit set to 1 megabyte:\n\n```\n$ mapbox-filter publish -j openmaptiles.json.js -u https://xxx.cloudfront.net/w --rtl-convert -t s3://my-map-bucket/w osm-planet.mbtiles -p80 +RTS -N16 -A1m\n```\n\n### MD5 database tuning\n\nWhen publishing the data, a new database of md5 hashes is automatically created to aid with\ndifferential uploads. Unfortunately, the access to the database is serialized. Therefore,\nit might be best to run the job in ramdisk. On Linux, this would mean changing directory\nsomewhere to `tmpfs`, e.g. `/dev/shm`. Create a symlink to the original `mbtiles` file\n(e.g. `/dev/shm/world.mbtiles`) and then run the command in the `/dev/shm` directory.\nThe md5 database will be created on a ramdisk.\n\nAlternatively, SSD disk or some enterprise storage system with write cache\nmight be fast enough with more assurance in case of power loss. \n\n## What next\n\nThis started as a way to learn typechecking in Haskell and how to make a typed AST using GADTs.\nIt took about 1 day to make it work and it practically worked on the first try. Haskell is impressive.\nObviously since the first day a lot of functionality and better performance was added.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fondrap%2Fmapbox-filter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fondrap%2Fmapbox-filter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fondrap%2Fmapbox-filter/lists"}