{"id":18413019,"url":"https://github.com/dkuku/tile_server","last_synced_at":"2025-04-07T11:32:17.191Z","repository":{"id":68616515,"uuid":"328047832","full_name":"dkuku/tile_server","owner":"dkuku","description":"implemented using elixir and phoenix","archived":false,"fork":false,"pushed_at":"2021-01-15T06:04:31.000Z","size":42,"stargazers_count":15,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-22T17:11:27.681Z","etag":null,"topics":["elixir","hakctoberfest","openstreetmap"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dkuku.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-09T01:09:05.000Z","updated_at":"2024-08-31T10:12:26.000Z","dependencies_parsed_at":"2023-02-23T03:31:32.190Z","dependency_job_id":null,"html_url":"https://github.com/dkuku/tile_server","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkuku%2Ftile_server","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkuku%2Ftile_server/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkuku%2Ftile_server/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkuku%2Ftile_server/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dkuku","download_url":"https://codeload.github.com/dkuku/tile_server/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247644626,"owners_count":20972332,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","hakctoberfest","openstreetmap"],"created_at":"2024-11-06T03:44:31.899Z","updated_at":"2025-04-07T11:32:17.184Z","avatar_url":"https://github.com/dkuku.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"Some background on mbtiles files from [mapbox/mbtiles-spec](https://github.com/mapbox/mbtiles-spec/blob/master/1.3/spec.md)\n\n\u003e MBTiles is a specification for storing tiled map data in SQLite databases for immediate usage and for transfer.\n\u003e ...\n\u003e The metadata table is used as a key/value store for settings. It MUST contain these two rows:\n\n\u003e name (string): The human-readable name of the tileset.\n\u003e format (string): The file format of the tile data: pbf, jpg, png, webp, or an IETF media type for other formats.\n\nWe can download an example file from [openmaptiles](https://openmaptiles.org/) or render some ourselves from openstreetmap data. In this tutorial I'll use a file has bundled the assets as `*.pbf` because it's the most complicated format to set up. With images you don't need any plugins for leaflet and the rendering should be much faster in user browser.\n\n `# sqlite3 priv/united_kingdom.mbtiles`\n ```   \n    After downloading a file we can have a look whats really inside:\n    sqlite\u003e .headers on\n    sqlite\u003e .tables\n    gpkg_contents  gpkg_tile_matrix    omtm    \n    gpkg_geometry_columns    gpkg_tile_matrix_setpackage_tiles\n    gpkg_metadata  images    tiles   \n    gpkg_metadata_reference  map\n    gpkg_spatial_ref_sysmetadata\n```\n    \nWe are interested in tiles and metadata tables:\n`sqlite\u003e select * from tiles limit 1;`\n\nzoom_level | tile_column | tile_row | tile_data\n--- | --- | --- | ---\n14    | 7763   | 10757    |\n    \n`sqlite\u003e select * from metadata where name is not 'json';`\n\nname   | value\n--- | ---\nattribution | \u003ca href=\"http://www.openmaptiles.org/\"\ncenter | -3.5793274999999998,55.070035000000004,14\ndescription | Extract from https://openmaptiles.org\nmaxzoom| 14\nminzoom| 0\nname   | OpenMapTiles\npixel_scale | 256\nmtime  | 1499626373833\nformat | pbf\nid| openmaptiles\nversion| 3.6.1\nmaskLevel   | 5\nbounds | -9.408655,49.00443,2.25,61.13564\nplanettime  | 1499040000000\nbasename    | europe_great-britain.mbtiles\n   \nThe tiles table keeps our tiles - these can be searched by row/column and zoom level and the metadata table has our info about this database.\n\nTo use it we will create a new phoenix project\n`mix phx.new tile_server --no-ecto --no-html --no-webpack`\nWe will skip the main libraries to simplify it a bit, the only addition we need is a library to connect with sqlite database:\n`{:sqlitex, \"~\u003e 1.7\"}`\nwe want to run it under superisor, for that we need to change the config.exs and application.ex\n```elixir\nconfig :tile_server, mbtiles_path: \"priv/united_kingdom.mbtiles\"\n```\n```elixir\nchildren = [\n   .....,\n %{\n   id: Sqlitex.Server,\n   start: {Sqlitex.Server, :start_link,\n [Application.get_env(:tile_server, :mbtiles_path), [name: TilesDB]]}\n }\n]\n```\nFirst thing we need to do is parse the metadata - in a new file `TileServer.Mbtiles` I'll create a function that does it\n```elixir\n  def get_metadata do\n    query = \"SELECT * FROM metadata\"\n\n    with {:ok, rows} \u003c- Sqlitex.Server.query(TilesDB, query) do\n Enum.reduce(rows, %{}, fn [name: name, value: value], acc -\u003e\n   Map.put(acc, String.to_atom(name), value)\n end)\n    else\n error -\u003e IO.inspect(error)\n    end\n  end\n```\nto check if its working we can display it on the homepage, the router needs to be modified:\n```elixir\n  pipeline :browser do\n    plug :accepts, [\"html\"]\n  end\n  scope \"/\", TileServerWeb do\n    pipe_through :browser\n    get \"/\", MapController, :index\n  end\n```\nand new map_controller.ex\n```elixir\ndefmodule TileServerWeb.MapController do\n  use TileServerWeb, :controller\n  alias TileServer.Mbtiles\n\n  def index(conn, _params) do\n    meta = Mbtiles.get_metadata()\n    text(conn, inspect(meta))\n  end\nend\n```\nOk, this works, other route is for getting the tiles, we need to create another function in our context:\n```elixir\n  def get_images(z, x, y) do\n    query =\n\"SELECT tile_data FROM tiles where zoom_level = ? and tile_column = ? and tile_row = ?\"\n\n    with {:ok, [data]} \u003c- Sqlitex.Server.query(TilesDB, query, bind: [z, x, y]),\n    [tile_data: tile_blob] \u003c- data,\n    {:blob, tile} \u003c- tile_blob, do: tile\n  end\n```\nHere we are using bind params to avoid sql injection attacks. We also are returning the data from the database without unpacking. To allow the client to process the compressed files we need to set a setting a content encoding header\n`{\"content-encoding\", \"gzip\"}`\n```elixir\n  def tile(conn, params) do\n    %{z: z, x: x, y: y} = parse_tile_params(params)\n\n    case Mbtiles.get_images(z, x, get_tms_y(z, y)) do\n :error -\u003e\n   conn |\u003e send_resp(404, \"tile not found\")\n\n tile -\u003e\n   conn\n   |\u003e prepend_resp_headers([{\"content-encoding\", \"gzip\"}])\n   |\u003e put_resp_content_type(\"application/octet-stream\")\n   |\u003e send_resp(200, tile)\n    end\n  end\n\n  defp get_tms_y(z, y), do: round(:math.pow(2, z) - 1 - y)\n\n  defp parse_tile_params(params) do\n    params\n    |\u003e Enum.map(fn {k, v} -\u003e {String.to_atom(k), String.to_integer(v)} end)\n    |\u003e Map.new()\n  end\n```\nThe other thing that may be weird here is the `get_tms_y/2` function: many of the mbtiles files are stored with the y coordinate in reverse order this can be easily seen if the tiles look weird on the map like completly not aligned between rows when displayed.\nLast puzzle piece is the router entry for our tiles\n```\n    get \"/\", MapController, :index\n    get \"/tiles/:z/:x/:y\", MapController, :tile\n```\nNow we are good to go, The only bottleneck I spotted is the the vector tiles need to be processed by javascript full screen needs processing power and this takes few seconds because the browser needs to process all the data. \nOtherwise phoenix does a good job here:\n```\n[info] GET /tiles/14/9050/5531   \n[info] Sent 200 in 3ms \n[info] GET /tiles/14/9048/5528\n[info] GET /tiles/14/9048/5530\n[info] GET /tiles/14/9049/5527\n[info] GET /tiles/14/9051/5527\n[info] Sent 200 in 3ms \n[info] Sent 200 in 4ms \n[info] Sent 200 in 4ms \n[info] Sent 200 in 5ms \n[info] GET /tiles/14/9051/5528   \n[info] Sent 200 in 6ms \n[info] GET /tiles/14/9048/5529   \n[info] GET /tiles/14/9049/5530   \n[info] GET /tiles/14/9052/5529   \n[info] GET /tiles/14/9050/5527   \n[info] GET /tiles/14/9051/5530   \n[info] Sent 200 in 3ms \n[info] Sent 200 in 4ms \n[info] Sent 200 in 4ms \n[info] Sent 200 in 5ms \n[info] Sent 200 in 5ms \n```\nThere is also an alternative to just unpack the files and serve it form the `/priv/static` folder.\nTo serve it this way we need [mb-util](https://github.com/mapbox/mbutil) app installed - The files are compressed, so we need either unpack them or rename to have `.gz` extension - then phoenix will serve it gzipped and add the header for us. \n```\n./mb-util --image_format=pbf countries.mbtiles static_tiles\n```\n```\n# add gz extension\nfind . -type f -exec mv {} {}.gz ';'\n\nor\n# unpack and store unpacked\ngzip -d -r -S .pbf *\nfind . -type f -exec mv '{}' '{}'.pbf \\;\n```\nThe endpoint.ex file needs to be modified:\n```elixir\n  plug Plug.Static,\n    at: \"/\",\n    only: ~w(css static_tiles fonts images js)\n```\nWhen playing with it I thought to myself that this may be also done using elixir and I created a mix task that does it. By running `mix mbtiles_unpack` in the repo directory the task will create a `priv/static/static_files` directory with a structure that can be served without any controllers.\n\nI packaged the backend code into a library, it can be installed using hex - source code is [here](https://github.com/dkuku/mbtiles) and also created a demo phoenix project that shows a how to use it.\nThe main files in phoenix are the map_controller and /priv/static/map_logic.js where I added the javascript part, all is done using [leaflet](https://leafletjs.com/) and the [vector grid plugin](https://github.com/Leaflet/Leaflet.VectorGrid). Demo repository can be found [tile_server](https://github.com/dkuku/tile_server)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdkuku%2Ftile_server","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdkuku%2Ftile_server","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdkuku%2Ftile_server/lists"}