Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/naturalatlas/tilestrata
A pluggable Node.js map tile server.
https://github.com/naturalatlas/tilestrata
maps tile-server
Last synced: 7 days ago
JSON representation
A pluggable Node.js map tile server.
- Host: GitHub
- URL: https://github.com/naturalatlas/tilestrata
- Owner: naturalatlas
- License: apache-2.0
- Created: 2014-12-12T14:58:13.000Z (about 10 years ago)
- Default Branch: master
- Last Pushed: 2021-07-21T19:26:45.000Z (over 3 years ago)
- Last Synced: 2025-02-01T05:12:42.906Z (14 days ago)
- Topics: maps, tile-server
- Language: JavaScript
- Homepage:
- Size: 318 KB
- Stars: 429
- Watchers: 20
- Forks: 42
- Open Issues: 8
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
- awesome-github-repos - naturalatlas/tilestrata - A pluggable Node.js map tile server. (JavaScript)
README
# TileStrata
[](https://www.npmjs.org/package/tilestrata)
[](https://travis-ci.org/naturalatlas/tilestrata)
[](https://codecov.io/github/naturalatlas/tilestrata)
*TileStrata is a pluggable "slippy map" tile server that emphasizes code-as-configuration.* The primary goal is painless extendability. It's clean, highly tested, performant, and integrates seamlessly with an elastic load balancer designed specifically for tile serving: [TileStrata Balancer](https://github.com/naturalatlas/tilestrata-balancer). Also, there's a built-in profiler and dashboard for debugging render times ([read more](#profiling--debugging-performance)).
```sh
$ npm install tilestrata --save
```
### Table of Contents
- [Introduction](#introduction)
- [Configuration](#configuration)
- [Basics](#configuration)
- [Express.js / Connect Integration](#integrate-with-expressjs--connect)
- [Metatile-Aware Load Balancing](#metatile-aware-load-balancing--layer-sharding)
- [Rebuilding the Tile Cache](#rebuilding-the-tile-cache)
- [Health Checks](#health-checks)
- [Profiling / Debugging Performance](#profiling--debugging-performance)
- [API Reference](#api-reference)
- [Plugin Developer Guide](#writing-tilestrata-plugins)
## Introduction
TileStrata consists of five main actors, usually implemented as plugins:
- [*"provider"*](#writing-providers) – Generates a new tile (e.g mapnik)
- [*"cache"*](#writing-caches) – Persists a tile for later requests (e.g. filesystem)
- [*"transform"*](#writing-transforms) – Takes a raw tile and transforms it (e.g. image scaling / compression)
- [*"request hook"*](#writing-request-hooks) – Called at the very beginning of a tile request.
- [*"response hook"*](#writing-response-hooks) – Called right before a tile is served to the client.
#### List of Plugins
- [tilestrata-mapnik](https://github.com/naturalatlas/tilestrata-mapnik) – Render tiles with [mapnik](http://mapnik.org/).
- [tilestrata-disk](https://github.com/naturalatlas/tilestrata-disk) – Cache map tiles to the filesystem (or serve from it)
- [tilestrata-dependency](https://github.com/naturalatlas/tilestrata-dependency) – Fetch tiles from other layers.
- [tilestrata-sharp](https://github.com/naturalatlas/tilestrata-sharp) – Compress, resize, transcode tiles (jpg, png, webp) using [libvips](https://www.npmjs.com/package/sharp).
- [tilestrata-gm](https://github.com/naturalatlas/tilestrata-gm) – Perform all sorts of image operations on tiles using [GraphicsMagick](https://www.npmjs.com/package/gm).
- [tilestrata-headers](https://github.com/naturalatlas/tilestrata-headers) – Set/override response headers.
- [tilestrata-blend](https://github.com/naturalatlas/tilestrata-blend) – Stack multiple layers together.
- [tilestrata-jsonp](https://github.com/naturalatlas/tilestrata-jsonp) – Serve utfgrids (and other JSON) as JSONP.
- [tilestrata-datadog](https://github.com/naturalatlas/tilestrata-datadog) – Send timing information to [Datadog](https://www.datadoghq.com/).
- [tilestrata-utfmerge](https://github.com/naturalatlas/tilestrata-utfmerge) – Merge UTF interactivity grids from mapnik.
- [tilestrata-vtile](https://github.com/naturalatlas/tilestrata-vtile) – Outputs mapnik vector tiles (protobufs).
- [tilestrata-vtile-raster](https://github.com/naturalatlas/tilestrata-vtile-raster) – Renders mapnik vector tiles into raster images.
- [tilestrata-vtile-composite](https://github.com/naturalatlas/tilestrata-vtile-composite) – Merge multiple vector tiles.
- [tilestrata-proxy](https://github.com/naturalatlas/tilestrata-proxy) – Fetches tiles from other servers
- [tilestrata-lru](https://github.com/naturalatlas/tilestrata-lru) – Caches tiles in memory (LRU)
- [tilestrata-etag](https://github.com/naturalatlas/tilestrata-etag) – Configurable Conditional GET support (with ETags)
- [tilestrata-bing](https://github.com/naturalatlas/tilestrata-bing) – Provider for Bing Maps tiles
- [tilestrata-underzoom](https://github.com/naturalatlas/tilestrata-underzoom) - Build mosaics of higher-zoom tiles
- [tilestrata-postgismvt](https://github.com/Stezii/tilestrata-postgismvt) – Outputs Mapbox Vector Tiles from a PostGIS database
- [tilestrata-postgis-geojson-tiles](https://github.com/naturalatlas/tilestrata-postgis-geojson-tiles) – Outputs GeoJSON tiles from a PostGIS database
## Configuration
```js
const tilestrata = require('tilestrata');
const disk = require('tilestrata-disk');
const sharp = require('tilestrata-sharp');
const mapnik = require('tilestrata-mapnik');
const dependency = require('tilestrata-dependency');
const strata = tilestrata();
// define layers
strata.layer('basemap')
.route('[email protected]')
.use(disk.cache({dir: '/var/lib/tiles/basemap'}))
.use(mapnik({
pathname: '/path/to/map.xml',
tileSize: 512,
scale: 2
}))
.route('tile.png')
.use(disk.cache({dir: '/var/lib/tiles/basemap'}))
.use(dependency('basemap', '[email protected]'))
.use(sharp(function(image, sharp) {
return image.resize(256);
}));
// start accepting requests
strata.listen(8080);
```
Once configured and started, tiles can be accessed via:
```
/:layer/:z/:x:/:y/:filename
```
### Routing Without Filenames
As of [2.1.0](https://github.com/naturalatlas/tilestrata/releases/tag/v2.1.0), if you desire a routing scheme that's closer to other tile servers (where there's no filename) like outlined in [#21](https://github.com/naturalatlas/tilestrata/pull/21), use the following format when registering routes:
```js
.route('*.png') // /layer/0/0/0.png
.route('*@2x.png') // /layer/0/0/[email protected]
```
### Integrate with [Express.js](http://expressjs.com/) / [Connect](https://github.com/senchalabs/connect)
TileStrata comes with middleware for Express that makes serving tiles from an existing application really simple, eliminating the need to call `listen` on `strata`.
```js
const tilestrata = require('tilestrata');
const strata = tilestrata();
strata.layer('basemap') /* ... */
strata.layer('contours') /* ... */
app.use(tilestrata.middleware({
server: strata,
prefix: '/maps'
}));
```
## Usage Notes
### Fault-Tolerant Initialization
By default, TileStrata will error when initializing if any of the layer handlers fail to initialize. If you would like to ignore errors so that _other_ layers are booted up and available, use the `skipFailures` option:
```js
var strata = tilestrata({ skipFailures: true });
```
### Metatile-Aware Load Balancing & Layer Sharding
TileStrata >= [2.0.0](https://github.com/naturalatlas/tilestrata/releases/tag/v2.0.0) supports integration with [TileStrata Balancer](https://github.com/naturalatlas/tilestrata-balancer), an elastic load balancer designed specifically for the nuances of tile serving – particularly [metatiles](http://wiki.openstreetmap.org/wiki/Meta_tiles). Generic load balancers have no knowledge of metatiles and thus will naively split tile requests out to multiple servers which leads to redundant rendering (slow and a waste of computing power).
As an added bonus, the balancer does not assume all servers in the pool have the same layers available. The balancer keeps track of the layers provided on each node so it knows where to route. In sum, the overview:
- **Fully elastic** w/minimal setup
- **Consistent routing** (improves local cache hits)
- **Metatile-aware** (prevents redundant rendering)
- **Layer-aware** (allows heterogeneous distribution of layers in the cluster)
[**View TileStrata Balancer Documentation →**](https://github.com/naturalatlas/tilestrata-balancer)
*Note: One could use cookie-persistence with traditional load balancers, but this forces users onto a single machine (not optimal).*
### Rebuilding the Tile Cache
If you update your map styles or data, you'll probably want to update your tiles. Rather than dump all of them at once and bring your tile server to a crawl, progressively rebuild the cache by requesting tiles with the `X-TileStrata-SkipCache` header. [TileMantle](https://github.com/naturalatlas/tilemantle) makes this process easy:
```
npm install -g tilemantle
tilemantle http://myhost.com/mylayer/{z}/{x}/{y}/t.png \
-p 44.9457507,-109.5939822 -b 30mi -z 10-14 \
-H "X-TileStrata-SkipCache:mylayer/t.png"
```
For the sake of the [tilestrata-dependency](https://github.com/naturalatlas/tilestrata-dependency) plugin, the value of the header is expected to be in the format:
```
X-TileStrata-SkipCache:*
X-TileStrata-SkipCache:[layer]/[file],[layer]/[file],...
```
In advanced use cases, it might be necessary for tiles to not be returned by the server until the cache is actually written (particularly when order matters due to dependencies). To achieve this, use:
```
X-TileStrata-CacheWait:1
```
### Health Checks
TileStrata includes a `/health` endpoint that will return a `200 OK` if it can accept connections. The response will always be JSON. By setting the `"healthy"` option to a function that accepts a callback you can take it a step further and control the status and data that it returns.
```js
// not healthy
const strata = tilestrata({
healthy: function(callback) {
callback(new Error('CPU is too high'), {loadavg: 3});
}
});
// healthy
const strata = tilestrata({
healthy: function(callback) {
callback(null, {loadavg: 1});
}
});
```
### Profiling / Debugging Performance
Unless the `TILESTRATA_NOPROFILE` environment variable is set, TileStrata keeps track of basic latency and size information (min, max, avg) for all steps in the tile serving flow for the lifetime of the process. Data is kept for every plugin on every route of every layer and is broken down by zoom level. To access it, visit: `/profile` in your browser. If this information needs to be kept private, you can set the `TILESTRATA_PASSWORD` environment variable to a password that TileStrata will prompt for (username is ignored). The page will have tables like the one below:
t.pngz1z2z3z4z5z6z7z8z9reqhook#0errors000000000dur_samples112111111dur_max454443465250586181dur_min454442465250586181dur_avg454442.5465250586181cache#0.geterrors000000000dur_samples112111111dur_max454544584762466053dur_min454544584762466053dur_avg454544584762466053hits000000000misses112111111provider#0errors000000000dur_samples112111111dur_max344396122119108115103129dur_min344364122119108115103129dur_avg344380122119108115103129size_samples112111111size_max500B501B576B578B504B540B501B776B736Bsize_min500B501B565B578B504B540B501B776B736Bsize_avg500B501B571B578B504B540B501B776B736Btransform#0errors000000000dur_samples112111111dur_max323335614957535069dur_min323334614957535069dur_avg323334.5614957535069reshook#0errors000000000dur_samples112111111dur_max454345636355486068dur_min454344636355486068dur_avg454344.5636355486068cache#0.seterrors000000000dur_samples112111111dur_max121313142723262927dur_min121310142723262927dur_avg121311.5142723262927
## API Reference
#### [TileServer](#tileserver)
##### server.listen(port, [hostname], [callback])
Starts accepting requests on the specified port. The arguments to this method are exactly identical to node's http.Server [listen()](http://nodejs.org/api/http.html#http_server_listen_port_hostname_backlog_callback) method. It returns the [http.Server](https://nodejs.org/api/http.html#http_class_http_server) instance.
##### server.close([callback])
Stops listening for incoming requests on the port. If [TileStrata Balancer](https://github.com/naturalatlas/tilestrata-balancer) is configured, it will also proactively notify it so that the node is removed from the pool.
##### server.layer(name, [opts])
Registers a new layer with the given name and returns its [TileLayer](#tilelayer) instance. If the layer already exists, the existing instance will be returned. Whatever name is used will be the first part of the url that can be used to fetch tiles: `/:layer/...`. The following options can be provided:
- **bbox**: A bounding box ([GeoJSON "bbox" format](http://geojson.org/geojson-spec.html#bounding-boxes)) that defines the valid extent of the layer. Any requests for tiles outside of this region will result in a 404 Not Found. This option can also be set to an array of bounding boxes for rare cases when a layer is noncontinuous.
- **minZoom**: The minimum `z` to return tiles for. Anything lesser will return a 404 Not Found.
- **maxZoom**: The maximum `z` to return tiles for. Anything greater will return a 404 Not Found.
##### server.getTile(layer, filename, x, y, z, callback)
Attempts to retrieve a tile from the specified layer (string). The callback will be invoked with three arguments: `err`, `buffer`, and `headers`.
##### server.uptime()
Returns an object containing "duration" and "start" (both in milliseconds). If the server hasn't started, the result will be `null`.
##### server.version
The version of TileStrata (useful to plugins, mainly).
#### [TileLayer](#tilelayer)
##### layer.route(filename, [options])
Registers a route and returns a [TileRequestHandler](#tilerequesthandler) instance to be configured. Setting `filename` to something like `"*.ext"` or `"*@2x.ext"` will omit the filename from the request and make the tiles available at `/{z}/{x}/{y}.ext` and `/{z}/{x}/{y}@2x.ext`, respectively (see [#21](https://github.com/naturalatlas/tilestrata/pull/21)).
The available options are:
- **cacheFetchMode**: Defines how cache fetching happens when multiple caches are configured. The mode can be `"sequential"` or `"race"`. If set to `"race"`, TileStrata will fetch from all caches simultaneously and return the first that wins.
#### [TileRequestHandler](#tilerequesthandler)
##### handler.use(plugin)
Registers a plugin, which is either a provider, cache, transform, request hook, response hook, or combination of them. See the READMEs on the prebuilt plugins and/or the ["Writing TileStrata Plugins"](#writing-tilestrata-plugins) section below for more info.
#### [TileRequest](#tilerequest)
A request contains these properties: `x`, `y`, `z`, `layer` (string), `filename`, `method`, `headers`, `qs`, and `hasFilename`.
If a tile request is in the filenameless format ([see here](#routing-without-filenames)), `hasFilename` will be `false`. To illustrate: if the request is to `/layer/0/0/[email protected]`, `filename` will be set to `[email protected]` (for compatibility with caches and plugins that expect a filename) and `hasFilename` will be `false`.
##### tile.clone()
Returns an identical copy of the tile request that's safe to mutate.
## Writing TileStrata Plugins
All plugins allow optional `init` and `destroy` lifecycle methods that will be called at startup and teardown. The first argument will be the [TileServer](#tileserver) instance, and the second will be the `callback`.
### Writing Request Hooks
A request hook implementation needs one method: `reqhook`. The hook's "req" will be a [http.IncomingMessage](http://nodejs.org/api/http.html#http_http_incomingmessage) and "res" will be the [http.ServerResponse](http://nodejs.org/api/http.html#http_class_http_serverresponse). This makes it possible to respond without even getting to the tile-serving logic (just don't call the callback).
```js
module.exports = function(options) {
return {
name: 'myplugin',
init: function(server, callback) {
callback(err);
},
reqhook: function(server, tile, req, res, callback) {
callback();
},
destroy: function(server, callback) {
callback(err);
}
};
};
```
### Writing Caches
A cache implementation needs two methods: `get`, `set`. If a cache fails (returns an error to the callback), the server will ignore the error and attempt to serve the tile from the registered provider.
```js
module.exports = function(options) {
return {
name: 'myplugin',
init: function(server, callback) {
callback(err);
},
get: function(server, tile, callback) {
callback(err, buffer, headers, /* refresh */);
},
set: function(server, tile, buffer, headers, callback) {
callback(err);
},
destroy: function(server, callback) {
callback(err);
}
};
};
```
A special behavior exists for when a cache returns a hit, but wants a new tile to be generated in the background. The use case: you have tile that's old enough it *should* be regenerated, but it's not old enough to warrant making the user wait for a new tile to be rendered. To accomplish this in a plugin, have `get()` return `true` as the fourth argument to the callback.
```js
callback(null, buffer, headers, true);
```
### Writing Providers
Providers are responsible for building tiles. A provider must define a `serve` method:
```js
module.exports = function(options) {
return {
name: 'myplugin',
init: function(server, callback) {
callback(err);
},
serve: function(server, tile, callback) {
callback(err, buffer, headers);
},
destroy: function(server, callback) {
callback(err);
}
};
};
```
### Writing Transforms
Transforms modify the result from a provider before it's served (and cached). A transform must define a `transform` method:
```js
module.exports = function(options) {
return {
name: 'myplugin',
init: function(server, callback) {
callback(err);
},
transform: function(server, tile, buffer, headers, callback) {
callback(err, buffer, headers);
},
destroy: function(server, callback) {
callback(err);
}
};
};
```
### Writing Response Hooks
A response hook implementation needs one method: `reshook`. The hook's "req" will be a [http.IncomingMessage](http://nodejs.org/api/http.html#http_http_incomingmessage) and "res" will be the [http.ServerResponse](http://nodejs.org/api/http.html#http_class_http_serverresponse). The "result" argument contains three properties: `headers`, `buffer`, and `status` — each of which can be modified to affect the final response.
```js
module.exports = function(options) {
return {
name: 'myplugin',
init: function(server, callback) {
callback(err);
},
reshook: function(server, tile, req, res, result, callback) {
callback();
},
destroy: function(server, callback) {
callback(err);
}
};
};
```
### Multi-Function Plugins
Sometimes a plugin must consist of multiple parts. For instance, a plugin tracking response times must register a request hook and response hook. To accommodate this, TileStrata supports arrays:
```js
module.exports = function() {
return [
{name: 'myplugin', reqhook: function(...) { /* ... */ }},
{name: 'myplugin', reshook: function(...) { /* ... */ }}
];
};
```
## Misc Utilities
### Concurrency Limiting
In some cases you'll want explicitly limit concurrency of a provider so that requests are queued if too many happen at once. To support this, we provide a `wrapWithMaxConcurrency` method that wraps providers. This is particularly useful with the [tilestrata-proxy](https://github.com/naturalatlas/tilestrata-proxy) plugin.
```js
const { wrapWithMaxConcurrency } = tilestrata.utils;
.use(wrapWithMaxConcurrency(provider(...), 5))
```
## Contributing
Before submitting pull requests, please update the [tests](test) and make sure they all pass.
```sh
$ npm test
```
## License
Copyright © 2014–2021 [Natural Atlas, Inc.](https://github.com/naturalatlas) & [Contributors](https://github.com/naturalatlas/tilestrata/graphs/contributors)
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at: http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.