{"id":13426183,"url":"https://github.com/mapbox/tile-reduce","last_synced_at":"2025-03-15T21:30:45.841Z","repository":{"id":29770996,"uuid":"33314938","full_name":"mapbox/tile-reduce","owner":"mapbox","description":"mapreduce vector tile processing","archived":false,"fork":false,"pushed_at":"2021-04-23T12:19:40.000Z","size":7132,"stargazers_count":185,"open_issues_count":9,"forks_count":32,"subscribers_count":135,"default_branch":"master","last_synced_at":"2025-03-09T01:32:01.479Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mapbox.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-04-02T15:01:28.000Z","updated_at":"2025-02-26T08:55:19.000Z","dependencies_parsed_at":"2022-09-06T08:52:09.617Z","dependency_job_id":null,"html_url":"https://github.com/mapbox/tile-reduce","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mapbox%2Ftile-reduce","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mapbox%2Ftile-reduce/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mapbox%2Ftile-reduce/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mapbox%2Ftile-reduce/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mapbox","download_url":"https://codeload.github.com/mapbox/tile-reduce/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243792287,"owners_count":20348609,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T00:01:28.362Z","updated_at":"2025-03-15T21:30:45.168Z","avatar_url":"https://github.com/mapbox.png","language":"JavaScript","readme":"# TileReduce\n\n[![Build Status](https://travis-ci.org/mapbox/tile-reduce.svg)](https://travis-ci.org/mapbox/tile-reduce)\n\nTileReduce is a geoprocessing library that implements [MapReduce](http://en.wikipedia.org/wiki/MapReduce) to let you run scalable distributed spatial analysis using [JavaScript](http://nodejs.org/) and [Mapbox Vector Tiles](https://www.mapbox.com/developers/vector-tiles/). TileReduce coordinates tasks across all available processors on a machine, so your analysis runs lightning fast.\n\n## Install\n\n```sh\nnpm install @mapbox/tile-reduce\n```\n\n## Usage\n\nA TileReduce processor is composed of two parts; the \"map\" script and the \"reduce\" script. The \"map\" portion comprises the expensive processing you want to distribute, while the \"reduce\" script comprises the quick aggregation step.\n\n### 'map' script\n\nThe map script operates on each individual tile. It's purpose is to receive one tile at a time, do analysis or processing on the tile, and write data and send results to the reduce script.\n\n[See the count example processor's map script](https://github.com/mapbox/tile-reduce/blob/master/examples/count/count.js)\n\n### 'reduce' script\n\nThe reduce script serves both to initialize TileReduce with job options, and to handle reducing results returned by the map script for each tile.\n\n[See the count example processor's reduce script](https://github.com/mapbox/tile-reduce/blob/master/examples/count/index.js)\n\n\n## Options\n\n### Basic Options\n\n#### zoom (required)\n\n`zoom` specifies the zoom level of tiles to retrieve from each source.\n\n```js\ntilereduce({\n\tzoom: 15,\n\t// ...\n})\n```\n\n#### map (required)\n\nPath to the map script, which will be executed against each tile\n\n```js\ntilereduce({\n\tmap: path.join(__dirname, 'map.js')\n\t// ...\n})\n```\n#### maxWorkers\n\nBy default, TileReduce creates one worker process per CPU. `maxWorkers` may be used to limit the number of workers created\n\n```js\ntilereduce({\n  maxWorkers: 3,\n  // ...\n})\n```\n\n#### output\n\nBy default, any data written from workers is piped to `process.stdout` on the main process. You can pipe to an alternative writable stream using the `output` option.\n\n```js\ntilereduce({\n\toutput: fs.createWriteStream('output-file'),\n\t// ...\n})\n```\n\n#### log\n\nDisables logging and progress output\n\n```js\ntilereduce({\n\tlog: false,\n\t// ...\n})\n```\n\n#### mapOptions\n\nPasses through arbitrary options to workers. Options are made available to map scripts as `global.mapOptions`\n\n```js\ntilereduce({\n\tmapOptions: {\n\t\tbufferSize: 4\n\t}\n\t// ...\n})\n```\n\n```js\n// map.js\nmodule.exports = function (sources, tile, write, done) {\n  global.mapOptions.bufferSize; // = 4\n};\n```\n\n### Specifying Sources (required)\n\nSources are specified as an array in the `sources` option:\n\n```js\ntilereduce({\n\tsources: [\n\t\t/* source objects */\n\t],\n\t// ...\n})\n```\n\n#### MBTiles sources:\n\n```js\ntilereduce({\n    sources: [\n      {\n        name: 'osmdata',\n        mbtiles: __dirname+'/latest.planet.mbtiles',\n        layers: ['osm']\n      }\n    ]\n})\n```\n\n[MBTiles](https://github.com/mapbox/mbtiles-spec) work well for optimizing tasks that request many tiles, since the data is stored on disk. Create your own MBTiles from vector data using [tippecanoe](https://github.com/mapbox/tippecanoe), or use [OSM QA Tiles](http://osmlab.github.io/osm-qa-tiles/), a continuously updated MBTiles representation of OpenStreetMap.\n\n#### URL\n\nRemote Vector Tile sources accessible over HTTP work well for mashups of datasets and datasets that would not be practical to fit on a single machine. Be aware that HTTP requests are slower than mbtiles, and throttling is typically required to avoid disrupting servers at high tile volumes. `maxrate` dictates how many requests per second will be made to each remote source.\n\n```js\nsources: [\n  {\n    name: 'streets',\n    url: 'https://b.tiles.mapbox.com/v4/mapbox.mapbox-streets-v6/{z}/{x}/{y}.vector.pbf',\n    layers: ['roads'],\n    maxrate: 10\n  }\n]\n```\n\n#### raw\n\nBy default, sources will be automatically converted from their raw Vector Tile representation to GeoJSON. If you set `raw: true` in an MBTiles or URL source, the [raw Vector Tile data](https://github.com/mapbox/vector-tile-js) will be provided, allowing you to lazily parse features as needed. This is useful in some situations for maximizing performance.\n\n```js\nsources: [\n  {\n    name: 'streets',\n    url: 'https://b.tiles.mapbox.com/v4/mapbox.mapbox-streets-v6/{z}/{x}/{y}.vector.pbf',\n    raw: true\n  }\n]\n```\n\n---\n\n### Specifying Job Area\n\nJobs run over a geographic region represented by a set of tiles. TileReduce also accepts several area definitions that will be automatically converted into tiles.\n\n#### BBOX\n\nA valid [bounding box](http://geojson.org/geojson-spec.html#bounding-boxes) array.\n\n```js\ntilereduce({\n\tbbox: [w, s, e, n],\n\t// ...\n})\n```\n\n#### GeoJSON\n\nA valid [GeoJSON geometry](http://geojson.org/geojson-spec.html#geojson-objects) of any type.\n\n```js\ntilereduce({\n\tgeojson: {\"type\": \"Polygon\", \"coordinates\": [/* coordinates */]},\n\t// ...\n})\n```\n\n#### Tile Array\n\nAn array of [quadtiles](https://msdn.microsoft.com/en-us/library/bb259689.aspx) represented as xyz arrays.\n\n```js\ntilereduce({\n\ttiles: [\n\t\t[x, y, z]\n\t],\n\t// ...\n})\n```\n\n#### Tile Stream\n\nTiles can be read from an object mode [node stream](https://nodejs.org/api/stream.html). Each object in the stream should be either a string in the format `x y z` or an array in the format `[x, y, z]`.\n\n```js\ntilereduce({\n\ttileStream: /* an object mode node stream */,\n\t// ...\n})\n```\n\nLine separated tile list files can easily be converted into the appropriate object mode streams using [binary-split](https://github.com/maxogden/binary-split):\n\n```js\nvar split = require('binary-split'),\n\tfs = require('fs');\n\ntilereduce({\n\ttileStream: fs.createReadStream('/path/to/tile-file').pipe(split()),\n\t// ...\n})\n```\n\n#### Source Cover\n\nWhen using MBTiles sources, a list of tiles to process can be automatically retrieved from the source metadata\n\n```js\ntilereduce({\n\tsourceCover: 'osmdata',\n\tsources: [\n\t\t{\n\t\t\tname: 'osmdata',\n\t\t\tmbtiles: __dirname+'/latest.planet.mbtiles'\n\t\t}\n\t]\n\t// ...\n})\n```\n\n## Events\n\nTileReduce returns an [EventEmitter]().\n\n### start\n\nFired once all workers are initialized and before the first tiles are sent for processing\n\n```js\ntilereduce({/* ... */})\n.on('start', function () {\n\tconsole.log('starting');\n});\n```\n\n### map \n\nFired just before a tile is sent to a worker. Receives the tile and worker number assigned to process the tile.\n\n```js\ntilereduce({/* ... */})\n.on('map', function (tile, workerId) {\n\tconsole.log('about to process ' + JSON.stringify(tile) +' on worker '+workerId);\n});\n```\n\n### reduce \n\nFired when a tile has finished processing. Receives data returned in the map function's `done` callback (if any), and the tile.\n\n```js\nvar count = 0;\ntilereduce({/* ... */})\n.on('reduce', function (result, tile) { \n\tconsole.log('got a count of ' + result + ' from ' + JSON.stringify(tile));\n\tcount++;\n});\n```\n\n### end\n\nFired when all queued tiles have been processed. Use this event to output final reduce results.\n\n```js\nvar count = 0;\ntilereduce({/* ... */})\n.on('end', function () {\n\tconsole.log('Total count was: ' + count);\n});\n```\n\n## Processor Examples\n\n- [osm-coverage](https://github.com/mapbox/osm-coverage) - a processor for computing statistics about [OpenStreetMap](http://www.openstreetmap.org/) coverage across countries.\n\n- [osm-sidewalker](https://github.com/mapbox/osm-sidewalker) - a processor for detecting potentially untagged sidewalks in [OpenStreetMap](http://www.openstreetmap.org/).\n\n## Development\n\n### Testing\n\n```sh\nnpm test\n```\n\n### Linting\n\n```sh\nnpm run lint\n```\n\n### Test Coverage\n\n```sh\nnpm run cover\n```\n","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmapbox%2Ftile-reduce","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmapbox%2Ftile-reduce","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmapbox%2Ftile-reduce/lists"}