{"id":16701792,"url":"https://github.com/michaelnisi/speculum","last_synced_at":"2026-05-20T02:52:09.054Z","repository":{"id":33938923,"uuid":"37663597","full_name":"michaelnisi/speculum","owner":"michaelnisi","description":"Transform concurrently","archived":false,"fork":false,"pushed_at":"2017-01-24T16:08:12.000Z","size":28,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-05-02T05:35:43.978Z","etag":null,"topics":["concurrent-streams","nodejs","transform-streams"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michaelnisi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-06-18T14:16:43.000Z","updated_at":"2020-05-19T03:11:10.000Z","dependencies_parsed_at":"2022-07-13T16:48:14.013Z","dependency_job_id":null,"html_url":"https://github.com/michaelnisi/speculum","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelnisi%2Fspeculum","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelnisi%2Fspeculum/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelnisi%2Fspeculum/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelnisi%2Fspeculum/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michaelnisi","download_url":"https://codeload.github.com/michaelnisi/speculum/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243538121,"owners_count":20307101,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrent-streams","nodejs","transform-streams"],"created_at":"2024-10-12T18:45:40.653Z","updated_at":"2026-05-20T02:52:04.036Z","avatar_url":"https://github.com/michaelnisi.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://secure.travis-ci.org/michaelnisi/speculum.svg)](http://travis-ci.org/michaelnisi/speculum)\n[![Coverage Status](https://coveralls.io/repos/github/michaelnisi/speculum/badge.svg?branch=master)](https://coveralls.io/github/michaelnisi/speculum?branch=master)\n\n# speculum - transform concurrently\n\n**speculum** is a stream multiplier. This [Node.js](https://nodesjs.org) package implements an interface that wraps multiple transform streams into a single, concurrently processing, [stream.Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform). In use cases where result order is not paramount, **speculum** can reduce run time.\n\nAn IO-heavy transform stream’s run time *T* grows linearly with the number *N* of chunks (units of IO work) *C*:\n\n*T = N * C*\n\n**speculum** divides the run time by the number of concurrent streams *X*:\n\n*T = N * C / X*\n\nSo far for theory.\n\n## Example\n\nHere is a, somewhat contrived but runnable, example comparing the run times from single to ten concurrent streams:\n\n```js\n'use strict'\n\n// example - measure one to ten concurrent streams\n\nconst assert = require('assert')\nconst speculum = require('speculum')\nconst stream = require('stream')\nconst util = require('util')\n\n// A transform that does asynchronous work.\nutil.inherits(Echo, stream.Transform)\nfunction Echo (opts) {\n  if (!(this instanceof Echo)) {\n    return new Echo(opts)\n  }\n  stream.Transform.call(this, opts)\n}\nEcho.prototype._transform = function (chunk, enc, cb) {\n  setTimeout(() =\u003e {\n    this.push(chunk)\n    cb()\n  }, 100)\n}\n\n// An input stream to read from.\nutil.inherits(Count, stream.Readable)\nfunction Count (opts, max) {\n  stream.Readable.call(this, opts)\n  this.count = 0\n  this.max = max\n}\nCount.prototype._read = function () {\n  let ok = false\n  do {\n    ok = this.push(String(this.count++))\n  } while (this.count \u003c this.max \u0026\u0026 ok)\n  if (this.count \u003e= this.max) {\n    this.push(null)\n  }\n}\n\n// Leverage x streams to transform, delayed echoing in this example, data from\n// our readable stream.\nfunction run (x, cb) {\n  const s = speculum(null, Echo, x)\n  s.on('end', cb)\n  s.on('error', cb)\n\n  const reader = new Count({ highWaterMark: 0 }, 10)\n\n  reader.pipe(s).resume()\n}\n\nfunction measure (x, cb) {\n  function time (t) {\n    return t[0] * 1e9 + t[1]\n  }\n  const t = process.hrtime()\n  run(x, (er) =\u003e {\n    const lat = time(process.hrtime(t))\n    console.log(x + ' X took ' + (lat / 1e6).toFixed(2) + ' ms')\n    cb(er)\n  })\n}\n\n(function go (max, x = 1) {\n  if (x \u003e max) return\n  measure(x, (er) =\u003e {\n    assert(!er, er)\n    go(max, x + 1)\n  })\n})(10)\n```\n\nYou can run this with:\n\n```\n$ node example.js\n```\n\nOn this MacBook Air (11-inch, Mid 2011), with Node v6.7.0, I get:\n\n```\n1 X took 1088.49 ms\n2 X took 525.67 ms\n3 X took 411.52 ms\n4 X took 317.99 ms\n5 X took 210.97 ms\n6 X took 210.99 ms\n7 X took 210.68 ms\n8 X took 210.24 ms\n9 X took 211.61 ms\n10 X took 104.25 ms\n```\n\n## Considerations\n\nClearly, we have to balance workload and overhead to use this efficiently. Specifically, we need an idea of how many chunks our stream may need to process, before we can choose an effective number of concurrent streams. But efficiency, of course, varies depending on duration and consistence of the work to be done inside our multiplied stream.\n\n**speculum** is a good fit if you want to reduce run time by leveraging existing transform streams concurrently, not minding unordered output. In other use cases, where you might be writing a concurrent transform stream from scratch, try [throughv](https://github.com/mcollina/throughv).\n\n## Exports\n\n### speculum(opts, create, x = 1)\n\n- `opts` `Object() | null | undefined` Options passed to the stream constructor.\n- `create` `function` A constructor function applied to create transform streams.\n- `x` `Number() | null | undefined` The number of concurrent transform streams defaults to one.\n\nThe **speculum** module exports a function that returns an instance of the `Speculum` class which extends `stream.Transform`. To access the `Speculum` class `require('speculum')`. The **speculum** stream round-robins transformers, constructed by `create`, and exposes their buffers and errors through its `stream.Readable` interface.\n\n## Installation\n\nWith [npm](https://npmjs.org/package/speculum), do:\n\n```\n$ npm install speculum\n```\n\n## License\n\n[MIT License](https://raw.github.com/michaelnisi/speculum/master/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaelnisi%2Fspeculum","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichaelnisi%2Fspeculum","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaelnisi%2Fspeculum/lists"}