{"id":17664436,"url":"https://github.com/tilfin/paced-work-stream","last_synced_at":"2026-05-19T10:34:24.661Z","repository":{"id":57317594,"uuid":"67903079","full_name":"tilfin/paced-work-stream","owner":"tilfin","description":"Node.js stream working at constant pace and concurrent","archived":false,"fork":false,"pushed_at":"2019-02-24T14:20:56.000Z","size":22,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-23T09:06:09.262Z","etag":null,"topics":["concurrent-processes","node-streams","nodejs","npm-module","promise-processing","stream"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tilfin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-09-11T01:18:36.000Z","updated_at":"2019-02-24T14:20:55.000Z","dependencies_parsed_at":"2022-08-25T19:50:50.188Z","dependency_job_id":null,"html_url":"https://github.com/tilfin/paced-work-stream","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tilfin%2Fpaced-work-stream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tilfin%2Fpaced-work-stream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tilfin%2Fpaced-work-stream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tilfin%2Fpaced-work-stream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tilfin","download_url":"https://codeload.github.com/tilfin/paced-work-stream/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246314150,"owners_count":20757463,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrent-processes","node-streams","nodejs","npm-module","promise-processing","stream"],"created_at":"2024-10-23T20:05:25.983Z","updated_at":"2026-05-19T10:34:24.614Z","avatar_url":"https://github.com/tilfin.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"PacedWorkStream\n===============\n\n[![NPM Version][npm-image]][npm-url]\n[![Node](https://img.shields.io/node/v/paced-work-stream.svg)]()\n[![Build Status](https://travis-ci.org/tilfin/paced-work-stream.svg?branch=master)](https://travis-ci.org/tilfin/paced-work-stream)\n[![Coverage Status](https://coveralls.io/repos/github/tilfin/paced-work-stream/badge.svg?branch=master)](https://coveralls.io/github/tilfin/paced-work-stream?branch=master)\n[![dependencies Status](https://david-dm.org/tilfin/paced-work-stream/status.svg)](https://david-dm.org/tilfin/paced-work-stream)\n\nNode.js transform stream working at constant pace and concurrent for object mode\n\n## Features\n\n* Work time at once can be specified. (workMS option)\n* Concurrent workers can be specified. (concurrency option)\n* Fires `done` event after when all workers have finished asynchrous -processes\n* Counting tag system to call `this.countTag(\u003ctag\u003e)` in `_workPromise`, you can get summarized results `tagCounts` grouped by tag.\n* Node.js 6.10 or later\n\n## Targets\n\n* API client that needs to handle the rate-limit\n* DB client that needs to handle the read/write capacity units like AWS DynamoDB\n\n## Install\n\n```\n$ npm install -save paced-work-stream\n```\n\n## How to Use\n\n### Create a PacedWorkStream\n\n**new PacedWorkStream(options, workPromise)**\n\n* `options` `\u003cObject\u003e`\n  * `concurrency` is the number of concurrent processes.\n  * `workMS` is milliseconds of work time at once that contains process-time and wait-time.\n  * `delay` is enable to start concurrent process in order delay for a time that divided workMS by concurrency, default is false. workPromise must return functions wrap each promise. Refer to the following figure for detailed operation pattern.\n  * `highWaterMark` is maximum object buffer size. If you use flow mode, you should set it at least concurrency.\n* `workPromise` is `function(item):` must return a _Promise_ processing the _item_ or a _Function_ that returns a _Promise_.\n\n![Delay Figure](https://github.com/tilfin/paced-work-stream/wiki/delay-figure.png)\n\n### Create subclass that extends PacedWorkStream\n\n* `super(options)` must be called in the constructor.\n* `_workPromise` method must be overrided and return a _Promise_ processing the _item_ or a _Function_ that returns a _Promise_.\n\n```javascript\nclass MyWorkStream extends PacedWorkStream {\n  constructor(options) {\n    super(options);\n  }\n  _workPromise(item) {\n    return () =\u003e {\n      this.countTag(item.tag);\n      return Promise.resolve(item.value);\n    };\n  }\n}\n```\n\n## Examples\n\n```javascript\nconst es = require('event-stream');\nconst devnull = require('dev-null');\nconst PacedWorkStream = require('paced-work-stream');\n\nconst pwStream = new PacedWorkStream({\n    concurrency: 2,\n    workMS: 1000,\n    highWaterMark: 5\n  }, function(item) {\n    console.log(new Date().toISOString(), 'Begin', item);\n\n    return new Promise((resolve, reject) =\u003e {\n        setTimeout(() =\u003e {\n          this.countTag('workDone');\n          console.log(new Date().toISOString(), 'End', item);\n          resolve();\n        }, 600); // workMS contains the time.\n      })\n  })\n  .on('done', function() {\n    console.log(this.tagCounts);\n  }).on('error', (err) =\u003e {\n    console.error(err);\n  });\n\nconst reader = es.readArray([11, 12, 21, 22, 31])\nreader.pipe(pwStream).pipe(devnull({ objectMode: true }));\n```\n\n* Pay attention to handling `done` event to get last `tagCounts` because workers haven't processed items on `finish` event.\n* If stream need not output, the stream must pipe dev-null.\n\n### Console output\n\n```\n$ node example.js\n2016-09-11T03:17:50.000Z Begin 11\n2016-09-11T03:17:50.003Z Begin 12\n2016-09-11T03:17:50.605Z End 11\n2016-09-11T03:17:50.605Z End 12\n2016-09-11T03:17:51.009Z Begin 21\n2016-09-11T03:17:51.009Z Begin 22\n2016-09-11T03:17:51.606Z End 21\n2016-09-11T03:17:51.606Z End 22\n2016-09-11T03:17:52.004Z Begin 31\n2016-09-11T03:17:52.607Z End 31\n{ workDone: 5 }\n```\n\n### Using with Promised Lifestream\n\n[Promised Lifestream](https://github.com/tilfin/promised-lifestream) is useful for stream pipeline. The following example gets the same result as above.\n\n```javascript\n'use strict';\n\nconst es = require('event-stream');\nconst PromisedLife = require('promised-lifestream');\n\nconst PacedWorkStream = require('paced-work-stream');\n\nconst pacedWorker = new PacedWorkStream({\n    concurrency: 2,\n    workMS: 1000,\n    highWaterMark: 5\n  }, function(item) {\n    console.log(new Date().toISOString(), 'Begin', item);\n\n    return new Promise((resolve, reject) =\u003e {\n        setTimeout(() =\u003e {\n          this.countTag('workDone');\n          console.log(new Date().toISOString(), 'End', item);\n          resolve();\n        }, 600); // workMS contains the time.\n      })\n  })\n\nPromisedLife([\n  es.readArray([11, 12, 21, 22, 31]),\n  pacedWorker\n])\n.then(() =\u003e {\n  console.log(pacedWorker.tagCounts);\n})\n.catch(err =\u003e {\n  console.error(err);\n});\n```\n\n\n## License\n\n  [MIT](LICENSE)\n\n[npm-image]: https://img.shields.io/npm/v/paced-work-stream.svg\n[npm-url]: https://npmjs.org/package/paced-work-stream\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftilfin%2Fpaced-work-stream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftilfin%2Fpaced-work-stream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftilfin%2Fpaced-work-stream/lists"}