https://github.com/almost/through2-concurrent

Simple Node.JS stream (streams2) Transform that runs the transform functions concurrently (with a set max concurrency)
https://github.com/almost/through2-concurrent

Last synced: 18 days ago
JSON representation

Simple Node.JS stream (streams2) Transform that runs the transform functions concurrently (with a set max concurrency)

• Host: GitHub
• URL: https://github.com/almost/through2-concurrent
• Owner: almost
• License: mit
• Created: 2014-10-10T11:27:14.000Z (about 10 years ago)
• Default Branch: master
• Last Pushed: 2020-08-11T20:24:18.000Z (over 4 years ago)
• Last Synced: 2024-11-14T02:05:29.735Z (28 days ago)
• Language: JavaScript
• Size: 25.4 KB
• Stars: 74
• Watchers: 5
• Forks: 11
• Open Issues: 4
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.txt

Awesome Lists containing this project

• awesome-nodejs-cn - through2-concurrent - 同时转换对象流 (包 / 文件流)
• awesome-nodejs - through2-concurrent - Simple Node.JS stream (streams2) Transform that runs the transform functions concurrently (with a set max concurrency). (Repository / Streams)
• awesome-nodejs-cn - through2-concurrent - **star:74** 同时转换对象流 (包 / 流)
• awesome-nodejs - through2-concurrent - Transform object streams concurrently. (Packages / Streams)
• awesome-nodejs - through2-concurrent - Simple Node.JS stream (streams2) Transform that runs the transform functions concurrently (with a set max concurrency) - ★ 60 (Streams)
• awesome-nodejs-streams - almost/through2-concurrent - Simple Node.JS stream (streams2) Transform that runs the transform functions concurrently (with a set max concurrency) (Modules)
• awesome-node - through2-concurrent - Transform object streams concurrently. (Packages / Streams)
• awesome-nodejs-cn - through2-concurrent - 同时转换对象流. (目录 / 流处理)

README

        
through2-concurrent

===================

[![NPM](https://nodei.co/npm/through2-concurrent.png?downloads&downloadRank)](https://nodei.co/npm/through2-concurrent/)

A simple way to create a Node.JS Transform stream which processes in

parallel. You can limit the concurrency (default is 16) and order is

*not* preserved (so chunks/objects can end up in a different order to

the order they started in if the transform functions take different

amounts of time).

Built using [through2](https://github.com/rvagg/through2) and has the

same API with the addition of a `maxConcurrency` option.

Non-`objectMode` streams are supported for completeness but I'm not

sure they'd be useful for anything.

Written by Thomas Parslow

([almostobsolete.net](http://almostobsolete.net) and

[tomparslow.co.uk](http://tomparslow.co.uk)) as part of Active Inbox

([activeinboxhq.com](http://activeinboxhq.com/)).

[![Build Status](https://travis-ci.org/almost/through2-concurrent.svg)](https://travis-ci.org/almost/through2-concurrent)

Install

-------

```bash

npm install --save through2-concurrent

```

Examples

--------

Process lines from a CSV in parallel. The order the results end up in

the `all` variable is not deterministic.

```javascript

var through2Concurrent = require('through2-concurrent');

var all = [];

fs.createReadStream('data.csv')

  .pipe(csv2())

  .pipe(through2Concurrent.obj(

    {maxConcurrency: 10},

    function (chunk, enc, callback) {

      var self = this;

      someThingAsync(chunk, function (newChunk) {

        self.push(newChunk);

        callback();

      });

  }))

  .on('data', function (data) {

    all.push(data)

  })

  .on('end', function () {

    doSomethingSpecial(all)

  })

```

Contributing

------------

Fixed or improved stuff? Great! Send me a pull request [through GitHub](http://github.com/almost/through2-concurrent)

or get in touch on Twitter [@almostobsolete][#tom-twitter] or email at [email protected]

[#tom]: http://www.almostobsolete.net

[#tom-twitter]: https://twitter.com/almostobsolete