https://github.com/devtin/stream-flow-async

A utility wrapper to asynchronously post-process data from a Readable stream that controls the stream flow to a maximum amount of concurrent un-resolved async operations in order to avoid back-pressuring
https://github.com/devtin/stream-flow-async

Last synced: 25 days ago
JSON representation

A utility wrapper to asynchronously post-process data from a Readable stream that controls the stream flow to a maximum amount of concurrent un-resolved async operations in order to avoid back-pressuring

• Host: GitHub
• URL: https://github.com/devtin/stream-flow-async
• Owner: devtin
• Created: 2021-09-06T16:33:01.000Z (over 4 years ago)
• Default Branch: development
• Last Pushed: 2021-09-06T16:36:41.000Z (over 4 years ago)
• Last Synced: 2025-11-28T12:55:24.232Z (4 months ago)
• Language: HTML
• Size: 53.7 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# stream-flow-async

A utility wrapper to asynchronously post-process data from a Readable stream that controls the stream flow to a maximum

amount of concurrent un-resolved async operations in order to avoid

back-pressuring.

## Usage

```js

const fs = require('fs');

const { streamFlowAsync } = require('stream-flow-async');

const db = require('./my-db-conntection.js');

const stream = fs.createReadStream('emails.txt');

const saveInDb = (email) => {

  return db.collection('emails').insertOne({ email })

}

streamFlowAsync({

    stream, // the stream

    handler: saveInDb, // an async function handler that will receive each chunk

    flow: 10 // maximum amount of concurrent un-resolved async functions

  })

  .then(() => {

    console.log('all saved!')

  })

```

### An example using csv-parser

```js

const fs = require('fs');

const csv = require('csv-parser');

const db = require('./my-db-conntection.js');

const stream = fs.createReadStream('millions-of-contacts.csv').pipe(csv());

const saveInDb = (contact) => {

  return db.collection('contacts').insertOne(contact)

}

streamFlowAsync({

    stream,

    handler: saveInDb, // receives each csv line parsed

    flow: 100

  })

  .then(() => {

    console.log('all saved!')

  })

```