{"id":22109290,"url":"https://github.com/pruvonet/promise-blocking-queue","last_synced_at":"2025-08-09T19:12:50.977Z","repository":{"id":35017994,"uuid":"194898780","full_name":"PruvoNet/promise-blocking-queue","owner":"PruvoNet","description":"Memory optimized promise blocking queue with concurrency control","archived":false,"fork":false,"pushed_at":"2023-06-20T09:09:09.000Z","size":327,"stargazers_count":4,"open_issues_count":10,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-28T19:09:36.821Z","etag":null,"topics":["async","await","batch","bluebird","concurrency","enqueue","fifo","job","limit","limited","priority","priorityqueue","promise","promises","queue","rate","ratelimit","task","throat","throttle"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PruvoNet.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-07-02T16:27:46.000Z","updated_at":"2023-06-20T08:49:15.000Z","dependencies_parsed_at":"2023-01-15T12:06:56.545Z","dependency_job_id":null,"html_url":"https://github.com/PruvoNet/promise-blocking-queue","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PruvoNet%2Fpromise-blocking-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PruvoNet%2Fpromise-blocking-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PruvoNet%2Fpromise-blocking-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PruvoNet%2Fpromise-blocking-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PruvoNet","download_url":"https://codeload.github.com/PruvoNet/promise-blocking-queue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227529007,"owners_count":17783972,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","await","batch","bluebird","concurrency","enqueue","fifo","job","limit","limited","priority","priorityqueue","promise","promises","queue","rate","ratelimit","task","throat","throttle"],"created_at":"2024-12-01T09:31:03.423Z","updated_at":"2024-12-01T09:31:14.073Z","avatar_url":"https://github.com/PruvoNet.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Npm Version](https://img.shields.io/npm/v/promise-blocking-queue.svg?style=popout)](https://www.npmjs.com/package/promise-blocking-queue)\n[![node](https://img.shields.io/node/v-lts/promise-blocking-queue)](https://www.npmjs.com/package/promise-blocking-queue)\n[![Build status](https://github.com/PruvoNet/promise-blocking-queue/actions/workflows/ci.yml/badge.svg?branch=master)](https://www.npmjs.com/package/promise-blocking-queue)\n[![Test Coverage](https://api.codeclimate.com/v1/badges/5cc9e9fe4871a315f2aa/test_coverage)](https://codeclimate.com/github/PruvoNet/promise-blocking-queue/test_coverage)\n[![Maintainability](https://api.codeclimate.com/v1/badges/5cc9e9fe4871a315f2aa/maintainability)](https://codeclimate.com/github/PruvoNet/promise-blocking-queue/maintainability)\n[![Known Vulnerabilities](https://snyk.io/test/github/PruvoNet/promise-blocking-queue/badge.svg?targetFile=package.json)](https://snyk.io/test/github/PruvoNet/promise-blocking-queue?targetFile=package.json)\n\n# Promise Blocking Queue\nMemory optimized promise blocking queue with concurrency control, specially designed to handle large data sets that must \nbe consumed using streams.  \n\nUseful for rate-limiting async (or sync) operations that consume large data sets. \nFor example, when interacting with a REST API or when doing CPU/memory intensive tasks.\n\n## Why\nIf we use `Bluebird.map()` for example, we are forced to load all the data in memory, \nbefore being able to consume it - Out Of Memory Exception is right around the corner.   \n\nIf we use [p-queue](https://github.com/sindresorhus/p-limit) (by the amazing [sindresorhus](https://github.com/sindresorhus)) \nfor example, we can utilize streams to avoid memory bloat, but we have no (easy) way to control \nthe stream flow without hitting that Out Of Memory Exception.\n\nThe solution - a blocking queue that returns a promise that will be resolved when the added item gains an available slot in the \nqueue, thus, allowing us to pause the stream consumption, until there is a _real_ need to consume the next item - keeping us \nmemory smart while maintaining concurrency level of data handling.\n\n## Install\n\n```shell\nnpm install promise-blocking-queue\n```\n\n## Usage example\n\nLet's assume we have a very large (a couple of GBs) file called `users.json` which contains a long list of users we want to add to our DB.  \nAlso, let's assume that our DB instance it very cheap, and as such we don't want to load it too much, so we only want to handle\n2 concurrent DB insert operations.  \n\nWe can achieve a short scalable solution like so:\n\n```typescript\nimport * as JSONStream from 'JSONStream';\nimport * as fs from 'fs';\nimport * as es from 'event-stream';\nimport * as sleep from 'sleep-promise';\nimport { BlockingQueue } from 'promise-blocking-queue';\n\nconst queue = new BlockingQueue({ concurrency: 2 });\nlet handled = 0;\nlet failed = 0;\nlet awaitDrain: Promise\u003cvoid\u003e | undefined;\n\nconst readStream = fs.createReadStream('./users.json', { flags: 'r', encoding: 'utf-8' });\nconst jsonReadStream = JSONStream.parse('*');\nconst jsonWriteStream = JSONStream.stringify();\nconst writeStream = fs.createWriteStream('./results.json');\n\nconst addUserToDB = async (user) =\u003e {\n    try {\n        console.log(`adding ${user.username}`);\n        // Simulate long running task\n        await sleep((handled + 1) * 100);\n        console.log(`added ${user.username} #${++handled}`);\n        const writePaused = !jsonWriteStream.write(user.username);\n        if (writePaused \u0026\u0026 !awaitDrain) {\n            // Down stream asked to pause the writes for now\n            awaitDrain = new Promise((resolve) =\u003e {\n                jsonWriteStream.once('drain', resolve);\n            });\n        }\n    } catch (err) {\n        console.log(`failed ${++failed}`, err);\n    }\n};\n\nconst handleUser = async (user) =\u003e {\n    // Wait until the down stream is ready to receive more data without increasing the memory footprint\n    if (awaitDrain) {\n        await awaitDrain;\n        awaitDrain = undefined;\n    }\n    return queue.enqueue(addUserToDB, user).enqueuePromise;\n};\n\n// Do not use async!\nconst mapper = (user, cb) =\u003e {\n    console.log(`streamed ${user.username}`);\n    handleUser(user)\n        .then(() =\u003e {\n            cb();\n        });\n    // Pause the read stream until we are ready to handle more data\n    return false;\n};\n\nconst onReadEnd = () =\u003e {\n    console.log('done read streaming');\n    // If nothing was written, idle event will not be fired\n    if (queue.pendingCount === 0 \u0026\u0026 queue.activeCount === 0) {\n        jsonWriteStream.end();\n    } else {\n        // Wait until all work is done\n        queue.on('idle', () =\u003e {\n            jsonWriteStream.end();\n        });\n    }\n};\n\nconst onWriteEnd = () =\u003e {\n    console.log(`done processing - ${handled} handled, ${failed} failed`);\n    process.exit(0);\n};\n\njsonWriteStream\n    .pipe(writeStream)\n    .on('error', (err) =\u003e {\n        console.log('error wrtie streaming', err);\n        process.exit(1);\n    })\n    .on('end', onWriteEnd)\n    .on('finish', onWriteEnd);\n\nreadStream\n    .pipe(jsonReadStream)\n    .pipe(es.map(mapper))\n    .on('data', () =\u003e {\n        // Do nothing\n    })\n    .on('error', (err) =\u003e {\n        console.log('error read streaming', err);\n        process.exit(1);\n    })\n    .on('finish', onReadEnd)\n    .on('end', onReadEnd);\n```\n\nIf `users.json` is like:\n\n```json\n[\n  {\n    \"username\": \"a\"\n  },\n  {\n    \"username\": \"b\"\n  },\n  {\n    \"username\": \"c\"\n  },\n  {\n    \"username\": \"d\"\n  }\n]\n```\n\nOutput will be:\n\n```bash\nstreamed a\nadding a\nstreamed b\nadding b\nstreamed c // c now waits in line to start and streaming is paused until then\nadded a #1\nadding c // c only gets handled after a is done\nstreamed d // d only get streamed after c has a spot in the queue\nadded b #2\nadding d // d only gets handled after b is done\ndone read streaming\nadded c #3\nadded d #4\ndone processing - 4 handled, 0 failed\n```\n\n`results.json` will be:\n\n```json\n[\n\"a\"\n,\n\"b\"\n,\n\"c\"\n,\n\"d\"\n]\n```\n\n## API\n\n### BlockingQueue(options)\n\nReturns a new `queue` instance, which is an `EventEmitter` subclass.\n\n#### options\n\nType: `object`\n\n##### concurrency\n\nType: `number`  \nDefault: `Infinity`  \nMinimum: `1`\n\nConcurrency limit.\n\n### queue\n\n`BlockingQueue` instance.\n\n#### .enqueue(fn, ...args)\n\nAdds a sync or async task to the queue\n\n##### Return value\n\nType: `object`\n\n###### enqueuePromise\n\nType: `Promise\u003cvoid\u003e`\n\nA promise that will be resolved when the queue has an available slot to run the task.  \nUsed to realize that it is a good time to add another task to the queue.\n\n###### fnPromise\n\nType: `Promise\u003cT\u003e`\n\nA promise that will be resolved with the result of `fn`.\n\n###### started\n\nType: `boolean`\n\nIndicates if the task has already started to run\n\n##### fn\n\nType: `Function`\n\nPromise/Value returning function.\n\n##### args\n\nType: `any[]`\n\nThe arguments to pass to the function\n\n#### activeCount\n\nThe number of promises that are currently running.\n\n#### pendingCount\n\nThe number of promises that are waiting to run.\n\n### Events\n\n#### empty\n\nEmitted when the queue becomes empty.\nUseful if, for example, you add additional items at a later time.\n\n#### idle\n\nEmitted when the queue becomes empty, and all promises have completed: `queue.activeCount === 0 \u0026\u0026 queue.pendingCount === 0`.\n\nThe difference with `empty` is that `idle` guarantees that all work from the queue has finished.\n`empty` merely signals that the queue is empty, but it could mean that some promises haven't completed yet.\n\n## Credits\nThe library is based on \n[p-limit](https://github.com/sindresorhus/p-limit) and [p-queue](https://github.com/sindresorhus/p-queue) (by the amazing [sindresorhus](https://github.com/sindresorhus))\n\n## Versions\n\nPromise Blocking Queue supports Node 6 LTS and higher.\n\n## Contributing\n\nAll contributions are happily welcomed!  \nPlease make all pull requests to the `master` branch from your fork and ensure tests pass locally.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpruvonet%2Fpromise-blocking-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpruvonet%2Fpromise-blocking-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpruvonet%2Fpromise-blocking-queue/lists"}