{"id":21222217,"url":"https://github.com/lostintime/node-multiconsumer-queue","last_synced_at":"2026-05-12T16:04:52.751Z","repository":{"id":57305180,"uuid":"113342170","full_name":"lostintime/node-multiconsumer-queue","owner":"lostintime","description":"A wrapper to build multi-consumer queues on NodeJS","archived":false,"fork":false,"pushed_at":"2018-02-21T16:18:52.000Z","size":22,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-11T08:28:02.190Z","etag":null,"topics":["nodejs","pubsub","queue"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lostintime.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-12-06T16:43:11.000Z","updated_at":"2017-12-06T16:44:55.000Z","dependencies_parsed_at":"2022-08-27T02:15:42.616Z","dependency_job_id":null,"html_url":"https://github.com/lostintime/node-multiconsumer-queue","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/lostintime/node-multiconsumer-queue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lostintime%2Fnode-multiconsumer-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lostintime%2Fnode-multiconsumer-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lostintime%2Fnode-multiconsumer-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lostintime%2Fnode-multiconsumer-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lostintime","download_url":"https://codeload.github.com/lostintime/node-multiconsumer-queue/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lostintime%2Fnode-multiconsumer-queue/sbom","scorecard":{"id":599428,"data":{"date":"2025-08-11","repo":{"name":"github.com/lostintime/node-multiconsumer-queue","commit":"8b7c3399a690e6fb3afdb1d8ba2b8290041ee9e6"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":1.7,"checks":[{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Code-Review","score":0,"reason":"Found 0/12 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":0,"reason":"15 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-v6h2-p8h4-qcjw","Warn: Project is vulnerable to: GHSA-h6ch-v84p-w6p9","Warn: Project is vulnerable to: GHSA-2pr6-76vf-7546","Warn: Project is vulnerable to: GHSA-8j8c-7jfh-h6hx","Warn: Project is vulnerable to: GHSA-fvqr-27wr-82fm","Warn: Project is vulnerable to: GHSA-4xc9-xhrj-v574","Warn: Project is vulnerable to: GHSA-x5rq-j2xg-h7qm","Warn: Project is vulnerable to: GHSA-jf85-cpcp-j695","Warn: Project is vulnerable to: GHSA-p6mc-m468-83gw","Warn: Project is vulnerable to: GHSA-29mw-wpgm-hmr9","Warn: Project is vulnerable to: GHSA-35jh-r3h4-6jhm","Warn: Project is vulnerable to: GHSA-f8q6-p94x-37v3","Warn: Project is vulnerable to: GHSA-hj48-42vr-x3v9","Warn: Project is vulnerable to: GHSA-35q2-47q7-3pc3","Warn: Project is vulnerable to: GHSA-c2qf-rxjj-qqgw"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-21T00:01:21.931Z","repository_id":57305180,"created_at":"2025-08-21T00:01:21.931Z","updated_at":"2025-08-21T00:01:21.931Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32946447,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-12T09:19:52.626Z","status":"ssl_error","status_checked_at":"2026-05-12T09:17:33.438Z","response_time":102,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nodejs","pubsub","queue"],"created_at":"2024-11-20T22:42:11.719Z","updated_at":"2026-05-12T16:04:52.723Z","avatar_url":"https://github.com/lostintime.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"MultiConsumer Queue\n===================\n\nA wrapper to build multi-consumer queues: tasks may be consumed by multiple processor groups, identified by an id.\n\n\n## Implementations\n\n  * [MultiConsumer Kue](https://npmjs.com/package/multiconsumer-kue) - implementation using [Kue](https://www.npmjs.com/package/kue)\n  * [MultiConsumer Bull](https://npmjs.com/package/multiconsumer-bull) - implementation using [Bull](https://www.npmjs.com/package/bull)\n\n## Design\n\nLibrary provides few classes and defines a set of interface which basically helps dispatching\n all jobs from input `Queue\u003cJob\u003e` to output `NamedQueue\u003cJob\u003e` using consumer `groupId`\n as output topic name.\n\nWhen new job processor is attached, specified `groupId` is added to a [`RedisLiveSet\u003cstring\u003e`](https://www.npmjs.com/package/redis-liveset)\nwhich is synchronized across all nodes.\n\nEx: you're pushing you're submitting `doSomethingUseful` named kue tasks, and have 2 different processors: `log` and `save` -\nwrapper will dispatch `doSomethingUseful` job data as `doSomethingUseful/log` and `doSomethingUseful/save` tasks\nand will attach your handlers to those names.\n\nImplementation consist of writing a `NamedQueue\u003cJob\u003e` for your queue backend:\n\n```typescript\nimport * as kue from \"kue\"\nimport { NamedQueue, ProcessCallback } from \"multiconsumer-queue\"\nimport { createStringsLiveSet } from \"redis-liveset\"\n\nexport class KueNamedQueue implements NamedQueue\u003ckue.Job\u003e {\n  constructor(private readonly _out: kue.Queue) {\n  }\n\n  add(name: string, data: any): void {\n    this._out.create(name, data).removeOnComplete(true).save()\n  }\n\n  process(name: string, fn: ProcessCallback\u003ckue.Job\u003e, n: number = 1): void {\n    this._out.process(name, n, fn)\n  }\n}\n```\n\nThen you can use it to new `MultiConsumerQueueImpl\u003cJob\u003e` instance,\n which implements consumer groups synchronization routine:\n\n```typescript\nimport * as kue from \"kue\"\nimport { EventBus, Queue, NamedQueue, MultiConsumerQueueImpl, NamedQueue, ProcessCallback } from \"multiconsumer-queue\"\n\n\nexport function createEventBus(queue: kue.Queue, redis: () =\u003e redis.RedisClient): EventBus\u003ckue.Job\u003e {\n  return new EventBusImpl((topic: string) =\u003e {\n    const source: Queue\u003ckue.Job\u003e = new NamedQueueWrap(topic, queue)\n    const out: NamedQueue\u003ckue.Job\u003e = new DynamicallyNamedQueue(\n      (groupId: string) =\u003e `${topic}/${groupId}`,\n      new KueNamedQueue(queue)      \n    )\n  \n    return new MultiConsumerQueueImpl(\n      source,\n      out,\n      createStringsLiveSet(`queueConsumerGroups/${topic}`, redis(), redis()),\n      (job) =\u003e job.data // this function extracts data from input job, to be passed to output queues\n    )\n  })\n}\n```\n\nAnd now you can use it:\n\n```typescript\nimport * as kue from \"kue\"\nimport * as redis from \"redis\"\n\nconst bus = createEventBus(kue.createQueue(), () =\u003e redis.createClient())\n\n// Process \"my-topic\" for logging\nbus.topic(\"my-topic\").process(\"log\", (job, cb) =\u003e {\n  console.log(\"got new job in topic \\\"my-topic\\\" with data\", job.data)\n  cb()\n})\n\n// Save all \"my-topic\" messages to database\nbus.topic(\"my-topic\").process(\"save\", (job, cb) =\u003e {\n  console.log(\"here we're going to save all messages from \\\"my-topic\\\" to database\")\n  cb()\n})\n\nbus.topic(\"my-topic\").add(\"Hello World!\")\n```\n\nNOTE: Wrapper implementation is not removing consumer groups from `RedisLiveQueue` so once you're\n not interested anymore for processing topic messages for specific `groupId` -\n you must remove that group and tasks manually\n\nGroup can be removed using `MultiConsumerQueueImpl.removeGroup()` method:\n\n```typescript\n\nconst bus = createEventBus(...)\n\n// deploy this to your servers to stop collecting tasks\nbus.topic(\"my-topic\").removeGroup(\"old-process-group\")\n\n```\n\nYou will still have to manually remove tasks already added for that group, or maybe those may expire, \nthis depends on how source `NamedQueue` is implemented.\n\n\n## Contribute\n\n\u003e Perfection is Achieved Not When There Is Nothing More to Add, \n\u003e But When There Is Nothing Left to Take Away\n\nFork, Contribute, Push, Create pull request, Thanks. \n\n\n## License\n\nAll code in this repository is licensed under the Apache License, Version 2.0. See [LICENCE](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flostintime%2Fnode-multiconsumer-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flostintime%2Fnode-multiconsumer-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flostintime%2Fnode-multiconsumer-queue/lists"}