https://github.com/lostintime/node-multiconsumer-queue
A wrapper to build multi-consumer queues on NodeJS
https://github.com/lostintime/node-multiconsumer-queue
nodejs pubsub queue
Last synced: about 1 month ago
JSON representation
A wrapper to build multi-consumer queues on NodeJS
- Host: GitHub
- URL: https://github.com/lostintime/node-multiconsumer-queue
- Owner: lostintime
- License: apache-2.0
- Created: 2017-12-06T16:43:11.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2018-02-21T16:18:52.000Z (over 8 years ago)
- Last Synced: 2025-10-11T08:28:02.190Z (9 months ago)
- Topics: nodejs, pubsub, queue
- Language: TypeScript
- Size: 21.5 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
MultiConsumer Queue
===================
A wrapper to build multi-consumer queues: tasks may be consumed by multiple processor groups, identified by an id.
## Implementations
* [MultiConsumer Kue](https://npmjs.com/package/multiconsumer-kue) - implementation using [Kue](https://www.npmjs.com/package/kue)
* [MultiConsumer Bull](https://npmjs.com/package/multiconsumer-bull) - implementation using [Bull](https://www.npmjs.com/package/bull)
## Design
Library provides few classes and defines a set of interface which basically helps dispatching
all jobs from input `Queue` to output `NamedQueue` using consumer `groupId`
as output topic name.
When new job processor is attached, specified `groupId` is added to a [`RedisLiveSet`](https://www.npmjs.com/package/redis-liveset)
which is synchronized across all nodes.
Ex: you're pushing you're submitting `doSomethingUseful` named kue tasks, and have 2 different processors: `log` and `save` -
wrapper will dispatch `doSomethingUseful` job data as `doSomethingUseful/log` and `doSomethingUseful/save` tasks
and will attach your handlers to those names.
Implementation consist of writing a `NamedQueue` for your queue backend:
```typescript
import * as kue from "kue"
import { NamedQueue, ProcessCallback } from "multiconsumer-queue"
import { createStringsLiveSet } from "redis-liveset"
export class KueNamedQueue implements NamedQueue {
constructor(private readonly _out: kue.Queue) {
}
add(name: string, data: any): void {
this._out.create(name, data).removeOnComplete(true).save()
}
process(name: string, fn: ProcessCallback, n: number = 1): void {
this._out.process(name, n, fn)
}
}
```
Then you can use it to new `MultiConsumerQueueImpl` instance,
which implements consumer groups synchronization routine:
```typescript
import * as kue from "kue"
import { EventBus, Queue, NamedQueue, MultiConsumerQueueImpl, NamedQueue, ProcessCallback } from "multiconsumer-queue"
export function createEventBus(queue: kue.Queue, redis: () => redis.RedisClient): EventBus {
return new EventBusImpl((topic: string) => {
const source: Queue = new NamedQueueWrap(topic, queue)
const out: NamedQueue = new DynamicallyNamedQueue(
(groupId: string) => `${topic}/${groupId}`,
new KueNamedQueue(queue)
)
return new MultiConsumerQueueImpl(
source,
out,
createStringsLiveSet(`queueConsumerGroups/${topic}`, redis(), redis()),
(job) => job.data // this function extracts data from input job, to be passed to output queues
)
})
}
```
And now you can use it:
```typescript
import * as kue from "kue"
import * as redis from "redis"
const bus = createEventBus(kue.createQueue(), () => redis.createClient())
// Process "my-topic" for logging
bus.topic("my-topic").process("log", (job, cb) => {
console.log("got new job in topic \"my-topic\" with data", job.data)
cb()
})
// Save all "my-topic" messages to database
bus.topic("my-topic").process("save", (job, cb) => {
console.log("here we're going to save all messages from \"my-topic\" to database")
cb()
})
bus.topic("my-topic").add("Hello World!")
```
NOTE: Wrapper implementation is not removing consumer groups from `RedisLiveQueue` so once you're
not interested anymore for processing topic messages for specific `groupId` -
you must remove that group and tasks manually
Group can be removed using `MultiConsumerQueueImpl.removeGroup()` method:
```typescript
const bus = createEventBus(...)
// deploy this to your servers to stop collecting tasks
bus.topic("my-topic").removeGroup("old-process-group")
```
You will still have to manually remove tasks already added for that group, or maybe those may expire,
this depends on how source `NamedQueue` is implemented.
## Contribute
> Perfection is Achieved Not When There Is Nothing More to Add,
> But When There Is Nothing Left to Take Away
Fork, Contribute, Push, Create pull request, Thanks.
## License
All code in this repository is licensed under the Apache License, Version 2.0. See [LICENCE](./LICENSE).