https://github.com/lostintime/node-multiconsumer-queue

A wrapper to build multi-consumer queues on NodeJS
https://github.com/lostintime/node-multiconsumer-queue

nodejs pubsub queue

Last synced: 6 days ago
JSON representation

A wrapper to build multi-consumer queues on NodeJS

• Host: GitHub
• URL: https://github.com/lostintime/node-multiconsumer-queue
• Owner: lostintime
• License: apache-2.0
• Created: 2017-12-06T16:43:11.000Z (about 7 years ago)
• Default Branch: master
• Last Pushed: 2018-02-21T16:18:52.000Z (almost 7 years ago)
• Last Synced: 2024-11-20T22:58:35.295Z (2 months ago)
• Topics: nodejs, pubsub, queue
• Language: TypeScript
• Size: 21.5 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
MultiConsumer Queue

===================

A wrapper to build multi-consumer queues: tasks may be consumed by multiple processor groups, identified by an id.

## Implementations

  * [MultiConsumer Kue](https://npmjs.com/package/multiconsumer-kue) - implementation using [Kue](https://www.npmjs.com/package/kue)

  * [MultiConsumer Bull](https://npmjs.com/package/multiconsumer-bull) - implementation using [Bull](https://www.npmjs.com/package/bull)

## Design

Library provides few classes and defines a set of interface which basically helps dispatching

 all jobs from input `Queue` to output `NamedQueue` using consumer `groupId`

 as output topic name.

When new job processor is attached, specified `groupId` is added to a [`RedisLiveSet`](https://www.npmjs.com/package/redis-liveset)

which is synchronized across all nodes.

Ex: you're pushing you're submitting `doSomethingUseful` named kue tasks, and have 2 different processors: `log` and `save` -

wrapper will dispatch `doSomethingUseful` job data as `doSomethingUseful/log` and `doSomethingUseful/save` tasks

and will attach your handlers to those names.

Implementation consist of writing a `NamedQueue` for your queue backend:

```typescript

import * as kue from "kue"

import { NamedQueue, ProcessCallback } from "multiconsumer-queue"

import { createStringsLiveSet } from "redis-liveset"

export class KueNamedQueue implements NamedQueue {

  constructor(private readonly _out: kue.Queue) {

  }

  add(name: string, data: any): void {

    this._out.create(name, data).removeOnComplete(true).save()

  }

  process(name: string, fn: ProcessCallback, n: number = 1): void {

    this._out.process(name, n, fn)

  }

}

```

Then you can use it to new `MultiConsumerQueueImpl` instance,

 which implements consumer groups synchronization routine:

```typescript

import * as kue from "kue"

import { EventBus, Queue, NamedQueue, MultiConsumerQueueImpl, NamedQueue, ProcessCallback } from "multiconsumer-queue"

export function createEventBus(queue: kue.Queue, redis: () => redis.RedisClient): EventBus {

  return new EventBusImpl((topic: string) => {

    const source: Queue = new NamedQueueWrap(topic, queue)

    const out: NamedQueue = new DynamicallyNamedQueue(

      (groupId: string) => `${topic}/${groupId}`,

      new KueNamedQueue(queue)      

    )

  

    return new MultiConsumerQueueImpl(

      source,

      out,

      createStringsLiveSet(`queueConsumerGroups/${topic}`, redis(), redis()),

      (job) => job.data // this function extracts data from input job, to be passed to output queues

    )

  })

}

```

And now you can use it:

```typescript

import * as kue from "kue"

import * as redis from "redis"

const bus = createEventBus(kue.createQueue(), () => redis.createClient())

// Process "my-topic" for logging

bus.topic("my-topic").process("log", (job, cb) => {

  console.log("got new job in topic \"my-topic\" with data", job.data)

  cb()

})

// Save all "my-topic" messages to database

bus.topic("my-topic").process("save", (job, cb) => {

  console.log("here we're going to save all messages from \"my-topic\" to database")

  cb()

})

bus.topic("my-topic").add("Hello World!")

```

NOTE: Wrapper implementation is not removing consumer groups from `RedisLiveQueue` so once you're

 not interested anymore for processing topic messages for specific `groupId` -

 you must remove that group and tasks manually

Group can be removed using `MultiConsumerQueueImpl.removeGroup()` method:

```typescript

const bus = createEventBus(...)

// deploy this to your servers to stop collecting tasks

bus.topic("my-topic").removeGroup("old-process-group")

```

You will still have to manually remove tasks already added for that group, or maybe those may expire, 

this depends on how source `NamedQueue` is implemented.

## Contribute

> Perfection is Achieved Not When There Is Nothing More to Add, 

> But When There Is Nothing Left to Take Away

Fork, Contribute, Push, Create pull request, Thanks. 

## License

All code in this repository is licensed under the Apache License, Version 2.0. See [LICENCE](./LICENSE).