https://github.com/calebboyd/redis-x-stream

An AsyncIterable interface for reading redis streams
https://github.com/calebboyd/redis-x-stream

async consumer iterable javascript job-queue nodejs nodejs-library queue redis redis-queue streams task-queue typescript typescript-library worker-queue

Last synced: 2 months ago
JSON representation

An AsyncIterable interface for reading redis streams

• Host: GitHub
• URL: https://github.com/calebboyd/redis-x-stream
• Owner: calebboyd
• License: mit
• Created: 2020-10-05T01:14:23.000Z (over 4 years ago)
• Default Branch: main
• Last Pushed: 2023-08-06T13:09:34.000Z (almost 2 years ago)
• Last Synced: 2024-11-02T09:51:35.855Z (8 months ago)
• Topics: async, consumer, iterable, javascript, job-queue, nodejs, nodejs-library, queue, redis, redis-queue, streams, task-queue, typescript, typescript-library, worker-queue
• Language: TypeScript
• Homepage:
• Size: 1.67 MB
• Stars: 9
• Watchers: 3
• Forks: 4
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# redis-x-stream

Create async iterables that emit redis stream entries. Requires Redis 5 or greater.

[![release](https://badgen.net/github/release/calebboyd/redis-x-stream)](https://www.npmjs.com/package/redis-x-stream)

[![license](https://badgen.net/badge/license/MIT/blue)](https://github.com/calebboyd/redis-x-stream/blob/main/LICENSE)

[![test](https://github.com/calebboyd/redis-x-stream/actions/workflows/test.yml/badge.svg)](https://github.com/calebboyd/redis-x-stream/actions)

## Getting Started

```javascript

import { RedisStream } from 'redis-x-stream'

import Redis from 'ioredis'

const myStream = 'my-stream'

await populate(myStream, 1e5)

let i = 0

for await (const [streamName, [id, keyvals]] of new RedisStream(myStream)) {

  i++;

}

console.log(`read ${i} stream entries from ${myStream}`)

async function populate(stream, count) {

  const writer = new Redis({ enableAutoPipelining: true })

  await Promise.all(

    Array.from(Array(count), (_, j) => writer.xadd(stream, '*', 'index', j))

  )

  writer.quit()

  await new Promise(resolve => writer.once('close', resolve))

  console.log(`wrote ${count} stream entries to ${stream}`)

}

```

## Usage

See the [API Docs](docs/classes/RedisStream.md#constructor) for available options.

## Advanced Usage

### Task Processing

If you have a cluster of processes reading redis stream entries you likely want to utilize redis consumer groups

A task processing application may look like the following:

```javascript

const control = {

  /* some control event emitter */

}

const stream = new RedisStream({

  streams: ['my-stream'],

  group: 'my-group',

  //eg. k8s StatefulSet hostname. or Cloud Foundry instance index

  consumer: 'tpc_' + process.env.SOME_ORDINAL_IDENTIFIER,

  block: Infinity,

  count: 10,

  deleteOnAck: true,

})

const lock = new Semaphore(11)

const release = lock.release.bind(lock)

control.on('new-source', (streamName) => {

  //Add an additional source stream to a blocked stream.

  stream.addStream(streamName)

})

control.on('shutdown', async () => {

  //drain will process all claimed entries (the PEL) and stop iteration

  await stream.drain()

})

async function tryTask(stream, streamName, id, entry) {

  //...process entry...

  stream.ack(streamName, id)

}

for await (const [streamName, [id, keyvals]] of stream) {

  await lock.acquire()

  void tryTask(stream, streamName, id, keyvals).finally(release)

}

```