Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/robinjoseph08/redisqueue
redisqueue provides a producer and consumer of a queue that uses Redis streams
https://github.com/robinjoseph08/redisqueue
consumer go golang producer producer-consumer redis streams
Last synced: 4 days ago
JSON representation
redisqueue provides a producer and consumer of a queue that uses Redis streams
- Host: GitHub
- URL: https://github.com/robinjoseph08/redisqueue
- Owner: robinjoseph08
- License: mit
- Created: 2019-07-07T04:36:54.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2024-03-08T06:24:44.000Z (8 months ago)
- Last Synced: 2024-10-25T00:09:25.280Z (10 days ago)
- Topics: consumer, go, golang, producer, producer-consumer, redis, streams
- Language: Go
- Homepage: https://godoc.org/github.com/robinjoseph08/redisqueue
- Size: 32.2 KB
- Stars: 130
- Watchers: 3
- Forks: 55
- Open Issues: 8
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE.txt
Awesome Lists containing this project
- awesome-go - redisqueue - redisqueue provides a producer and consumer of a queue that uses Redis streams. (Messaging / Search and Analytic Databases)
- zero-alloc-awesome-go - redisqueue - redisqueue provides a producer and consumer of a queue that uses Redis streams. (Messaging / Search and Analytic Databases)
- awesome-ccamel - robinjoseph08/redisqueue - redisqueue provides a producer and consumer of a queue that uses Redis streams (Go)
- awesome-golang-repositories - redisqueue
- awesome-go-extra - redisqueue - 07-07T04:36:54Z|2022-05-17T10:56:56Z| (Messaging / Advanced Console UIs)
README
# redisqueue
[](https://pkg.go.dev/github.com/robinjoseph08/redisqueue/v2?tab=doc)
[](https://travis-ci.org/robinjoseph08/redisqueue)
[](https://coveralls.io/github/robinjoseph08/redisqueue?branch=master)
[](https://goreportcard.com/report/github.com/robinjoseph08/redisqueue)
`redisqueue` provides a producer and consumer of a queue that uses [Redis
streams](https://redis.io/topics/streams-intro).
## Features
- A `Producer` struct to make enqueuing messages easy.
- A `Consumer` struct to make processing messages concurrenly.
- Claiming and acknowledging messages if there's no error, so that if a consumer
dies while processing, the message it was working on isn't lost. This
guarantees at least once delivery.
- A "visibility timeout" so that if a message isn't processed in a designated
time frame, it will be be processed by another consumer.
- A max length on the stream so that it doesn't store the messages indefinitely
and run out of memory.
- Graceful handling of Unix signals (`SIGINT` and `SIGTERM`) to let in-flight
messages complete.
- A channel that will surface any errors so you can handle them centrally.
- Graceful handling of panics to avoid crashing the whole process.
- A concurrency setting to control how many goroutines are spawned to process
messages.
- A batch size setting to limit the total messages in flight.
- Support for multiple streams.
## Installation
`redisqueue` requires a Go version with Modules support and uses import
versioning. So please make sure to initialize a Go module before installing
`redisqueue`:
```sh
go mod init github.com/my/repo
go get github.com/robinjoseph08/redisqueue/v2
```
Import:
```go
import "github.com/robinjoseph08/redisqueue/v2"
```
## Example
Here's an example of a producer that inserts 1000 messages into a queue:
```go
package main
import (
"fmt"
"github.com/robinjoseph08/redisqueue/v2"
)
func main() {
p, err := redisqueue.NewProducerWithOptions(&redisqueue.ProducerOptions{
StreamMaxLength: 10000,
ApproximateMaxLength: true,
})
if err != nil {
panic(err)
}
for i := 0; i < 1000; i++ {
err := p.Enqueue(&redisqueue.Message{
Stream: "redisqueue:test",
Values: map[string]interface{}{
"index": i,
},
})
if err != nil {
panic(err)
}
if i%100 == 0 {
fmt.Printf("enqueued %d\n", i)
}
}
}
```
And here's an example of a consumer that reads the messages off of that queue:
```go
package main
import (
"fmt"
"time"
"github.com/robinjoseph08/redisqueue/v2"
)
func main() {
c, err := redisqueue.NewConsumerWithOptions(&redisqueue.ConsumerOptions{
VisibilityTimeout: 60 * time.Second,
BlockingTimeout: 5 * time.Second,
ReclaimInterval: 1 * time.Second,
BufferSize: 100,
Concurrency: 10,
})
if err != nil {
panic(err)
}
c.Register("redisqueue:test", process)
go func() {
for err := range c.Errors {
// handle errors accordingly
fmt.Printf("err: %+v\n", err)
}
}()
fmt.Println("starting")
c.Run()
fmt.Println("stopped")
}
func process(msg *redisqueue.Message) error {
fmt.Printf("processing message: %v\n", msg.Values["index"])
return nil
}
```