Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/vmihailenco/taskq
Golang asynchronous task/job queue with Redis, SQS, IronMQ, and in-memory backends
https://github.com/vmihailenco/taskq
go golang ironmq ironmq-backend message-queue queue redis sqs task-queue taskqueue
Last synced: 22 days ago
JSON representation
Golang asynchronous task/job queue with Redis, SQS, IronMQ, and in-memory backends
- Host: GitHub
- URL: https://github.com/vmihailenco/taskq
- Owner: vmihailenco
- License: bsd-2-clause
- Created: 2017-02-04T10:11:14.000Z (almost 8 years ago)
- Default Branch: v3
- Last Pushed: 2023-10-16T11:02:33.000Z (about 1 year ago)
- Last Synced: 2024-10-01T15:03:22.184Z (about 1 month ago)
- Topics: go, golang, ironmq, ironmq-backend, message-queue, queue, redis, sqs, task-queue, taskqueue
- Language: Go
- Homepage: https://taskq.uptrace.dev/
- Size: 711 KB
- Stars: 1,239
- Watchers: 19
- Forks: 99
- Open Issues: 32
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
- awesome-golang-repositories - taskq - memory backends (Repositories)
README
# Golang asynchronous task/job queue with Redis, SQS, IronMQ, and in-memory backends
[](https://pkg.go.dev/github.com/vmihailenco/taskq/v3)
[](https://taskq.uptrace.dev/)
[](https://discord.gg/rWtp5Aj)
> taskq is brought to you by :star: [**uptrace/uptrace**](https://github.com/uptrace/uptrace).
> Uptrace is an open source and blazingly fast
> [distributed tracing tool](https://get.uptrace.dev/compare/distributed-tracing-tools.html) powered
> by OpenTelemetry and ClickHouse. Give it a star as well!
## Features
- Redis, SQS, IronMQ, and in-memory backends.
- Automatically scaling number of goroutines used to fetch (fetcher) and process messages (worker).
- Global rate limiting.
- Global limit of workers.
- Call once - deduplicating messages with same name.
- Automatic retries with exponential backoffs.
- Automatic pausing when all messages in queue fail.
- Fallback handler for processing failed messages.
- Message batching. It is used in SQS and IronMQ backends to add/delete messages in batches.
- Automatic message compression using snappy / s2.
Resources:
- [**Get started**](https://taskq.uptrace.dev/guide/golang-task-queue.html)
- [Examples](https://github.com/vmihailenco/taskq/tree/v3/example)
- [Discussions](https://github.com/uptrace/bun/discussions)
- [Chat](https://discord.gg/rWtp5Aj)
- [Reference](https://pkg.go.dev/github.com/vmihailenco/taskq/v3)
## Getting started
To get started, see [Golang Task Queue](https://taskq.uptrace.dev/) documentation.
**Producer**:
```go
import (
"github.com/vmihailenco/taskq/v3"
"github.com/vmihailenco/taskq/v3/redisq"
)
// Create a queue factory.
var QueueFactory = redisq.NewFactory()
// Create a queue.
var MainQueue = QueueFactory.RegisterQueue(&taskq.QueueOptions{
Name: "api-worker",
Redis: Redis, // go-redis client
})
// Register a task.
var CountTask = taskq.RegisterTask(&taskq.TaskOptions{
Name: "counter",
Handler: func() error {
IncrLocalCounter()
return nil
},
})
ctx := context.Background()
// And start producing.
for {
// Call the task without any args.
err := MainQueue.Add(CountTask.WithArgs(ctx))
if err != nil {
panic(err)
}
time.Sleep(time.Second)
}
```
**Consumer**:
```go
// Start consuming the queue.
if err := MainQueue.Start(context.Background()); err != nil {
log.Fatal(err)
}
```
## See also
- [Golang ORM](https://github.com/uptrace/bun) for PostgreSQL, MySQL, MSSQL, and SQLite
- [Golang PostgreSQL](https://bun.uptrace.dev/postgres/)
- [Golang HTTP router](https://github.com/uptrace/bunrouter)
- [Golang ClickHouse](https://github.com/uptrace/go-clickhouse)
## Contributors
Thanks to all the people who already contributed!
