https://github.com/mariotoffia/godynamodb-queue
A queue implementation in DynamoDB
https://github.com/mariotoffia/godynamodb-queue
dynamodb go queue sqs
Last synced: 3 months ago
JSON representation
A queue implementation in DynamoDB
- Host: GitHub
- URL: https://github.com/mariotoffia/godynamodb-queue
- Owner: mariotoffia
- License: apache-2.0
- Created: 2023-10-02T08:53:17.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2023-10-05T13:13:31.000Z (about 2 years ago)
- Last Synced: 2025-03-27T08:48:26.756Z (7 months ago)
- Topics: dynamodb, go, queue, sqs
- Language: Go
- Homepage:
- Size: 65.4 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.adoc
- License: LICENSE
Awesome Lists containing this project
README
:author_name: Mario Toffia
:author_email:
:author: {author_name}
:email: {author_email}
:source-highlighter: highlightjs
:toc:
:toc-title: Table of Contents
:toclevels: 3
:homepage: github.com/mariotoffia/godynamodb-queue
:stem: latexmath
:doctype: book
:imagesdir: ./assets
:icons: font
= DynamoDB as Queue in golang
.Status
|===
|Build Status |Documentation
|image:https://github.com/mariotoffia/godynamodb-queue/workflows/Go%20Build%20and%20Test/badge.svg["Go Build and Test"]
|link:https://pkg.go.dev/mod/github.com/mariotoffia/godynamodb-queue[image:https://pkg.go.dev/badge/mariotoffia/godynamodb-queue/repository.svg["Go Reference"]]
|===
== Introduction
This is a flexible queue, using DynamoDB as backend. This supports both a _QueueName_ and one to many _ClientID_s hence, it is possible to have multiple isolated clients under the same _QueueName_.
By default, the queue is set into _FIFO_ mode, but may be altered at runtime to _LIFO_.
Since multiple `DynamoDbQueue`/processes can simultaneously read/write/delete from the queue it will use a `visibilityTimeout` to ensure that only one consumer can process a message at a time.
The DynamoDB Table have the following schema:
[cols="1,1,1,1,1,1", options="header"]
|===
|PK |SK |hidden_until |owner |TTL |event
|queueName-clientID
|`{unix-64-bit-timestamp (nanoseconds) + "_" + random_string}`
|`{now()+visibilityTimeout}`
|owner
|ttl
|`{events.SQSMessage}``
|===
When the query is commenced, it will query for all messages within PK (oldest first) and that
now() is greater than hidden_until.
NOTE: The DynamoDB table must be configured with the TTL option on the TTL column in order for it to
be deleted automatically.
It has convenience functions to create tables with proper configuration in runtime (as well as drop them).
.Create Table
[source,go]
----
ctx := context.Background()
config, _ := awsconfig.LoadDefaultConfig(ctx)
created, _ := dynamodbqueue.NewDynamoDBQueue(ctx, cfg, 0).
UseTable("queue-table").
CreateQueueTable(ctx)
----
== Sample Usage
It is very flexible to use, you may modify the `DynamoDbQueue` at runtime to change the mode of operation, queue, client, visibility timeout and so on. It is even possible to change the TTL for the message to live at runtime.
.Simple Example
[source,go]
----
ctx := context.Background()
config, _ := awsconfig.LoadDefaultConfig(ctx)
queue := dynamodbqueue.NewDynamoDBQueue(ctx, cfg, 0). <1>
UseQueueName("testQueue").
UseClientID("testClientId")
msgs, _ := queue.PushMessages(ctx, 0 /*defaultTTL*/, events.SQSMessage{ // <2>
MessageAttributes: map[string]events.SQSMessageAttribute{
"test2": { DataType: "String", StringValue: aws.String("test2") },
},
Body: "test body",
})
// Check the number of messages in the queue
count, _ := queue.Count(ctx)
// Poll max 10 messages (return as fast as possible)
msgs, _ = queue.PollMessages( // <3>
ctx,
0, /*noTimeout*/
time.Minute*14, /*visibilityTimeout*/
10, /*maxMessages*/
)
// Use the messages and then delete them
queue.DeleteMessages(ctx, ToReceiptHandles(msgs)...) // <4>
----
<1> Create a queue manager with queueName and clientID.
<2> Push one or more messages to the queue
<3> Poll for messages, max 10 messages and exit as fast as possible. "Own" the messages for 14 minutes.
<4> Since owner, it is possible to delete the messages using the receipt handles.
When the timeout is set to a positive value, it will continuously try to get maxMessages before exiting the loop.
When pulling the messages, it uses DynamoDBs ability to set a criteria before setting ownership of message, therefore it is safe for different clients, and processes to use this queue simoulatinously.
== The Idea
This idea, was born when the usage of SQS queues was not sufficient flexible. I had a set of queues and a set of clients and there where no possibility to dequeue messages that has e.g. a discriminator to select on. I.e it would consume other
clients messages and push back them into the queue, making it very inefficient. I do love _SQS_ but it was not usable for that particular use-case.