https://github.com/reddec/pqueue
Persistent queue in Go based on BBolt
https://github.com/reddec/pqueue
Last synced: over 1 year ago
JSON representation
Persistent queue in Go based on BBolt
- Host: GitHub
- URL: https://github.com/reddec/pqueue
- Owner: reddec
- License: mit
- Created: 2021-11-01T15:25:24.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2021-11-10T11:58:18.000Z (over 4 years ago)
- Last Synced: 2025-03-09T00:06:40.216Z (over 1 year ago)
- Language: Go
- Size: 41 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Persistent queue
Persistent queue based on [bbolt DB](https://github.com/boltdb/bolt).
Supposed to be used as embeddable persistent queue to any Go application.
Features:
- messages are not limited by RAM size, stream-oriented data
- works good for small (automatic inlining) and for big streams
- simple, portable storage structure
- go-routine safe
- supports multiple-writers and multiple-readers
- supports ack/nack (commit with discard)
See [go-docs](https://pkg.go.dev/github.com/reddec/pqueue) for examples and details.
Requirements:
- go 1.17
## Motivation
I wanted to create an application for resource-constrained devices (ie: AWS Lightsail, Raspberry Pi Zero W, etc..)
which should store (always) and forward (eventually) information with very unreliable network
connection (ie: days without a link).
Information could be small (sensors) or huge (webhooks). Devices themselves may experience a power outage.
It means:
- stored information (number and individual records) may grow much above RAM
- stored information should not be marked as processed before explicit commit
- full-packed solutions like RabbitMQ/Kafka/etc can not be applied
- due to distributed nature of the system duplicates inevitable, however, should be a cheap way to deduplicate (ie: unique ID)