https://github.com/modfin/pqdocket
A simple Postgres-powered work queue based on lib/pq
https://github.com/modfin/pqdocket
Last synced: 11 months ago
JSON representation
A simple Postgres-powered work queue based on lib/pq
- Host: GitHub
- URL: https://github.com/modfin/pqdocket
- Owner: modfin
- License: mit
- Created: 2025-08-01T03:57:48.000Z (11 months ago)
- Default Branch: master
- Last Pushed: 2025-08-12T06:42:49.000Z (11 months ago)
- Last Synced: 2025-08-12T08:31:18.753Z (11 months ago)
- Language: Go
- Size: 23.4 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# pqdocket
[](https://github.com/modfin/pqdocket/actions/workflows/build-and-test.yaml)
A simple Postgres-powered work queue based on `lib/pq`.
## Design goals
- Support long-running jobs, don't occupy a connection / transaction while a task is running.
- Be able to schedule tasks at a timestamp in the future.
- Fast task claiming based using `SELECT ... FOR UPDATE SKIP LOCKED`
- Uses postgres `LISTEN/NOTIFY` to:
- Reduce polling while idle.
- Immediately begin processing tasks on all worker instances when tasks are scheduled, uses `LISTEN/NOTIFY` for this.
- At-Least-Once execution, see notes below.
## Notes on At-Least-Once execution
When used with a correct setup and with well-behaved tasks, it typically will operate at Exactly-Once. But some problems can cause it to not be able to reach Exactly-Once execution. And then we choose At-Least-Once over At-Most-Once.
For example:
- Postgres or the Go worker instances, crashing during task execution.
- Buggy tasks that occasionally block / deadlock forever. The task will then after the claim time (task execution timeout) have expired, be scheduled again.
- Tasks with too low claim time.
- Tasks that don't extend their claim time if they run longer than expected.