{"id":49723459,"url":"https://github.com/NikolayS/PgQue","last_synced_at":"2026-05-25T17:00:59.323Z","repository":{"id":351001503,"uuid":"1208944956","full_name":"NikolayS/PgQue","owner":"NikolayS","description":"PgQue – Zero-bloat Postgres queue built on top of on battle-proven Skype's PgQ. One SQL file to install, pg_cron to tick.","archived":false,"fork":false,"pushed_at":"2026-05-23T17:49:51.000Z","size":4014,"stargazers_count":1405,"open_issues_count":30,"forks_count":31,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-05-23T19:07:55.932Z","etag":null,"topics":["job-queue","pg-tle","postgres","postgres-extension","postgresql","queue","tle","trusted-language-extensions"],"latest_commit_sha":null,"homepage":"","language":"PLpgSQL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NikolayS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-13T00:06:44.000Z","updated_at":"2026-05-23T16:40:51.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/NikolayS/PgQue","commit_stats":null,"previous_names":["nikolays/pgque"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/NikolayS/PgQue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NikolayS%2FPgQue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NikolayS%2FPgQue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NikolayS%2FPgQue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NikolayS%2FPgQue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NikolayS","download_url":"https://codeload.github.com/NikolayS/PgQue/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NikolayS%2FPgQue/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33484522,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-25T14:31:05.219Z","status":"ssl_error","status_checked_at":"2026-05-25T14:31:02.878Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["job-queue","pg-tle","postgres","postgres-extension","postgresql","queue","tle","trusted-language-extensions"],"created_at":"2026-05-09T03:00:58.128Z","updated_at":"2026-05-25T17:00:59.305Z","avatar_url":"https://github.com/NikolayS.png","language":"PLpgSQL","funding_links":[],"categories":["PLpgSQL"],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003ePgQue – PgQ, universal edition\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\u003cstrong\u003eZero-bloat Postgres queue. One SQL file to install, \u003ccode\u003epg_cron\u003c/code\u003e or \u003ccode\u003epg_timetable\u003c/code\u003e to tick.\u003c/strong\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/NikolayS/pgque/actions/workflows/ci.yml\"\u003e\u003cimg src=\"https://github.com/NikolayS/pgque/actions/workflows/ci.yml/badge.svg\" alt=\"CI\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.postgresql.org/\"\u003e\u003cimg src=\"https://img.shields.io/badge/PostgreSQL-14--18-336791?logo=postgresql\u0026logoColor=white\" alt=\"PostgreSQL 14-18\"\u003e\u003c/a\u003e\n  \u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-Apache_2.0-blue.svg\" alt=\"License\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/citusdata/pg_cron\"\u003e\u003cimg src=\"https://img.shields.io/badge/pg__cron-optional-336791\" alt=\"pg_cron\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/NikolayS/pgque\"\u003e\u003cimg src=\"https://img.shields.io/badge/anti--extension-%5Ci_and_go-orange\" alt=\"Anti-Extension\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://news.ycombinator.com/item?id=47817349\"\u003e\u003cimg src=\"https://img.shields.io/badge/Hacker%20News-discussion-ff6600?logo=ycombinator\u0026logoColor=white\" alt=\"Discussion on Hacker News\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"docs/images/death_spiral.gif\" alt=\"Death spiral of a SKIP LOCKED queue under sustained load — the failure mode PgQue avoids by construction\" width=\"720\"\u003e\u003c/p\u003e\n\nDiscussion on [Hacker News](https://news.ycombinator.com/item?id=47817349).\n\n*For teams who want a durable event stream inside Postgres. The model is closer to Kafka (log) than to ActiveMQ or RabbitMQ (task message queue). Shared event log, independent per-consumer cursors, zero bloat under sustained load. Pure SQL and PL/pgSQL, any Postgres 14+ — managed or self-hosted, no sidecar daemon. The rest of this README walks the history, comparison, and install paths that back up the claim.*\n\n## Contents\n\n- [Why PgQue](#why-pgque)\n- [Latency trade-off](#latency-trade-off)\n- [Three latencies](#three-latencies)\n- [Comparison](#comparison)\n- [Installation](#installation)\n- [Roles and grants](#roles-and-grants)\n- [Project status](#project-status)\n- [Docs](#docs)\n- [Quick start](#quick-start)\n- [Client libraries](#client-libraries)\n- [Benchmarks](#benchmarks)\n- [Subconsumers / cooperative consumers](#subconsumers--cooperative-consumers)\n- [Architecture](#architecture)\n- [Roadmap](#roadmap)\n- [Contributing](#contributing)\n- [License](#license)\n\nPgQue brings back [PgQ](https://github.com/pgq/pgq) — one of the longest-running Postgres queue architectures in production — in a form that runs on any Postgres platform, managed providers included.\n\nPgQ was designed at Skype in 2006 to run messaging for hundreds of millions of users, and it ran on large self-managed Postgres deployments for over a decade. Standard PgQ depends on a C extension (`pgq`) and an external daemon (`pgqd`), neither of which run on most managed Postgres providers.\n\nPgQue rebuilds that battle-tested engine in pure PL/pgSQL, so the zero-bloat queue pattern works anywhere you can run SQL — without adding another distributed system to your stack.\n\nIt is the same engine – PgQ – repackaged for managed Postgres, and provided with client libraries for TypeScript, Python, and Go.\n\n**The anti-extension.** Pure SQL + PL/pgSQL on any Postgres 14+ — including RDS, Aurora, Cloud SQL, AlloyDB, Supabase, Neon, and most other managed providers. No C extension, no `shared_preload_libraries`, no provider approval, no restart.\n\nHistorical context, two decks:\n\n- [Marko Kreen (Skype), PGCon 2009 — PgQ](https://www.pgcon.org/2009/schedule/attachments/91_pgq.pdf)\n- [Alexander Kukushkin (Microsoft), 2026 — Rediscovering PgQ](https://speakerdeck.com/cyberdemn/rediscovering-pgq)\n\nExternal coverage:\n\n- [PgQue: Two Snapshots and a Diff](https://thebuild.com/blog/2026/05/03/pgque-two-snapshots-and-a-diff/) by Christophe Pettus — walk-through of the snapshot/diff mechanism: how two consecutive tick snapshots determine event visibility and why that avoids row-level locks and dead tuple bloat.\n- [HN discussion](https://news.ycombinator.com/item?id=47817349)\n\n## Why PgQue\n\nMost Postgres queues rely on `SKIP LOCKED` plus `DELETE` and/or `UPDATE`. That holds up in toy examples and then turns into dead tuples, VACUUM pressure, index bloat, and performance drift under sustained load.\n\nPgQue avoids that whole class of problems. It uses **snapshot-based batching** and **TRUNCATE-based table rotation** instead of per-row deletion. The hot path stays predictable:\n\n- **Zero bloat by design** — no dead tuples in the main queue path\n- **No performance decay** — it does not get slower because it has been running for months\n- **Built for heavy-loaded systems** — the sustained-load regime the original PgQ architecture was designed for\n- **Real Postgres guarantees** — ACID transactions, transactional enqueue/consume, WAL, backups, replication, SQL visibility\n- **Works on managed Postgres** — no custom build, no C extension, no separate daemon\n\nPgQue gives you queue semantics **inside** Postgres, with Postgres durability and transactional behavior, without the bloat tax most in-database queues eventually hit.\n\n## Latency trade-off\n\nPgQue is built around **snapshot-based batching**, not row-by-row claiming. That's what gives it zero bloat in the hot path, stable behavior under sustained load, and clean ACID semantics inside Postgres.\n\nThe trade-off is **end-to-end delivery latency** — the gap between `send` and when a consumer can `receive` the event. In the default configuration, end-to-end delivery typically lands around ~50–150 ms: PgQue ticks **10 times per second** (every 100 ms) by default, so the wait for the next tick is ~50 ms on average and at most ~100 ms, plus the consumer's poll interval. Per-call latency (the `send` / `receive` / `ack` functions themselves) stays in the microsecond range.\n\nWays to reduce delivery latency: tune the tick period (for example `pgque.set_tick_period_ms(50)` for 20 ticks/sec; accepted periods are exact divisors of 1000 ms) and queue thresholds; use `force_next_tick()` for tests and demos or to force an immediate batch. Future versions may add logical-decoding-based wake-ups for sub-millisecond delivery without burning more WAL on ticking.\n\nIf your top priority is single-digit-millisecond dispatch, PgQue is the wrong tool. If your priority is **stability under load without bloat**, that is where PgQue fits.\n\n## Three latencies\n\n\"Queue latency\" is three numbers, not one:\n\n1. **Producer latency** — `send` / `insert_event`. Sub-ms.\n2. **Subscriber latency** — `next_batch` over a pre-built batch. Sub-ms.\n3. **End-to-end delivery** — `send` → consumer visibility. ≈ tick period (default 100 ms). Tunable from 1 ms to 1000 ms via `pgque.set_tick_period_ms(ms)`. Does not grow with load.\n\nSee [docs/three-latencies.md](docs/three-latencies.md) for the breakdown, tick-cadence trade-off table, and comparison with UPDATE/DELETE-based designs.\n\n## Comparison\n\n| Feature | PgQue | PgQ | PGMQ | River | Que | pg-boss |\n|---|---|---|---|---|---|---|\n| Snapshot-based batching (no row locks) | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |\n| Zero bloat under sustained load | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |\n| No external daemon or worker binary | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |\n| Pure SQL install, managed Postgres ready | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ |\n| Language-agnostic SQL API | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |\n| Multiple independent consumers (fan-out) | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |\n| Built-in retry with backoff | ✅ | ✅ | ⚠️ | ✅ | ✅ | ✅ |\n| Built-in dead letter queue | ✅ | ❌ | ⚠️ | ⚠️ | ❌ | ✅ |\n\n**Legend:** ✅ yes · ❌ no · ⚠️ partial / indirect\n\n**Notes:**\n\n- **[PgQ](https://github.com/pgq/pgq)** is the Skype-era queue engine (~2007) PgQue is derived from. Same snapshot/rotation architecture, but requires C extensions and an external daemon (`pgqd`) — unavailable on managed Postgres. PgQue removes both constraints.\n- **No external daemon:** PgQue uses pg_cron (or your own scheduler) for ticking; PGMQ uses visibility timeouts. River, Que, and pg-boss require a Go / Ruby / Node.js worker binary.\n- **[Que](https://github.com/que-rb/que)** uses advisory locks (not SKIP LOCKED) — no dead tuples from *claiming*, but completed jobs are still DELETEd. Brandur's [bloat post](https://brandur.org/postgres-queues) was about Que at Heroku. Ruby-only.\n- **PGMQ retry** is visibility-timeout re-delivery (`read_ct` tracking) — no configurable backoff or max attempts.\n- **pg-boss fan-out** is copy-per-queue `publish()`/`subscribe()`, not a shared event log with independent cursors.\n- **Category:** River, Que, and pg-boss (and Oban, graphile-worker, solid_queue, good_job) are **job queue frameworks**. PgQue is an **event/message queue** optimized for high-throughput streaming with fan-out.\n\n### What differentiates PgQue\n\n**1. Zero event-table bloat, by design.** SKIP LOCKED queues (PGMQ, River, pg-boss, Oban, graphile-worker) UPDATE + DELETE rows, creating dead tuples that require VACUUM. Under sustained load this causes documented failures:\n\n- [Brandur/Heroku (2015)](https://brandur.org/postgres-queues) — 60k backlog in one hour.\n- [PlanetScale (2026)](https://planetscale.com/blog/keeping-a-postgres-queue-healthy) — death spiral at 800 jobs/sec with OLAP on the side.\n- [River issue #59](https://github.com/riverqueue/river/issues/59) — autovacuum starvation.\n\nOban Pro shipped table partitioning to mitigate it; PGMQ ships aggressive autovacuum settings. PgQue's TRUNCATE rotation creates zero dead tuples by construction. No tuning. Immune to xmin horizon pinning.\n\n**2. Native fan-out.** Each registered consumer maintains its own cursor on a shared event log and independently receives all events. That is different from competing-consumers (SKIP LOCKED) where each job goes to one worker. pg-boss has fan-out but it is copy-per-queue (one INSERT per subscriber per event). PgQue's model is a position on a shared log — no data duplication, atomic batch boundaries, late subscribers catch up. Closer to Kafka topics than to a job queue.\n\n### When to use PgQue vs. a job queue\n\n- **Choose PgQue** when you want event-driven fan-out, no bloat to tune around, and a language-agnostic SQL API, and you do not need per-job priorities or a worker framework.\n- **Choose a job queue** when you need per-job lifecycle, sub-3ms latency, priority queues, cron scheduling, unique jobs, or deep ecosystem integration (Elixir/Go/Node.js/Ruby).\n\n## Installation\n\n**Requirements:** Postgres 14+, and something that calls `pgque.ticker()` periodically. With `pg_cron`, `pgque.start()` schedules a single 1-second `pg_cron` slot that internally re-ticks every **100 ms (10 ticks/sec)** by default — see [Tick rate](#tick-rate) for tuning. `pg_cron` is pre-installed or one-command available on all major managed Postgres providers (RDS, Aurora, Cloud SQL, AlloyDB, Supabase, Neon); on self-managed Postgres, follow the [pg_cron setup guide](https://github.com/citusdata/pg_cron#setting-up-pg_cron). Any external scheduler (system `cron`, systemd, a worker loop in your app) works as an alternative — see below.\n\nGet the source — `\\i sql/pgque.sql` resolves relative to the cwd, so run psql from the repo root:\n\n```bash\ngit clone https://github.com/NikolayS/pgque\ncd pgque\n```\n\nInside a psql session:\n\n```sql\nbegin;\n\\i sql/pgque.sql\ncommit;\n```\n\nOr from the shell, same single-transaction guarantee via `psql --single-transaction`:\n\n```bash\nPAGER=cat psql --no-psqlrc --single-transaction -d mydb -f sql/pgque.sql\n```\n\nWith `pg_cron` available in the same database as PgQue, `pgque.start()` creates the default ticker and maintenance jobs. The ticker uses a one-second pg_cron slot and calls `pgque.ticker_loop()`, which ticks every 100 ms by default (10 ticks/sec) with a commit between ticks:\n\n```sql\nselect pgque.start();\n```\n\nWith `pg_timetable`, run the external pg_timetable worker against the database where PgQue is installed, then schedule PgQue with the same 10 ticks/sec default. For example:\n\n```bash\npg_timetable --dbname=mydb --clientname=pgque\n```\n\nThe `--clientname=pgque` flag is required; pg_timetable only executes chains whose `job_client_name` matches the running worker's client name. Keep that worker running; unlike `pg_cron`, pg_timetable is an external scheduler process, not a Postgres extension background worker. See the [pg_timetable docs](https://github.com/cybertec-postgresql/pg_timetable) for production service setup.\n\n```sql\nselect pgque.start_timetable();      -- default: 10 ticks/sec\n-- or explicitly:\nselect pgque.start_timetable(10);\n```\n\n`pgque.stop_timetable()` removes the PgQue pg_timetable jobs. `pgque.stop()` also stops whichever PgQue scheduler is active. Calling `pgque.start_timetable()` automatically removes existing PgQue `pg_cron` jobs first; `pgque.start()` does the same for PgQue pg_timetable jobs. `pgque.set_tick_period_ms(ms)` still controls the loop cadence; for example `100` means 10 ticks/sec, `200` means 5 ticks/sec, and `1000` means 1 tick/sec.\n\n### Tick rate\n\nPgQue ticks **10 times per second by default** (every 100 ms), even though `pg_cron`'s minimum schedule is 1 second. `pgque.start()` schedules a single 1-second pg_cron slot that calls `CALL pgque.ticker_loop()`; `pgque.start_timetable()` schedules the same loop through one pg_timetable `@every 1 second` job. The procedure then re-invokes `pgque.ticker()` every `tick_period_ms` ms inside that slot, committing between iterations so each tick gets its own transaction (snapshot semantics; bounded held-xmin so rotation isn't blocked).\n\nTune at runtime — no need to call `start()` again, the change picks up on the next scheduler slot (≤1 s):\n\n```sql\nselect pgque.set_tick_period_ms(50);    -- 20 ticks/sec, ~25 ms median e2e\nselect pgque.set_tick_period_ms(10);    -- 100 ticks/sec, ~5 ms median e2e\nselect pgque.set_tick_period_ms(1000);  -- 1 tick/sec, the pgqd-compatible cadence\n```\n\nAllowed values: exact divisors of `1000` in the `1`..`1000` ms range. Inspect the current rate with `select * from pgque.status();`.\n\nTrade-offs to keep in mind when raising the rate:\n- **Idle queues are cheap.** The 100 ms default is the *check cadence*, not a promise to write 10 ticks/sec forever. With no producer writes, most ticker calls return `NULL`; PgQue backs off toward `ticker_idle_period` (default 1 minute), so inactive queues produce occasional metadata writes, not hundreds of MiB/day. The larger WAL numbers apply only to queues that actually materialize ticks continuously. As a planning unit, a forced-tick PG18 measurement isolated about **280 bytes of WAL per materialized tick per queue**; that projects to roughly **240 MiB/day** at 10 materialized ticks/sec versus **24 MiB/day** at 1 materialized tick/sec. See [docs/tick-frequency.md](docs/tick-frequency.md) for caveats and tuning guidance.\n- **NOTIFY rate.** `pgque.ticker()` emits `pg_notify('pgque_\u003cqueue\u003e', ...)` per tick. Postgres's NOTIFY queue is global (8 GiB SLRU); slow LISTEN consumers can fall behind at very high rates.\n- **Metadata-table dead tuples.** `pgque.tick` and `pgque.subscription` are UPDATEd on every tick. PgQue rotates these tables to keep dead-tuple peaks bounded; at sub-50 ms tick periods, drop the rotation period proportionally.\n- **`pg_cron` background workers.** `pgque.ticker_loop()` holds one pg_cron worker for ~1 s per slot (vs. ~10 ms for the previous 1-second-cadence ticker). With pg_cron's default `cron.max_running_jobs = 32`, that bounds roughly **30 pgque-bearing databases per cluster** before the worker pool saturates. Not a per-database concern; matters if you fan PgQue across many databases on one instance.\n\n**pg_cron in a different database.** `pg_cron` runs jobs in one designated database (`cron.database_name`, typically `postgres`). If your PgQue schema lives in a different database, use the [cross-database pattern](https://github.com/citusdata/pg_cron#creating-a-cron-job-in-a-different-database) to call `pgque.ticker_loop()`, `pgque.maint_retry_events()`, and `pgque.maint()` across databases. *Todo: a future release will detect this and emit the correct `cron.schedule_in_database` calls from `pgque.start()` automatically.*\n\n**Scheduler log hygiene.** Every `pg_cron` job execution writes a row to `cron.job_run_details`, with no built-in purge. PgQue's internal sub-second loop does **not** make this worse — there is still only **one** `pg_cron` slot per second per job, regardless of `tick_period_ms`, so the per-second row count is the same as a 1 tick/sec schedule. Across PgQue's four scheduled jobs (ticker, retry, maint, rotate_step2), that is roughly **5,000 rows per hour** on top of any other pg_cron jobs, growing forever unless you intervene. Prefer a PgQue-specific purge job; disable `cron.log_run` globally only if you do not need successful-run history for any pg_cron jobs. See [the tutorial](docs/tutorial.md#production-cadence-use-pg_cron) for both recipes. *(Independent issue: `pg_cron` itself has no per-job log toggle as of 1.6.)* If you use pg_timetable, monitor and purge pg_timetable's own execution history tables according to its retention policy; PgQue does not manage those logs.\n\nWithout `pg_cron` or `pg_timetable`, PgQue still installs. Drive ticking and maintenance from your application or an external scheduler:\n\n```bash\nPAGER=cat psql --no-psqlrc -d mydb -c \"select pgque.ticker()\"              # at your chosen tick period\nPAGER=cat psql --no-psqlrc -d mydb -c \"select pgque.maint_retry_events()\"  # every 30 seconds\nPAGER=cat psql --no-psqlrc -d mydb -c \"select pgque.maint()\"               # every 30 seconds\n```\n\nFor sub-second ticking from an external driver, loop `pgque.ticker()` at your target rate; `tick_period_ms` is only consulted by `pgque.ticker_loop()` (the pg_cron path).\n\n**Important:** PgQue does not deliver messages without a working ticker. Enqueueing still works, but consumers will see nothing new because no ticks are created. If you do not use `pg_cron`, run `pgque.ticker()`, `pgque.maint_retry_events()`, and `pgque.maint()` yourself. Skipping `maint_retry_events()` means nack'd events will never be redelivered.\n\nFor existing installs, follow the SQL-file upgrade procedure in [docs/upgrading.md](docs/upgrading.md). To uninstall: `\\i sql/pgque_uninstall.sql`.\n\n### Optional: install as a [`pg_tle`](https://github.com/aws/pg_tle) extension\n\nThis is opt-in. The default `\\i sql/pgque.sql` install stays the recommended path; use pg_tle only if you specifically want PgQue managed as a real Postgres extension.\n\nWhat you get with pg_tle: `pg_extension` membership, `alter extension pgque update` for version upgrades, and `drop extension pgque cascade` for atomic uninstall. What you give up: pg_tle is itself a C extension preloaded via `shared_preload_libraries`, which is the dependency the default install avoids. Available on AWS RDS / Aurora and self-hosted Postgres; check your provider's extension list otherwise.\n\n**Prerequisites.** Run the installer as a role that holds `pgtle_admin` plus `CREATEROLE` (Postgres roles are cluster-global, so the wrapper creates `pgque_reader` / `pgque_writer` / `pgque_admin` outside the TLE body). pg_tle must also be in `shared_preload_libraries`. On managed providers, set this via the parameter group / cluster config UI and reboot. On self-hosted Postgres, **append** `pg_tle` to the existing list — overwriting it disables anything else you preload (e.g. `pg_cron`):\n\n```sql\nshow shared_preload_libraries;                                -- inspect current list first\nalter system set shared_preload_libraries = 'pg_cron,pg_tle'; -- preserve existing entries\n-- restart Postgres, then in the target database:\ncreate extension pg_tle;\n```\n\nOnce pg_tle is loaded, register and create PgQue:\n\n```sql\n\\i sql/pgque-tle.sql\ncreate extension pgque;\n```\n\nTo uninstall: `\\i sql/pgque-tle-uninstall.sql`.\n\n## Roles and grants\n\nThe install creates three roles. Application users do not need superuser — grant them whichever role matches their access pattern.\n\nPgQue mirrors upstream PgQ's split: `pgque_reader` (consume) and `pgque_writer` (produce) are **siblings**, not parent/child. `pgque_admin` is a member of both. Apps that produce **and** consume must be granted both roles explicitly.\n\n| Role | Purpose | Granted access |\n|---|---|---|\n| `pgque_reader` | Consumers, dashboards, metrics, debugging | Read-only info (`get_queue_info`, `get_consumer_info`, `get_batch_info`, `version`, `select` on all tables) **and** the consume API (`subscribe`, `unsubscribe`, `receive`, `ack`, `nack`) plus the underlying PgQ primitives (`next_batch*`, `get_batch_events`, `finish_batch`, `event_retry`, `register_consumer*`, `unregister_consumer`) and the [experimental cooperative consumer functions](docs/reference.md#cooperative-consumers--subconsumers). |\n| `pgque_writer` | Producers | The produce API (`send`, `send_batch`) and the underlying primitive (`insert_event`). Does **not** inherit `pgque_reader` — a producer-only role cannot ack/finish/inspect consumer batches. |\n| `pgque_admin`  | Operators, migrations | Member of both `pgque_reader` and `pgque_writer`, plus full schema/table/sequence access. `uninstall()` is revoked from both `pgque_admin` and PUBLIC (superuser-only — it drops the schema). |\n\nTypical app setup:\n\n```sql\n\\i sql/pgque.sql\nselect pgque.start();                     -- optional pg_cron ticker + maint\n\n-- Produce + consume in the same app: grant BOTH roles.\ncreate user app_orders with password '...';          -- replace with a real password\ngrant pgque_reader to app_orders;\ngrant pgque_writer to app_orders;\n\n-- Pure producer (e.g. a webhook ingester that only sends).\ncreate user app_webhook with password '...';\ngrant pgque_writer to app_webhook;\n\n-- Pure consumer / dashboard / metrics.\ncreate user metrics with password '...';              -- replace with a real password\ngrant pgque_reader to metrics;\n```\n\nDDL-class operations (`create_queue`, `drop_queue`, `start`, `stop`, `maint`, `maint_retry_events`, `ticker`, `force_next_tick`, `set_queue_config`) are not granted to either `pgque_reader` or `pgque_writer`. The schema-wide blanket `revoke execute … from public` strips PUBLIC, and `pgque_admin` is the only role that retains `execute` on these helpers — perform them as an admin / migration role.\n\n**Roles are global, not per-queue.** A `pgque_reader` granted to an app can ack any consumer's batch and read any other consumer's active batch payloads. Do not grant `pgque_reader` to mutually untrusted applications sharing one database unless you add your own schema-level or database-level isolation. See [docs/reference.md — Roles scope](docs/reference.md#roles-are-global-not-per-queue) for details and recommended isolation patterns.\n\n## Project status\n\nPgQue is **early-stage** as a product and API layer. PgQ itself has run at Skype scale for over a decade. What's new here is the packaging, modernization, managed-Postgres compatibility, and the higher-level PgQue API around that core.\n\nThe default install stays small; additional APIs live under `sql/experimental/` until they are worth promoting. See [blueprints/PHASES.md](blueprints/PHASES.md).\n\n## Docs\n\n- [Tutorial](docs/tutorial.md) — a hands-on walkthrough. Start here if you are new.\n- [Reference](docs/reference.md) — every shipped function and role.\n- [Upgrading](docs/upgrading.md) — SQL-file upgrade procedure for existing installs.\n- [Examples](docs/examples.md) — patterns: fan-out, exactly-once, batch loading, recurring jobs.\n- [Benchmarks](docs/benchmarks.md) — throughput measurements and methodology.\n- [Tick frequency tuning](docs/tick-frequency.md) — latency/WAL trade-offs, idle tick behavior, and pg_cron logging caveats.\n- [PgQ concepts](docs/pgq-concepts.md) — glossary (batch, tick, rotation) for contributors.\n- [PgQ history](docs/pgq-history.md) — where this engine came from.\n\n## Quick start\n\n```sql\n-- tx 1: create queue + consumer\nselect pgque.create_queue('orders');\nselect pgque.subscribe('orders', 'processor');\n\n-- tx 2: send a message\nselect pgque.send('orders', '{\"order_id\": 42, \"total\": 99.95}'::jsonb);\n\n-- tx 3: advance the queue if you are not using pg_cron\n-- force_next_tick bumps the event-seq threshold; ticker() then inserts the tick.\n-- Each select below is its own implicit transaction in psql autocommit —\n-- do NOT wrap these in begin/commit (the tick must see the send committed).\nselect pgque.force_next_tick('orders');\nselect pgque.ticker();\n\n-- tx 4: receive — every returned row carries the same batch_id\nselect * from pgque.receive('orders', 'processor', 100);\n--  msg_id | batch_id |  type   |             payload              | retry_count | ...\n-- --------+----------+---------+----------------------------------+-------------+----\n--       1 |        1 | default | {\"total\": 99.95, \"order_id\": 42} |             |\n-- (jsonb sorts object keys by length then alphabetically, so the input\n--  '{\"order_id\": 42, \"total\": 99.95}' comes back with \"total\" first)\n\n-- tx 5: ack the batch_id from the previous result\nselect pgque.ack(1);\n```\n\nSend, tick, and receive **must** run in separate transactions — that's a hard requirement of PgQ's snapshot-based design, not a recommendation. A `tick` records a snapshot of committed transaction IDs; a `send` in the same xact is still in-progress at that moment and gets excluded from the next batch's visibility window, so the event never surfaces. In normal operation, `pg_cron` or an external scheduler drives `pgque.ticker()`; `force_next_tick()` is mainly for demos, tests, and manual operation. In application code, capture `batch_id` from any returned row and pass it to `ack`.\n\nThe scriptable psql idiom (replaces tx 4 + tx 5 above):\n\n```sql\nselect batch_id from pgque.receive('orders', 'processor', 100) limit 1 \\gset\nselect pgque.ack(:batch_id);\n```\n\nLonger walkthrough in the [tutorial](docs/tutorial.md); patterns like fan-out, exactly-once, and recurring jobs in [examples](docs/examples.md).\n\n## Client libraries\n\nPgQue is SQL-first, so any Postgres driver works. First-party client libraries live in this repo for **Python**, **Go**, and **TypeScript**, all published at `v0.2.0`.\n\n### Python (`pgque-py`) — psycopg 3\n\n```bash\npip install pgque-py\n```\n\n```python\nfrom pgque import Consumer, connect\n\nwith connect(\"postgresql://localhost/mydb\") as client:\n    client.send(\"orders\", {\"order_id\": 42}, type=\"order.created\")\n    client.conn.commit()\n\nconsumer = Consumer(\n    \"postgresql://localhost/mydb\",\n    queue=\"orders\",\n    name=\"processor\",\n)\n\n@consumer.on(\"order.created\")\ndef handle_order(msg):\n    process_order(msg.payload)\n\nconsumer.start()\n```\n\n### Go (`github.com/NikolayS/pgque-go`) — pgx/v5\n\n```bash\ngo get github.com/NikolayS/pgque-go@v0.2.0\n```\n\n```go\nclient, _ := pgque.Connect(ctx, \"postgresql://localhost/mydb\")\ndefer client.Close()\n\n_, _ = client.Send(ctx, \"orders\", pgque.Event{\n    Type:    \"order.created\",\n    Payload: map[string]any{\"order_id\": 42},\n})\n\nconsumer := client.NewConsumer(\"orders\", \"processor\")\nconsumer.Handle(\"order.created\", func(ctx context.Context, msg pgque.Message) error {\n    return processOrder(msg)\n})\nconsumer.Start(ctx)\n```\n\n### TypeScript (`pgque`) — node-postgres\n\n```bash\nnpm install pgque        # or: bun add pgque\n```\n\n```ts\nimport { connect } from 'pgque';\n\nconst client = await connect('postgresql://localhost/mydb');\ntry {\n  await client.send('orders', {\n    type: 'order.created',\n    payload: { order_id: 42 },\n  });\n  await client.subscribe('orders', 'processor');\n\n  const messages = await client.receive('orders', 'processor', 100);\n  if (messages.length \u003e 0) await client.ack(messages[0]!.batchId);\n} finally {\n  await client.close();\n}\n```\n\n### Any language\n\n```sql\nselect pgque.send('orders', '{\"order_id\": 42}'::jsonb);\n\n-- without pg_cron, advance the queue manually (omit if a ticker is running).\n-- Run as separate transactions — do not wrap in begin/commit.\nselect pgque.force_next_tick('orders');\nselect pgque.ticker();\n\n-- receive returns rows; every row carries the same batch_id\nselect * from pgque.receive('orders', 'processor', 100);\n\n-- ack the batch_id from any returned row (capture it in the driver)\nselect pgque.ack(1);  -- replace with the batch_id from above\n```\n\n## Benchmarks\n\nPreliminary laptop numbers: ~86k ev/s batched PL/pgSQL insert, ~2.4M ev/s\nprimitive batch read rate (`get_batch_events`), zero dead-tuple growth under a\n30-minute sustained test with a blocked xmin horizon (a concurrent long-running\ntransaction holding an assigned XID — the worst case for SKIP LOCKED queues).\nThe batch read figure reflects raw PgQ primitive throughput, not end-to-end\n`receive()`/`ack()` consumer throughput. See\n[docs/benchmarks.md](docs/benchmarks.md) for the full table and methodology.\n\nPreliminary cross-system measurements live in [`benchmark/`](benchmark/).\nNumbers there are for reference and exploration, not a final verdict —\nbenchmarking Postgres queues is hard (cf. Brendan Gregg) and the\nmethodology continues to evolve.\n\nThe dedicated subconsumer demo harness lives in\n[`benchmark/subconsumer-scaling/`](benchmark/subconsumer-scaling/). It fixes the\nper-message downstream work at 250 ms and varies only consumer parallelism, so\nthe scaling story is easy to see without mixing in producer cadence or tick\ntuning.\n\n## Subconsumers / cooperative consumers\n\nThis is the use case that keeps coming up: the queue itself is fast, but the\ndownstream side effect is not. If one message means one transactional email API\ncall (Resend, SendGrid), one SMS request, one webhook, or one slow HTTP POST,\nthen consumer-side parallelism is what decides whether the backlog melts or\nlingers.\n\nPgQue does not need a second queue to show that effect. One main consumer can\nfetch a batch and fan the work out to a pool of subconsumers. To make the point\nconcrete, the demo harness preloads the same 160-message backlog every time and\nreplaces the email-provider call with a fixed `sleep(250 ms)` per message — an\nintentional stand-in for a service like Resend or SendGrid. That means one\nworker should top out near 4 messages / second. Then we increase only the\nnumber of subconsumers.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"docs/images/backlog_race.gif\" alt=\"Backlog drain race for 1, 2, 4, 8, and 16 subconsumers on the same 160-message queue with 250 ms of work per message\" width=\"760\"\u003e\u003c/p\u003e\n\nObserved drain times from the demo run above:\n\n| Subconsumers | Avg throughput | Drain time |\n|---:|---:|---:|\n| 1  | 4.0 msg/s  | 40.4 s |\n| 2  | 7.9 msg/s  | 20.3 s |\n| 4  | 15.8 msg/s | 10.1 s |\n| 8  | 31.3 msg/s | 5.1 s  |\n| 16 | 61.8 msg/s | 2.6 s  |\n\nThe static view below keeps the y-axis on **throughput**. That makes the\nscaling story more obvious: one worker buys you ~4 messages / second, and the\nobserved line tracks the ideal `4 × workers` line closely.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"docs/images/scaling_linearity.png\" alt=\"Observed throughput vs ideal linear scaling for 1, 2, 4, 8, and 16 subconsumers on the same 160-message backlog\" width=\"760\"\u003e\u003c/p\u003e\n\nThese are demo numbers, not a product claim. The point is narrower and more\nuseful: when downstream work costs ~250 ms / message, one worker buys you ~4\nmessages / second, and extra subconsumers scale throughput and backlog drain\nclose to linearly until some other bottleneck shows up.\n\n## Architecture\n\nPgQue keeps PgQ's proven core architecture — snapshot-based batch isolation, three-table TRUNCATE rotation on the hot path, separate retry / delayed / dead-letter tables, and independent per-consumer cursors — and adds a modern API layer on top. See [blueprints/SPECx.md](blueprints/SPECx.md) for the full specification and [docs/pgq-concepts.md](docs/pgq-concepts.md) for the batch/tick/rotation glossary.\n\n## Roadmap\n\n| Feature | Done |\n|---|---|\n| PgQ core engine | ✅ |\n| Modern Postgres support (14-18, 19devel) | ✅ |\n| Pure SQL / PL/pgSQL install | ✅ |\n| Managed Postgres support | ✅ |\n| No daemon / no C extension | ✅ |\n| `pg_cron`, `pg_timetable`, or external ticking | ✅ |\n| Sub-second ticking with `pg_cron` (default 10 ticks/sec, tunable) | ✅ |\n| System-table rotation / bloat mitigation |  |\n| [Cooperative consumers / subconsumers](#subconsumers--cooperative-consumers) | 🔬 experimental |\n| Queue splitter |  |\n| Queue mover |  |\n| Modern `send`, `receive`, `ack`, `nack` API | ✅ |\n| `send_batch` API | ✅ |\n| Improved `send_batch` performance | ✅ |\n| Dead-letter queue after retry limit | ✅ |\n| Go library | ✅ |\n| TypeScript library | ✅ |\n| Python library | ✅ |\n| Rust library |  |\n| Java library |  |\n| Ruby library |  |\n| Basic queue/consumer info views | ✅ |\n| Advanced observability / health views |  |\n| LISTEN/NOTIFY consumer wakeups on tick |  |\n| Delayed / scheduled delivery (`send_at`) |  |\n| Queue config JSON API |  |\n| Queue pause / resume |  |\n| OpenTelemetry / Prometheus metrics export |  |\n| Admin CLI |  |\n| Cross-database `pg_cron` scheduling |  |\n| Message TTL / expiry |  |\n| Per-tenant isolation / multi-schema installs |  |\n| `pg_tle` extension package | ✅ |\n| Migration guides |  |\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidelines.\n\nSee [blueprints/SPECx.md](blueprints/SPECx.md) for the specification and implementation plan. New code should follow red/green TDD: write the failing test first, then fix it. Agents and AI coding tools should also read [CLAUDE.md](CLAUDE.md).\n\n## License\n\nApache-2.0. See [LICENSE](LICENSE).\n\nPgQue includes code derived from [PgQ](https://github.com/pgq/pgq) (ISC license, Marko Kreen / Skype Technologies OU). See [NOTICE](NOTICE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNikolayS%2FPgQue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNikolayS%2FPgQue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNikolayS%2FPgQue/lists"}