{"id":40909533,"url":"https://github.com/i-christian/pgqueue","last_synced_at":"2026-01-24T16:22:39.867Z","repository":{"id":329098664,"uuid":"1117792907","full_name":"i-christian/pgqueue","owner":"i-christian","description":"A simple task queue in Go","archived":false,"fork":false,"pushed_at":"2025-12-19T16:59:39.000Z","size":60,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-20T22:32:56.038Z","etag":null,"topics":["background-jobs","go","golang","postgres","task-queue"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/i-christian.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-16T20:27:58.000Z","updated_at":"2025-12-19T16:57:35.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/i-christian/pgqueue","commit_stats":null,"previous_names":["i-christian/pgqueue"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/i-christian/pgqueue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-christian%2Fpgqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-christian%2Fpgqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-christian%2Fpgqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-christian%2Fpgqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/i-christian","download_url":"https://codeload.github.com/i-christian/pgqueue/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-christian%2Fpgqueue/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28651817,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-22T01:17:37.254Z","status":"online","status_checked_at":"2026-01-22T02:00:07.137Z","response_time":144,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["background-jobs","go","golang","postgres","task-queue"],"created_at":"2026-01-22T03:00:37.058Z","updated_at":"2026-01-22T03:01:09.411Z","avatar_url":"https://github.com/i-christian.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pgqueue\n\n![GitHub go.mod Go version](https://img.shields.io/github/go-mod/go-version/i-christian/pgqueue)\n[![License](https://img.shields.io/github/license/i-christian/pgqueue)](./LICENSE)\n[![Project Status](https://img.shields.io/badge/status-learning--project-orange)](#)\n\n**pgqueue** is a lightweight, asynchronous, durable, PostgreSQL-backed job queue for Go.\n\nIt is designed to be **simple**, **safe**, and **easy to reason about**, using only PostgreSQL and standard SQL.\n\n\u003e ⚠️ **Project status**\n\u003e This is primarily a **learning project**, which I have created to explore how background job queues work internally.\n\u003e That aside, pgqueue aims to follow solid, production-style patterns and is suitable for real-world experimentation and small-to-medium workloads.\n\n---\n\n## Features\n\n* ✅ Distributed-safe workers\n* ⏱ Delayed execution\n* 🔁 Automatic retries with exponential backoff + jitter\n* 🚦 Job priorities\n* 🧠 Deduplication support\n* ⏰ Cron jobs (run once across many servers)\n* 📊 Queue metrics \u0026 stats\n* 🪵 Structured logging (`slog` middleware)\n* 💥 Crash-resilient, at-least-once delivery\n\n---\n\n## Why pgqueue?\n\nIf you already use PostgreSQL, you don’t need Redis, SQS, or Kafka **just to run background jobs**.\n\nPostgreSQL is already:\n\n* Durable\n* Transactional\n* Highly available\n* Operationally familiar\n\n`pgqueue` builds a background job queue using:\n\n* `SELECT … FOR UPDATE SKIP LOCKED`\n* Advisory locking semantics\n* Transactions for correctness\n* `LISTEN / NOTIFY` for fast wake-ups\n\n---\n\n## Architecture Overview\nThis diagram shows how producers, PostgreSQL, workers, and cron jobs interact inside **pgqueue**.\n```mermaid\nflowchart LR\n    %% Nodes\n    P[\"Producers\u003cbr/\u003equeue.Enqueue()\"]\n    C[\"Cron Scheduler\u003cbr/\u003eScheduleCron()\"]\n\n    T[\"PostgreSQL\u003cbr/\u003etasks table\"]\n    A[\"tasks_archive\"]\n    N[\"LISTEN / NOTIFY\"]\n\n    W[\"Worker Pool\u003cbr/\u003eStartConsumer(n)\"]\n    M[\"ServeMux\"]\n    H[\"Task Handlers\"]\n    R[\"Retry \u0026 Rescue\"]\n    S[\"Metrics / Stats\"]\n\n    %% Flows\n    P --\u003e T\n    C --\u003e T\n    T --\u003e N\n    N --\u003e W\n    W --\u003e M\n    M --\u003e H\n    H --\u003e|success| T\n    H --\u003e|failure| R\n    R --\u003e T\n    T --\u003e A\n    W --\u003e S\n\n    %% Styles\n    classDef producer fill:#E3F2FD,stroke:#1565C0,stroke-width:2px;\n    classDef postgres fill:#E8F5E9,stroke:#2E7D32,stroke-width:2px;\n    classDef worker fill:#FFF8E1,stroke:#EF6C00,stroke-width:2px;\n    classDef handler fill:#F3E5F5,stroke:#6A1B9A,stroke-width:2px;\n    classDef metrics fill:#ECEFF1,stroke:#455A64,stroke-width:2px;\n\n    class P,C producer;\n    class T,A,N postgres;\n    class W,M,R worker;\n    class H handler;\n    class S metrics;  \n```\n\n---\n\n## Installation\n\n```bash\ngo get github.com/i-christian/pgqueue\n```\n\n---\n## Initilise queue's client with options\n```go\nclient, err := pgqueue.NewClient(\n    db,\n    pgqueue.WithRescueConfig(5*time.Minute, 30*time.Minute),\n\t\tpgqueue.WithCleanupConfig(1*time.Hour, 24*time.Hour, pgqueue.ArchiveStrategy),\n    // Enables cron job scheduling, which is disabled by default \n\t\tpgqueue.WithCronEnabled(),\n  )\nif err != nil {\n    log.Fatalf(\"Failed to init queue: %v\", err)\n\n```\n\n## Enqueue a Job\n\n```go\ntype EmailPayload struct {\n    Subject string `json:\"subject\"`\n}\n\nclient.Enqueue(\n    ctx,\n    \"task:send:email\",\n    EmailPayload{Subject: \"Welcome!\"},\n)\n```\n\n### Enqueue with Options\n\n```go\nclient.Enqueue(\n    ctx,\n    \"task:send:email\",\n    payload,\n    pgqueue.WithPriority(pgqueue.HighPriority),\n    pgqueue.WithDelay(5*time.Minute),\n    pgqueue.WithMaxRetries(10),\n    pgqueue.WithDedup(\"email:user:123\"),\n)\n```\n\nSupported options include:\n\n* Priority\n* Delayed execution\n* Retry limits\n* Deduplication keys\n\n---\n\n## Start Workers (ServeMux)\n\n`pgqueue` uses a `ServeMux` to route tasks by type, similar to `http.ServeMux`.\n\n```go\nmux := pgqueue.NewServeMux()\n\n// Middleware runs for every task\nmux.Use(pgqueue.SlogMiddleware(client.Logger, client.Metrics))\n\n// Exact match\nmux.HandleFunc(\"task:send:email\", sendEmailHandler)\n\n// Prefix match\nmux.HandleFunc(\"task:cleanup:\", cleanupHandler)\nmux.HandleFunc(\"task:report:\", reportHandler)\n\n// Start worker pool\nserver := pgqueue.NewServer(db, connStr, 3, mux)\nif err := server.Start(); err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tlog.Println(\"Worker server started...\")\n\n```\n\n---\n\n## ⚠️ Bounded Task Types (Important)\n\nTask types **must be bounded**.\n\n### ✅ Good (bounded)\n\n```\ntask:send:email\ntask:cleanup:expired-sessions\ntask:report:daily\n```\n\n### ❌ Bad (unbounded)\n\n```\ntask:report:user:123\ntask:email:user:UUID\n```\n\n### Why this matters\n\n* Routing is based on task type or prefix\n* Metrics are keyed by task type\n* Unbounded types can cause **unbounded memory growth**\n\n**Rule of thumb:**\nUse task **categories**, not per-entity identifiers.\n\n---\n\n## Cron Jobs\n\nRun scheduled jobs **once**, even when multiple workers or servers are running.\n\n```go\ncronID, err := client.ScheduleCron(\n\t\"0 * * * *\",\n\t\"hourly-report\",\n\tTaskReportBase+\"hourly\",\n\tReportPayload{ReportName: \"Hourly\"},\n)\nif err != nil {\n\tlog.Fatal(err)\n}\n\njobs, _ := client.ListCronJobs()\nfor _, job := range jobs {\n\tfmt.Printf(\n\t\t\"Cron %d → next: %s\\n\",\n\t\tjob.ID,\n\t\tjob.NextRun.Format(time.DateTime),\n\t)\n}\n\n// Optional cleanup\nclient.RemoveCron(cronID)\n```\n\n---\n\n## Retries \u0026 Backoff\n\n* At-least-once execution\n* Automatic retries on failure\n* Exponential backoff: `2^attempts`\n* Jitter added to prevent thundering-herd effects\n* Max retries configurable per job\n\n---\n\n## Queue Stats\n\n```go\nstats, _ := client.Stats(ctx)\n\nfmt.Printf(\n    \"Pending: %d | Processing: %d | Failed: %d | Done: %d\\n\",\n    stats.Pending,\n    stats.Processing,\n    stats.Failed,\n    stats.Done,\n)\n```\n\n---\n\n## Examples\n\nA complete, runnable example demonstrating:\n\n* Worker pools\n* ServeMux routing\n* slog logging\n* Priorities\n* Retries\n* Cron jobs\n\n➡️ **See the full example here:**\n👉 [Examples](https://github.com/i-christian/pgqueue/tree/main/examples)\n\n---\n\n## Guarantees\n\npgqueue provides the following guarantees:\n\n✔ **At-least-once execution**\n✔ **No concurrent double-processing of the same task**\n✔ **Safe concurrency across multiple workers and processes**\n✔ **Crash resilience** (workers can die at any point)\n\n---\n\n## When **Not** to Use pgqueue\n\npgqueue is not a replacement for high-throughput message brokers.\n\nAvoid pgqueue if you need:\n\n* Ultra-low latency (\u003c1ms)\n* Massive fan-out (millions of jobs per second)\n* Cross-region replication\n* Exactly-once semantics\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fi-christian%2Fpgqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fi-christian%2Fpgqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fi-christian%2Fpgqueue/lists"}