{"id":14371997,"url":"https://github.com/janbjorge/pgqueuer","last_synced_at":"2026-10-06T17:48:01.858Z","repository":{"id":236383269,"uuid":"788904329","full_name":"janbjorge/pgqueuer","owner":"janbjorge","description":"PostgreSQL-backed background job and task queue for Python.","archived":false,"fork":false,"pushed_at":"2026-10-02T10:18:13.000Z","size":2951,"stargazers_count":1533,"open_issues_count":8,"forks_count":36,"subscribers_count":7,"default_branch":"main","last_synced_at":"2026-10-02T17:38:58.325Z","etag":null,"topics":["asyncio","background-jobs","job-queue","postgres","postgresql","python","queue","task-queue"],"latest_commit_sha":null,"homepage":"https://janbjorge.github.io/pgqueuer/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/janbjorge.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","claude":"CLAUDE.md","gemini":null,"cursor":null,"copilot":null,"dco":null,"cla":null,"disclosure":null},"funding":{"github":["janbjorge"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"lfx_crowdfunding":null,"polar":null,"buy_me_a_coffee":"jeybee","thanks_dev":null,"custom":null}},"created_at":"2024-04-19T10:11:43.000Z","updated_at":"2026-10-02T10:18:16.000Z","dependencies_parsed_at":"2026-10-06T15:00:22.067Z","dependency_job_id":null,"html_url":"https://github.com/janbjorge/pgqueuer","commit_stats":{"total_commits":195,"total_committers":3,"mean_commits":65.0,"dds":0.02051282051282055,"last_synced_commit":"a3802fd5b43f7f4c0ae5ca8fc0672a5f43c1dc48"},"previous_names":["janbjorge/pgqueuer"],"tags_count":112,"template":false,"template_full_name":null,"purl":"pkg:github/janbjorge/pgqueuer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janbjorge%2Fpgqueuer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janbjorge%2Fpgqueuer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janbjorge%2Fpgqueuer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janbjorge%2Fpgqueuer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/janbjorge","download_url":"https://codeload.github.com/janbjorge/pgqueuer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janbjorge%2Fpgqueuer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":343170754,"owners_count":38095621,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-10-03T21:59:58.778Z","status":"online","status_checked_at":"2026-10-06T02:00:06.537Z","response_time":86,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asyncio","background-jobs","job-queue","postgres","postgresql","python","queue","task-queue"],"created_at":"2024-08-27T23:01:32.907Z","updated_at":"2026-10-06T17:48:01.845Z","avatar_url":"https://github.com/janbjorge.png","language":"Python","funding_links":["https://github.com/sponsors/janbjorge","https://buymeacoffee.com/jeybee"],"categories":["Python"],"sub_categories":[],"readme":"# 🚀 PgQueuer: PostgreSQL-backed background job and task queue for Python\n\n[![CI](https://github.com/janbjorge/pgqueuer/actions/workflows/ci.yml/badge.svg)](https://github.com/janbjorge/pgqueuer/actions/workflows/ci.yml?query=branch%3Amain) [![pypi](https://img.shields.io/pypi/v/pgqueuer.svg)](https://pypi.python.org/pypi/pgqueuer) [![downloads](https://static.pepy.tech/badge/pgqueuer/month)](https://pepy.tech/project/pgqueuer) [![stars](https://img.shields.io/github/stars/janbjorge/pgqueuer?style=flat)](https://github.com/janbjorge/pgqueuer/stargazers) [![versions](https://img.shields.io/pypi/pyversions/pgqueuer.svg)](https://github.com/janbjorge/pgqueuer)\n\n[📚 Docs](https://janbjorge.github.io/pgqueuer/) · [💻 Source](https://github.com/janbjorge/pgqueuer/) · [💬 Discord](https://discord.gg/C7YMBzcRMQ)\n\n**Your PostgreSQL database is already a job queue.**\n\nPgQueuer turns PostgreSQL into a fast, reliable background job processor. Jobs live in the same database as your application data. One stack, full ACID guarantees, and no separate message broker to run.\n\n## AI-assisted setup\n\nAlready using a coding agent? Copy the onboarding prompt from the\n[PgQueuer documentation](https://janbjorge.github.io/pgqueuer/#let-an-agent-introduce-you).\nThe prompt directs your agent to the maintained setup route and canonical docs.\n\n## Features\n\n- 💡 **Minimal footprint**: one `pip install`; bring your existing PostgreSQL connection and start enqueueing\n- 🔁 **Transactional enqueue**: commit a job in the same transaction as your data; no dual-write drift\n- ⚛️ **Safe concurrency**: workers claim jobs with `FOR UPDATE SKIP LOCKED` (never double-processed), with per-entrypoint limits and serialized dispatch when you need them\n- 🚀 **Instant dispatch**: `LISTEN/NOTIFY` wakes workers the moment a job lands (with a polling fallback)\n- ⏰ **Scheduling \u0026 deferral**: cron-style recurring tasks and `execute_after`, no separate scheduler process\n- 📊 **Observability**: completion tracking, Prometheus metrics, tracing (Logfire/Sentry), and a live dashboard\n- 🧪 **In-memory mode**: run the whole queue without Postgres for tests and prototyping\n\n## Why PostgreSQL?\n\nIf you already run PostgreSQL, it can do double duty as your job queue. That means one fewer service to operate, and your queue and data stay consistent because they share the same database and transactions.\n\n```text\n┌──────────┐  enqueue   ┌────────────┐  NOTIFY   ┌──────────┐\n│ Your App │───────────▶│            │──────────▶│ Worker 1 │──┐\n└──────────┘            │            │           └──────────┘  │\n                        │ PostgreSQL │  NOTIFY   ┌──────────┐  │\n                        │            │──────────▶│ Worker 2 │──┤\n                        │            │           └──────────┘  │\n                        │            │  NOTIFY   ┌──────────┐  │\n                        │            │──────────▶│ Worker N │──┤\n                        └────────────┘           └──────────┘  │\n                              ▲  FOR UPDATE SKIP LOCKED         │\n                              └─────────────────────────────────┘\n```\n\n## Installation\n\nPgQueuer targets Python 3.10+ and PostgreSQL 14+:\n\n```bash\npip install pgqueuer\npgq install        # create tables and functions in your database\n```\n\nThe CLI reads `PGHOST`, `PGUSER`, `PGDATABASE` and related environment variables. Use `pgq sql install` to preview the SQL (or pipe it to another client: `pgq sql install | psql`), `--prefix myapp_` to namespace tables, or `pgq uninstall` to remove the schema.\n\n## Quick Start\n\nPgQueuer pairs **consumers** (workers that process jobs) with **producers** (code that enqueues jobs).\n\n### 1. Define a consumer\n\nEach entrypoint is a job handler. Run it with the CLI: `pgq run examples.consumer:main`.\n\n```python\nimport asyncpg\nfrom pgqueuer import PgQueuer\nfrom pgqueuer.db import AsyncpgDriver\nfrom pgqueuer.models import Job\n\nasync def main() -\u003e PgQueuer:\n    connection = await asyncpg.connect()\n    pgq = PgQueuer(AsyncpgDriver(connection))\n\n    @pgq.entrypoint(\"fetch\")\n    async def process(job: Job) -\u003e None:\n        print(f\"Processed: {job!r}\")\n\n    return pgq\n```\n\n### 2. Enqueue jobs\n\nFrom your web app, script, or anywhere else with a database connection:\n\n```python\nimport asyncpg\nfrom pgqueuer.db import AsyncpgDriver\nfrom pgqueuer.queries import Queries\n\nasync def main() -\u003e None:\n    connection = await asyncpg.connect()\n    queries = Queries(AsyncpgDriver(connection))\n    await queries.enqueue(\"fetch\", b\"hello world\")\n```\n\nThe job arrives instantly via `LISTEN/NOTIFY`, and your consumer's `process` handler picks it up.\n\n### Enqueue inside a transaction\n\nThis is what a database-backed queue buys you: the job and your business data commit together, or not at all.\n\n```python\norder_id = 42\n\nasync with connection.transaction():\n    await connection.execute(\n        \"INSERT INTO orders (id, status) VALUES ($1, 'paid')\", order_id\n    )\n    await queries.enqueue(\"send_receipt\", str(order_id).encode())\n    # If the transaction rolls back, the job is never enqueued.\n```\n\n## Run without a database\n\n`PgQueuer.in_memory()` is a drop-in replacement that implements the same ports as the real backend, so your handlers stay identical. Good for unit tests and prototyping.\n\n```python\nimport asyncio\nfrom pgqueuer import PgQueuer\nfrom pgqueuer.models import Job\nfrom pgqueuer.domain.types import QueueExecutionMode\n\nasync def main() -\u003e None:\n    pq = PgQueuer.in_memory()\n\n    @pq.entrypoint(\"send_email\")\n    async def send_email(job: Job) -\u003e None:\n        print(f\"Sending: {job.payload!r}\")\n\n    await pq.qm.queries.enqueue([\"send_email\"], [b\"alice\"], [0])\n    await pq.qm.run(mode=QueueExecutionMode.drain)\n\nasyncio.run(main())\n```\n\nThe in-memory adapter has no durability or multi-process coordination, so use the PostgreSQL backend for production. See the [in-memory reference](https://janbjorge.github.io/pgqueuer/reference/in-memory/).\n\n## Documentation\n\n| Topic | What's inside |\n|-------|---------------|\n| [Core concepts](docs/getting-started/core-concepts.md) | Consumers, producers, entrypoints, the job lifecycle |\n| [Scheduling](docs/guides/scheduling.md) | Cron-style recurring tasks and deferred execution |\n| [Concurrency control](docs/guides/concurrency-control.md) | Per-entrypoint limits and serialized dispatch |\n| [Completion tracking](docs/guides/completion-tracking.md) | Wait for jobs to finish with `CompletionWatcher` |\n| [Shared resources](docs/guides/shared-resources.md) | Inject DB pools, HTTP clients, and models into handlers |\n| [Custom executors](docs/guides/custom-executors.md) | Retry strategies and exponential backoff |\n| [Drivers](docs/reference/drivers.md) | asyncpg, psycopg async/sync: choosing and configuring |\n| [Architecture](docs/reference/architecture.md) | Ports \u0026 adapters, `SKIP LOCKED`, design decisions |\n| [Observability](docs/integrations/prometheus.md) | Prometheus metrics, [tracing](docs/integrations/tracing.md), and the dashboard |\n| [Framework integration](examples/) | FastAPI ([example](examples/fastapi_usage.py)) and Flask ([example](examples/flask_sync_usage.py)) |\n\n## Monitor your queues\n\nLaunch the interactive dashboard to watch queue activity in real time:\n\n```bash\npgq dashboard --interval 10 --limit 25\n```\n\n```text\n+---------------------------+-------+------------+--------------------------+------------+----------+\n|          Created          | Count | Entrypoint | Time in Queue (HH:MM:SS) |   Status   | Priority |\n+---------------------------+-------+------------+--------------------------+------------+----------+\n| 2024-05-05 16:44:26+00:00 |  49   |    sync    |         0:00:01          | successful |    0     |\n| 2024-05-05 16:44:27+00:00 |  12   |   fetch    |         0:00:03          | queued     |    0     |\n| 2024-05-05 16:44:28+00:00 |   3   |  api_call  |         0:00:00          | picked     |    5     |\n+---------------------------+-------+------------+--------------------------+------------+----------+\n```\n\n## Development\n\nPgQueuer uses [Testcontainers](https://testcontainers.com/?language=python) to spin up an ephemeral PostgreSQL instance for the test suite. Just have Docker running.\n\n```bash\nuv sync --all-extras --frozen  # install dependencies\nuv run ruff check . \u0026\u0026 uv run ruff format . --check \u0026\u0026 uv run lint-imports \u0026\u0026 uv run mypy . \u0026\u0026 uv run pytest  # full check suite\n```\n\n## License\n\nPgQueuer is MIT licensed. See [LICENSE](LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjanbjorge%2Fpgqueuer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjanbjorge%2Fpgqueuer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjanbjorge%2Fpgqueuer/lists"}