{"id":13647256,"url":"https://github.com/janbjorge/PgQueuer","last_synced_at":"2025-04-22T02:31:00.762Z","repository":{"id":236383269,"uuid":"788904329","full_name":"janbjorge/pgqueuer","owner":"janbjorge","description":"PgQueuer is a Python library leveraging PostgreSQL for efficient job queuing.","archived":false,"fork":false,"pushed_at":"2025-04-18T13:37:23.000Z","size":1230,"stargazers_count":1243,"open_issues_count":3,"forks_count":21,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-04-19T02:18:35.451Z","etag":null,"topics":["postgres","python","queue"],"latest_commit_sha":null,"homepage":"https://pgqueuer.readthedocs.io/en/stable/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/janbjorge.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-04-19T10:11:43.000Z","updated_at":"2025-04-18T13:37:26.000Z","dependencies_parsed_at":"2024-08-02T01:26:15.967Z","dependency_job_id":"824ddec9-782c-46ff-a655-564abd614b10","html_url":"https://github.com/janbjorge/pgqueuer","commit_stats":{"total_commits":195,"total_committers":3,"mean_commits":65.0,"dds":0.02051282051282055,"last_synced_commit":"a3802fd5b43f7f4c0ae5ca8fc0672a5f43c1dc48"},"previous_names":["janbjorge/pgqueuer"],"tags_count":82,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janbjorge%2Fpgqueuer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janbjorge%2Fpgqueuer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janbjorge%2Fpgqueuer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janbjorge%2Fpgqueuer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/janbjorge","download_url":"https://codeload.github.com/janbjorge/pgqueuer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250163592,"owners_count":21385265,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["postgres","python","queue"],"created_at":"2024-08-02T01:03:26.341Z","updated_at":"2025-04-22T02:31:00.741Z","avatar_url":"https://github.com/janbjorge.png","language":"Python","readme":"# 🚀 PGQueuer - Building Smoother Workflows One Queue at a Time 🚀\n\n[![CI](https://github.com/janbjorge/pgqueuer/actions/workflows/ci.yml/badge.svg)](https://github.com/janbjorge/pgqueuer/actions/workflows/ci.yml?query=branch%3Amain) [![pypi](https://img.shields.io/pypi/v/pgqueuer.svg)](https://pypi.python.org/pypi/pgqueuer) [![downloads](https://static.pepy.tech/badge/pgqueuer/month)](https://pepy.tech/project/pgqueuer) [![versions](https://img.shields.io/pypi/pyversions/pgqueuer.svg)](https://github.com/janbjorge/pgqueuer)\n\n---\n\n- 📚 **Documentation**: [Explore the Docs](https://pgqueuer.readthedocs.io/en/latest/)\n- 🔍 **Source Code**: [View on GitHub](https://github.com/janbjorge/pgqueuer/)\n- 💬 **Join the Discussion**: [Discord Community](https://discord.gg/C7YMBzcRMQ)\n\n---\n\nPGQueuer is a minimalist, high-performance job queue library for Python, leveraging PostgreSQL's robustness. Designed with simplicity and efficiency in mind, PGQueuer offers real-time, high-throughput processing for background jobs using PostgreSQL's LISTEN/NOTIFY and `FOR UPDATE SKIP LOCKED` mechanisms.\n\n## Features\n\n- **💡 Simple Integration**: Seamlessly integrates with Python applications using PostgreSQL, providing a clean and lightweight interface.\n- **⚛️ Efficient Concurrency Handling**: Supports `FOR UPDATE SKIP LOCKED` to ensure reliable concurrency control and smooth job processing without contention.\n- **🚧 Real-time Notifications**: Uses PostgreSQL's `LISTEN` and `NOTIFY` commands for real-time job status updates.\n- **👨‍🎓 Batch Processing**: Supports large job batches, optimizing enqueueing and dequeuing with minimal overhead.\n- **⏳ Graceful Shutdowns**: Built-in signal handling ensures safe job processing shutdown without data loss.\n- **⌛ Recurring Job Scheduling**: Register and manage recurring tasks using cron-like expressions for periodic execution.\n\n## Installation\n\nInstall PGQueuer via pip:\n\n```bash\npip install pgqueuer\n```\n\n## Quick Start\n\nBelow is a minimal example of how to use PGQueuer to process data.\n\n### Step 1: Write a consumer\n\n```python\nfrom __future__ import annotations\n\nfrom datetime import datetime\n\nimport asyncpg\n\nfrom pgqueuer import PgQueuer\nfrom pgqueuer.db import AsyncpgDriver\nfrom pgqueuer.models import Job, Schedule\n\n\nasync def main() -\u003e PgQueuer:\n    connection = await asyncpg.connect()\n    driver = AsyncpgDriver(connection)\n    pgq = PgQueuer(driver)\n\n    # Entrypoint for jobs whose entrypoint is named 'fetch'.\n    @pgq.entrypoint(\"fetch\")\n    async def process_message(job: Job) -\u003e None:\n        print(f\"Processed message: {job!r}\")\n\n    # Define and register recurring tasks using cron expressions\n    # The cron expression \"* * * * *\" means the task will run every minute\n    @pgq.schedule(\"scheduled_every_minute\", \"* * * * *\")\n    async def scheduled_every_minute(schedule: Schedule) -\u003e None:\n        print(f\"Executed every minute {schedule!r} {datetime.now()!r}\")\n\n    return pgq\n```\n\nThe above example is located in the examples folder, and can be run by using the `pgq` cli.\n```bash\npgq run examples.consumer.main\n```\n\n### Step 2: Write a producer\n\n```python\nfrom __future__ import annotations\n\nimport sys\n\nimport asyncpg\nimport uvloop\n\nfrom pgqueuer.db import AsyncpgDriver\nfrom pgqueuer.queries import Queries\n\n\nasync def main(N: int) -\u003e None:\n    connection = await asyncpg.connect()\n    driver = AsyncpgDriver(connection)\n    queries = Queries(driver)\n    await queries.enqueue(\n        [\"fetch\"] * N,\n        [f\"this is from me: {n}\".encode() for n in range(1, N + 1)],\n        [0] * N,\n    )\n\n\nif __name__ == \"__main__\":\n    N = 1_000 if len(sys.argv) == 1 else int(sys.argv[1])\n    uvloop.run(main(N))\n```\n\nRun the producer:\n```bash\npython3 examples/producer.py 10000\n```\n\n## Dashboard\n\nMonitor job processing statistics in real-time using the built-in dashboard:\n\n```bash\npgq dashboard --interval 10 --tail 25 --table-format grid\n```\nThis provides a real-time, refreshing view of job queues and their status.\n\nExample output:\n\n```bash\n+---------------------------+-------+------------+--------------------------+------------+----------+\n|          Created          | Count | Entrypoint | Time in Queue (HH:MM:SS) |   Status   | Priority |\n+---------------------------+-------+------------+--------------------------+------------+----------+\n| 2024-05-05 16:44:26+00:00 |  49   |    sync    |         0:00:01          | successful |    0     |\n...\n+---------------------------+-------+------------+--------------------------+------------+----------+\n```\n\n## Why Choose PGQueuer?\n\n- **Built for Scale**: Handles thousands of jobs per second, making it ideal for high-throughput applications.\n- **PostgreSQL Native**: Utilizes advanced PostgreSQL features for robust job handling.\n- **Flexible Concurrency**: Offers rate and concurrency limiting to cater to different use-cases, from bursty workloads to critical resource-bound tasks.\n\n## License\n\nPGQueuer is MIT licensed. See [LICENSE](LICENSE) for more information.\n\n---\nReady to supercharge your workflows? Install PGQueuer today and take your job management to the next level!\n\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjanbjorge%2FPgQueuer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjanbjorge%2FPgQueuer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjanbjorge%2FPgQueuer/lists"}