Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/janbjorge/pgqueuer
PgQueuer is a Python library leveraging PostgreSQL for efficient job queuing.
https://github.com/janbjorge/pgqueuer
postgres python queue
Last synced: 21 days ago
JSON representation
PgQueuer is a Python library leveraging PostgreSQL for efficient job queuing.
- Host: GitHub
- URL: https://github.com/janbjorge/pgqueuer
- Owner: janbjorge
- License: mit
- Created: 2024-04-19T10:11:43.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2024-09-28T11:53:08.000Z (about 1 month ago)
- Last Synced: 2024-10-03T08:15:24.425Z (about 1 month ago)
- Topics: postgres, python, queue
- Language: Python
- Homepage: https://pgqueuer.readthedocs.io/en/latest/index.html
- Size: 531 KB
- Stars: 966
- Watchers: 7
- Forks: 13
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
- awesome-repositories - janbjorge/pgqueuer - PgQueuer is a Python library leveraging PostgreSQL for efficient job queuing. (Python)
README
### Readme
## 🚀 PGQueuer - Building Smoother Workflows One Queue at a Time 🚀
[](https://github.com/janbjorge/pgqueuer/actions/workflows/ci.yml?query=branch%3Amain)
[](https://pypi.python.org/pypi/pgqueuer)
[](https://pepy.tech/project/pgqueuer)
[](https://github.com/janbjorge/pgqueuer)
---
📚 **Documentation**: [Explore the Docs 📖](https://pgqueuer.readthedocs.io/en/latest/)
🔍 **Source Code**: [View on GitHub 💾](https://github.com/janbjorge/pgqueuer/)
💬 **Join the Discussion**: [Discord Community](https://discord.gg/C7YMBzcRMQ)
---
## PGQueuer
PGQueuer is a minimalist, high-performance job queue library for Python, leveraging the robustness of PostgreSQL. Designed for simplicity and efficiency, PGQueuer uses PostgreSQL's LISTEN/NOTIFY to manage job queues effortlessly.
### Features
- **Simple Integration**: Easy to integrate with existing Python applications using PostgreSQL.
- **Efficient Concurrency Handling**: Utilizes PostgreSQL's `FOR UPDATE SKIP LOCKED` for reliable and concurrent job processing.
- **Real-time Notifications**: Leverages `LISTEN` and `NOTIFY` for real-time updates on job status changes.
### Installation
To install PGQueuer, simply install with pip the following command:
```bash
pip install pgqueuer
```
### Example Usage
Here's how you can use PGQueuer in a typical scenario processing incoming data messages:
#### Write and run a consumer
Start a long-lived consumer that will begin processing jobs as soon as they are enqueued by another process. In this case we want to be a bit more carefull as we want gracefull shutdowns, `pgqueuer run` will setup signals to
ensure this.
```python
from __future__ import annotations
import asyncpg
from pgqueuer.db import AsyncpgDriver, dsn
from pgqueuer.models import Job
from pgqueuer.qm import QueueManager
async def main() -> QueueManager:
connection = await asyncpg.connect(dsn())
driver = AsyncpgDriver(connection)
qm = QueueManager(driver)
# Setup the 'fetch' entrypoint
@qm.entrypoint("fetch")
async def process_message(job: Job) -> None:
print(f"Processed message: {job}")
return qm
```
```bash
python3 -m pgqueuer run tools.consumer.main
```
#### Write and run a producer
Start a short-lived producer that will enqueue 10,000 jobs.
```python
from __future__ import annotations
import asyncio
import sys
import asyncpg
from pgqueuer.db import AsyncpgDriver
from pgqueuer.queries import Queries
async def main(N: int) -> None:
connection = await asyncpg.connect()
driver = AsyncpgDriver(connection)
queries = Queries(driver)
await queries.enqueue(
["fetch"] * N,
[f"this is from me: {n}".encode() for n in range(1, N+1)],
[0] * N,
)
if __name__ == "__main__":
print(sys.argv)
N = 1_000 if len(sys.argv) == 1 else int(sys.argv[1])
asyncio.run(main(N))
```
```bash
python3 tools/producer.py 10000
```