{"id":50529073,"url":"https://github.com/basalam/baqueue","last_synced_at":"2026-06-03T11:01:38.920Z","repository":{"id":362064509,"uuid":"1257028446","full_name":"basalam/baqueue","owner":"basalam","description":"A powerful Python queue management package. Multi-driver support, batch jobs, scheduling, auto-balancing, and a beautiful real-time monitoring dashboard.","archived":false,"fork":false,"pushed_at":"2026-06-02T11:32:31.000Z","size":395,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-02T13:20:36.947Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/basalam.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-06-02T09:49:27.000Z","updated_at":"2026-06-02T11:32:35.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/basalam/baqueue","commit_stats":null,"previous_names":["basalam/baqueue"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/basalam/baqueue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basalam%2Fbaqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basalam%2Fbaqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basalam%2Fbaqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basalam%2Fbaqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/basalam","download_url":"https://codeload.github.com/basalam/baqueue/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/basalam%2Fbaqueue/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33860971,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-03T02:00:06.370Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-03T11:01:38.188Z","updated_at":"2026-06-03T11:01:38.913Z","avatar_url":"https://github.com/basalam.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BaQueue\n\n[![PyPI version](https://img.shields.io/pypi/v/baqueue.svg)](https://pypi.org/project/baqueue/)\n[![Python versions](https://img.shields.io/pypi/pyversions/baqueue.svg)](https://pypi.org/project/baqueue/)\n[![License: MIT](https://img.shields.io/pypi/l/baqueue.svg)](https://github.com/basalam/baqueue/blob/master/LICENSE)\n[![CI](https://github.com/basalam/baqueue/actions/workflows/ci.yml/badge.svg)](https://github.com/basalam/baqueue/actions/workflows/ci.yml)\n\nA powerful Python queue management package. Multi-driver support, batch jobs, scheduling, auto-balancing, and a beautiful real-time monitoring dashboard.\n\n![Overview](./baqueue/screen/1.png)\n\n### Jobs - Filterable job list with status, attempts, and duration\n![Jobs](./baqueue/screen/2.png)\n\n### Queues - Per-queue detail cards with pending/processing/completed/failed\n![Queues](./baqueue/screen/3.png)\n\n## Table of Contents\n\n- [Features](#features)\n- [Quick Start](#quick-start)\n  - [Define a Job](#define-a-job)\n  - [Dispatch Jobs](#dispatch-jobs)\n  - [Batch Jobs](#batch-jobs)\n  - [Run Workers](#run-workers)\n  - [Pruning](#pruning)\n    - [Auto-pruning](#auto-pruning-runs-alongside-baqueue-work)\n    - [Manual pruning](#manual-pruning)\n  - [Retry Failed Jobs](#retry-failed-jobs)\n  - [Dashboard](#dashboard)\n    - [Workers Tab (Supervisor/Worker Monitoring)](#workers-tab-supervisorworker-monitoring)\n  - [Drivers](#drivers)\n- [Examples](#examples)\n- [Testing](#testing)\n- [CLI Commands](#cli-commands)\n- [Benchmarks](#benchmarks)\n  - [Test 1: 1,000 jobs / 5 workers](#test-1-1000-jobs--5-workers)\n  - [Test 2: 5,000 jobs / 10 workers](#test-2-5000-jobs--10-workers)\n  - [Stress Test Options](#stress-test-options)\n  - [Run with Live Dashboard](#run-with-live-dashboard)\n- [License](#license)\n\n## Features\n- **Multi-driver**: SQLite (default), Redis, PostgreSQL, or In-Memory\n- **Auto-balancing**: Dynamically scale workers based on queue pressure\n- **Auto-pruning**: Completed jobs are deleted about 5 seconds after they finish; failed/cancelled jobs are kept up to 1 day — all configurable\n- **Disk-full cleanup**: Storage-full/OOM driver errors trigger emergency cleanup of terminal jobs and old metrics, then retry once\n- **Pruning**: Remove old jobs by status, tag, or age\n- **Monitoring Dashboard**: Real-time WebSocket-powered UI with date filtering\n- **CLI**: Manage workers, scheduler, dashboard, and pruning from the command line\n- **Cross-process**: SQLite driver shares state between dashboard and workers without external dependencies\n\n## Quick Start\n\n```bash\n# Install (SQLite + in-memory drivers work out of the box, zero extra dependencies)\npip install baqueue\n\n# With Redis support\npip install \"baqueue[redis]\"\n\n# With PostgreSQL support\npip install \"baqueue[postgres]\"\n\n# With dashboard\npip install \"baqueue[dashboard]\"\n\n# Everything\npip install \"baqueue[all]\"\n```\n\n\u003e **Installing from source (development):**\n\u003e ```bash\n\u003e git clone https://github.com/basalam/baqueue.git\n\u003e cd baqueue\n\u003e pip install -e \".[dev]\"   # editable install with all extras + test/build tooling\n\u003e ```\n\n### Define a Job\n\n```python\nfrom baqueue import Job\n\nclass SendEmail(Job):\n    queue = \"emails\"\n    max_attempts = 3\n    backoff = \"exponential\"\n\n    async def handle(self, to: str, subject: str, body: str):\n        await send_email(to, subject, body)\n\n    async def on_failure(self, error, payload):\n        print(f\"Failed to send email: {error}\")\n```\n\nOr use the decorator:\n\n```python\nfrom baqueue import Job\n\n@Job.as_job(queue=\"emails\", max_attempts=3)\nasync def send_email(to, subject, body):\n    ...\n```\n\n### Dispatch Jobs\n\n```python\nfrom baqueue import Queue, BaQueueConfig\nfrom baqueue.config import DriverConfig\n\n# Configure (SQLite driver by default - works across processes)\nQueue.configure(BaQueueConfig(driver=DriverConfig(name=\"sqlite\")))\nawait Queue.connect()\n\n# Push a job\nawait Queue.push(SendEmail, to=\"user@example.com\", subject=\"Hi\", body=\"Hello!\")\n\n# Push with delay (60 seconds)\nawait Queue.later(SendEmail, delay=60, to=\"user@example.com\", subject=\"Reminder\", body=\"...\")\n\n# Bulk push (much faster for large volumes)\nawait Queue.bulk([\n    (SendEmail, {\"to\": \"a@b.com\", \"subject\": \"Hi\", \"body\": \"A\"}),\n    (SendEmail, {\"to\": \"c@d.com\", \"subject\": \"Hi\", \"body\": \"B\"}),\n])\n```\n\n### Batch Jobs\n\n```python\nfrom baqueue import Batch\n\nresult = await Batch(driver, [\n    (SendEmail, {\"to\": \"a@b.com\", \"subject\": \"Hi\", \"body\": \"Hey\"}),\n    (SendEmail, {\"to\": \"c@d.com\", \"subject\": \"Hi\", \"body\": \"Hey\"}),\n]).name(\"newsletter\").then(OnAllDone).catch(OnAnyFailed).dispatch()\n```\n\n### Run Workers\n\n```python\nfrom baqueue.supervisor import Supervisor\nfrom baqueue.config import SupervisorConfig\n\nsupervisor = Supervisor(\n    driver=Queue.get_driver(),\n    config=SupervisorConfig(\n        queues=[\"emails\", \"payments\"],\n        min_workers=3,\n        max_workers=10,\n        balance=\"auto\",\n    ),\n)\nawait supervisor.start()\n```\n\nOr via CLI:\n\n```bash\nbaqueue work -q emails -q payments -w 3 -b auto\n```\n\n### Pruning\n\n#### Auto-pruning (runs alongside `baqueue work`)\n\nWhen `baqueue work` is running, a background pruner cycles every 5s and applies these defaults across every driver:\n\n| Status              | Default lifetime | Config field                |\n|---------------------|------------------|-----------------------------|\n| `completed`         | 5 seconds        | `prune_completed_seconds`   |\n| `failed`, `cancelled` | 1 day          | `prune_other_seconds`       |\n| metrics rows        | 7 days           | `prune_metrics_seconds`     |\n| pruner cycle        | every 5s         | `prune_interval_seconds`    |\n| enable/disable      | `True`           | `auto_prune`                |\n\nOverride from a JSON config file (`baqueue -c config.json work`):\n\n```json\n{\n  \"auto_prune\": true,\n  \"prune_completed_seconds\": 600,\n  \"prune_other_seconds\": 172800,\n  \"prune_interval_seconds\": 30,\n  \"auto_cleanup_on_disk_full\": true\n}\n```\n\nOr from Python:\n\n```python\nfrom baqueue import BaQueueConfig\nconfig = BaQueueConfig(\n    prune_completed_seconds=600,    # 10 minutes\n    prune_other_seconds=172800,     # 2 days\n    prune_interval_seconds=30,\n    auto_cleanup_on_disk_full=True, # enabled by default\n)\n```\n\nOr from the CLI:\n\n```bash\nbaqueue work --prune-completed-seconds 600 --prune-other-seconds 172800\nbaqueue work --no-auto-prune              # disable the background pruner\nbaqueue work --no-disk-full-cleanup       # disable emergency storage cleanup\n```\n\nThe legacy hour-based fields (`prune_completed_hours`, `prune_failed_hours`, `prune_cancelled_hours`, `prune_metrics_hours`) are still respected for backward compatibility — when set to a positive value they override the corresponding `*_seconds` field.\n\n#### Disk-full emergency cleanup\n\n`auto_cleanup_on_disk_full` is enabled by default. When a driver write/update/delete operation sees a storage-full style error (SQLite disk full, PostgreSQL disk/memory exhausted, Redis OOM/maxmemory), BaQueue runs an emergency cleanup that removes terminal jobs (`completed`, `failed`, `cancelled`) and old metrics, then retries the failed operation once. If cleanup does not free enough space, the original driver error is still raised.\n\n#### Manual pruning\n\n```python\n# Remove completed jobs older than 24 hours\nawait Queue.prune(status=\"completed\", hours=24)\n\n# Remove jobs by tag\nawait Queue.prune(tag=\"batch:newsletter\")\n```\n\n### Retry Failed Jobs\n\nBulk-retry failed jobs from the CLI, from Python, or from the dashboard.\n\n**CLI:**\n\n```bash\n# Retry every failed job (asks for confirmation)\nbaqueue retry-failed\n\n# Skip the confirmation prompt\nbaqueue retry-failed -y\n\n# Limit to a specific queue\nbaqueue retry-failed -q emails\n\n# Combine filters: queue + tag + age window\nbaqueue retry-failed -q emails -t campaign --hours 24\n\n# Use a non-default driver\nbaqueue retry-failed -d redis --driver-url redis://localhost:6379/0\n```\n\n**Python:**\n\n```python\n# Retry every failed job\ncount = await Queue.retry_failed()\n\n# Retry only failed jobs in a queue\ncount = await Queue.retry_failed(queue=\"emails\")\n\n# Filter by tag and creation window\nfrom baqueue.serializer import _now_ts\ncount = await Queue.retry_failed(\n    queue=\"emails\",\n    tag=\"campaign\",\n    created_from=_now_ts() - 24 * 3600,\n)\n```\n\n**Dashboard:** open the **Jobs** tab, set the Status filter to `Failed`, then click the amber **Retry All** button. The current Queue / Tag / date-range filters are respected.\n\nEach matched job is released back onto its queue with `delay=0`, the same path used by single-job retry.\n\n### Dashboard\n\n```bash\n# Start the dashboard (uses SQLite by default)\nbaqueue dashboard\n\n# Open http://localhost:9100\n```\n\nThe dashboard includes:\n- Real-time overview with pending/processing/completed/failed counters sourced from live job state (not bounded metric logs)\n- Date range filtering (custom range + presets: 1h, 24h, 7d, 30d)\n- Job detail modal with timeline, payload data, and error trace\n- Queue breakdown with progress bars\n- Worker monitoring with active/idle status\n- Dark/light theme toggle\n- **Scheduled-job badge** with hover tooltip showing exact execution time, plus a \"Scheduled For\" entry in the job timeline\n- **Bulk \"Retry All\"** button when the Jobs view is filtered to `failed` (respects the active queue/tag/date filters)\n- **Queue filter as a dropdown** auto-populated from active queues (no manual typing)\n- **Mobile-friendly** sidebar drawer with hamburger toggle on screens ≤900px\n\nRun in one terminal:\n```bash\nbaqueue dashboard\n```\n\nDispatch jobs in another terminal:\n```bash\npython examples/simple_job.py\n```\n\n#### Workers Tab (Supervisor/Worker Monitoring)\n\nTo see active supervisors/workers in the `Workers` tab, `work` and `dashboard`\nmust point to the same backend (same driver and same URL/path).\n\nExample with SQLite:\n\nTerminal 1:\n```bash\nbaqueue work -d sqlite --driver-url .baqueue.db -q default -w 3\n```\n\nTerminal 2:\n```bash\nbaqueue dashboard -d sqlite --driver-url .baqueue.db\n```\n\nThen open:\n```text\nhttp://localhost:9100\n```\n\nQuick troubleshooting:\n- Check `http://localhost:9100/api/supervisors` (should return a non-empty `supervisors` list while workers are running).\n- If `api/supervisors` is empty, `work` and `dashboard` are likely using different driver URLs/paths.\n- `memory` driver is single-process only, so separate `work` and `dashboard` processes will not share worker state.\n\nDriver-specific CLI examples:\n\nSQLite (shared local file):\n```bash\nbaqueue work -d sqlite --driver-url .baqueue.db -q default -w 3\nbaqueue dashboard -d sqlite --driver-url .baqueue.db\n```\n\nRedis (shared Redis DB):\n```bash\nbaqueue work -d redis --driver-url redis://localhost:6379/0 -q default -w 3\nbaqueue dashboard -d redis --driver-url redis://localhost:6379/0\n```\n\nPostgreSQL (shared database/schema):\n```bash\nbaqueue work -d postgres --driver-url postgresql://user:pass@localhost/dbname -q default -w 3\nbaqueue dashboard -d postgres --driver-url postgresql://user:pass@localhost/dbname\n```\n\nMemory (single-process only):\n```bash\n# Use an in-process example to run workers + dashboard together.\npython examples/dashboard_demo.py\n```\n\n### Drivers\n\n**SQLite (default, zero-config, cross-process):**\n```python\nQueue.configure(BaQueueConfig(\n    driver=DriverConfig(name=\"sqlite\")\n))\n```\n\n**Redis:**\n```python\nQueue.configure(BaQueueConfig(\n    driver=DriverConfig(name=\"redis\", url=\"redis://localhost:6379/0\")\n))\n```\n\n**PostgreSQL:**\n```python\nQueue.configure(BaQueueConfig(\n    driver=DriverConfig(name=\"postgres\", url=\"postgresql://user:pass@localhost/dbname\")\n))\n```\n\n**Memory (single-process testing only):**\n```python\nQueue.configure(BaQueueConfig(\n    driver=DriverConfig(name=\"memory\")\n))\n```\n\n## Examples\n\n```bash\n# Simple job processing\npython examples/simple_job.py\n\n# Batch processing\npython examples/batch_example.py\n\n# Scheduled jobs\npython examples/scheduled_example.py\n\n# Dashboard demo (open http://localhost:9100)\npython examples/dashboard_demo.py\n\n# Delayed jobs demo — shows the \"Scheduled\" badge with varied delays\npython examples/delayed_jobs_demo.py\n\n# Stress test (see Benchmarks section below)\npython examples/stress_test.py --jobs 1000 --workers 5 --bulk\n```\n\n## Testing\n\nThe full test suite lives in `tests/` and runs with one command:\n\n```bash\n# Run everything\nbaqueue test\n\n# Quiet output, stop at the first failure\nbaqueue test -q -x\n\n# Run only retry-failed related tests\nbaqueue test -k \"RetryFailed or retry_failed\"\n\n# Re-run just the tests that failed last time\nbaqueue test --last-failed\n\n# Filter by marker (markers defined in pyproject.toml)\nbaqueue test -m \"not slow\"\n```\n\n`baqueue test` is a thin wrapper around `pytest`, so it picks up the project's\n`tool.pytest.ini_options` config (asyncio mode, marker definitions, etc.).\nYou can also run pytest directly:\n\n```bash\npip install baqueue[dev]\npytest tests/ -v\n```\n\nCoverage includes:\n\n- Serializer / payload roundtrip (incl. `delay_until`)\n- Backoff strategies (`fixed`, `linear`, `exponential`, explicit list)\n- `Job` + `FunctionJob` + `@Job.as_job` decorator\n- `Queue` facade — push / later / bulk / prune / `retry_failed`\n- Cross-driver contract tests (memory + sqlite, parameterized)\n- Worker lifecycle: success / failure / retry / timeout\n- Supervisor pool + delayed-job promotion\n- Scheduler interval dispatch\n- Pruner by status / tag / age\n- Batch builder + completion callbacks\n- DashboardAPI (overview, jobs_list, retry, bulk retry-failed, prune, stats)\n- CLI command surface (help text, validation, `retry-failed` abort flow)\n\n## CLI Commands\n\n```\nbaqueue work          Start processing jobs\nbaqueue schedule      Start the job scheduler\nbaqueue dashboard     Launch the monitoring dashboard\nbaqueue prune         Prune old jobs\nbaqueue retry-failed  Retry all failed jobs (filter by queue/tag/age)\nbaqueue status        Show queue status\nbaqueue test          Run the test suite\n```\n\nUse `-h` on any command for options:\n```bash\nbaqueue -h\nbaqueue work -h\nbaqueue dashboard -h\n```\n\n## Benchmarks\n\nStress tests run on **Windows 10, Python 3.11, SQLite driver**, using `examples/stress_test.py`.\n\nThe stress test dispatches jobs across 5 queues (`fast`, `slow`, `flaky`, `heavy`, `notifications`) with varying execution times and a ~30% failure rate on the `flaky` queue, exercising retries and backoff.\n\n### Test 1: 1,000 jobs / 5 workers\n\n```bash\npython examples/stress_test.py --jobs 1000 --workers 5 --bulk\n```\n\n```\n============================================================\n  RESULTS\n============================================================\n  Total time:    30.38s\n  Completed:     993\n  Failed:        7\n  Throughput:    32.9 jobs/s\n  Success rate:  99.3%\n============================================================\n```\n\n| Metric            | Value          |\n|-------------------|----------------|\n| Dispatch speed    | 28,426 jobs/s  |\n| Processing speed  | 32.9 jobs/s    |\n| Total time        | 30.4s          |\n| Success rate      | 99.3%          |\n\n### Test 2: 5,000 jobs / 10 workers\n\n```bash\npython examples/stress_test.py --jobs 5000 --workers 10 --bulk\n```\n\n```\n============================================================\n  RESULTS\n============================================================\n  Total time:    49.95s\n  Completed:     4965\n  Failed:        35\n  Throughput:    100.1 jobs/s\n  Success rate:  99.3%\n============================================================\n```\n\n| Metric            | Value          |\n|-------------------|----------------|\n| Dispatch speed    | ~50,000 jobs/s |\n| Processing speed  | 100.1 jobs/s   |\n| Total time        | 49.9s          |\n| Success rate      | 99.3%          |\n\n### Stress Test Options\n\n```bash\npython examples/stress_test.py [OPTIONS]\n\nOptions:\n  --jobs, -j      Number of jobs to dispatch (default: 1000)\n  --workers, -w   Number of concurrent workers (default: 5)\n  --bulk          Use bulk insert for faster dispatching\n  --dashboard     Launch live dashboard on http://localhost:9100\n```\n\n**Job types used in the stress test:**\n\n| Job       | Queue           | Latency        | Failure Rate | Max Attempts |\n|-----------|-----------------|----------------|--------------|--------------|\n| FastJob   | `fast`          | 10-50ms        | 0%           | 3            |\n| SlowJob   | `slow`          | 100-300ms      | 0%           | 2            |\n| FlakyJob  | `flaky`         | 20-80ms        | ~30%         | 3            |\n| HeavyJob  | `heavy`         | 50-150ms       | 0%           | 1            |\n| Notify    | `notifications` | 10-40ms        | 0%           | 2            |\n\n### Run with Live Dashboard\n\n```bash\npython examples/stress_test.py --jobs 3000 --workers 8 --bulk --dashboard\n# Open http://localhost:9100 to watch progress in real-time\n```\n\n## License\n\n[MIT](./LICENSE) © Basalam and BaQueue Contributors\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbasalam%2Fbaqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbasalam%2Fbaqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbasalam%2Fbaqueue/lists"}