{"id":24995332,"url":"https://github.com/launchplatform/bq","last_synced_at":"2026-03-07T20:31:49.621Z","repository":{"id":239486608,"uuid":"799632108","full_name":"LaunchPlatform/bq","owner":"LaunchPlatform","description":"BeanQueue, a lightweight Python task queue framework based on SQLAlchemy, PostgreSQL SKIP LOCKED queries and NOTIFY / LISTEN statements.","archived":false,"fork":false,"pushed_at":"2025-08-09T19:21:21.000Z","size":226,"stargazers_count":20,"open_issues_count":2,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-09-02T08:44:03.364Z","etag":null,"topics":["postgresql","python","queue","sqlalchemy","task-queue","worker"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LaunchPlatform.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-05-12T18:00:19.000Z","updated_at":"2025-08-28T20:17:20.000Z","dependencies_parsed_at":"2024-06-16T20:05:51.741Z","dependency_job_id":"27760da3-253b-4968-9fb9-0a44c8ce0465","html_url":"https://github.com/LaunchPlatform/bq","commit_stats":null,"previous_names":["launchplatform/bq"],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/LaunchPlatform/bq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LaunchPlatform%2Fbq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LaunchPlatform%2Fbq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LaunchPlatform%2Fbq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LaunchPlatform%2Fbq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LaunchPlatform","download_url":"https://codeload.github.com/LaunchPlatform/bq/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LaunchPlatform%2Fbq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30229743,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-07T19:01:10.287Z","status":"ssl_error","status_checked_at":"2026-03-07T18:59:58.103Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["postgresql","python","queue","sqlalchemy","task-queue","worker"],"created_at":"2025-02-04T15:35:14.301Z","updated_at":"2026-03-07T20:31:49.602Z","avatar_url":"https://github.com/LaunchPlatform.png","language":"Python","readme":"# BeanQueue [![CircleCI](https://dl.circleci.com/status-badge/img/gh/LaunchPlatform/bq/tree/master.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/LaunchPlatform/bq/tree/master)\n\nBeanQueue, a lightweight Python task queue framework based on [SQLAlchemy](https://www.sqlalchemy.org/), PostgreSQL [SKIP LOCKED queries](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/) and [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) / [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) statements.\n\n**Notice**: Still in its early stage, we built this for [BeanHub](https://beanhub.io)'s internal usage. May change rapidly. Use at your own risk for now.\n\n## Features\n\n- **Super lightweight**: Under 1K lines\n- **Easy-to-deploy**: Only relies on PostgreSQL\n- **Transactional**: Commit your tasks with other database entries altogether without worrying about data inconsistencies\n- **Easy-to-use**: Built-in command line tools for processing tasks and helpers for generating task models\n- **Auto-notify**: Automatic generation of NOTIFY statements for new or updated tasks, ensuring fast task processing\n- **Retry**: Built-in and customizable retry policies\n- **Schedule**: Schedule tasks to run later\n- **Worker heartbeat and auto-reschedule**: Each worker keeps updating heartbeat, if one is found dead, the others will reschedule the tasks\n- **Customizable**: Custom Task, Worker and Event models. Use it as a library and build your own work queue\n\n## Install\n\n```bash\npip install beanqueue\n```\n\n## Usage\n\nYou can define a basic task processor like this\n\n```python\nfrom sqlalchemy.orm import Session\n\nimport bq\nfrom .. import models\nfrom .. import image_utils\n\napp = bq.BeanQueue()\n\n@app.processor(channel=\"images\")\ndef resize_image(db: Session, task: bq.Task, width: int, height: int):\n    image = db.query(models.Image).filter(models.Image.task == task).one()\n    image_utils.resize(image, size=(width, height))\n    db.add(image)\n    # by default the `processor` decorator has `auto_complete` flag turns on,\n    # so it will commit the db changes for us automatically\n```\n\nThe `db` and `task` keyword arguments are optional.\nIf you don't need to access the task object, you can simply define the function without these two parameters.\nWe also provide an optional `savepoint` argument in case if you want to rollback database changes you made.\n\nTo submit a task, you can either use `bq.Task` model object to construct the task object, insert into the\ndatabase session and commit.\n\n```python\nimport bq\nfrom .db import Session\nfrom .. import models\n\ndb = Session()\ntask = bq.Task(\n    channel=\"files\",\n    module=\"my_pkgs.files.processors\",\n    name=\"upload_to_s3_for_backup\",\n)\nfile = models.File(\n    task=task,\n    blob_name=\"...\",\n)\ndb.add(task)\ndb.add(file)\ndb.commit()\n```\n\nOr, you can use the `run` helper like this:\n\n```python\nfrom .processors import resize_image\nfrom .db import Session\nfrom .. import my_models\n\ndb = Session()\n# a Task model generated for invoking resize_image function\ntask = resize_image.run(width=200, height=300)\n# associate task with your own models\nimage = my_models.Image(task=task, blob_name=\"...\")\ndb.add(image)\n# we have Task model SQLALchemy event handler to send NOTIFY \"\u003cchannel\u003e\" statement for you,\n# so that the workers will be woken up immediately\ndb.add(task)\n# commit will make the task visible to worker immediately\ndb.commit()\n```\n\nTo run the worker, you can do this:\n\n```bash\nBQ_PROCESSOR_PACKAGES='[\"my_pkgs.processors\"]' bq process images\n```\n\nThe `BQ_PROCESSOR_PACKAGES` is a JSON list contains the Python packages where you define your processors (the functions you decorated with `bq.processors.registry.processor`).\nTo submit a task for testing purpose, you can do\n\n```bash\nbq submit images my_pkgs.processors resize_image -k '{\"width\": 200, \"height\": 300}'\n```\n\nTo create tables for BeanQueue, you can run\n\n```bash\nbq create_tables\n```\n\n### Schedule\n\nIn most cases, a task will be executed as soon as possible after it is created.\nTo run a task later, you can set a datetime value to the `scheduled_at` attribute of the task model.\nFor example:\n\n```python\nimport datetime\n\ndb = Session()\ntask = resize_image.run(width=200, height=300)\ntask.scheduled_at = func.now() + datetime.timedelta(minutes=3)\ndb.add(task)\n```\n\nPlease note that currently, workers won't wake up at the next exact moment when the scheduled tasks are ready to run.\nIt has to wait until the polling times out, and eventually, it will see the task's scheduled_at time exceeds the current datetime.\nTherefore, depending on your `POLL_TIMEOUT` setting and the number of your workers when they started processing, the actual execution may be inaccurate.\nIf you set the `POLL_TIMEOUT` to 60 seconds, please expect less than 60 seconds of delay.\n\n### Retry\n\nTo automatically retry a task after failure, you can specify a retry policy to the processor.\n\n```python\nimport datetime\nimport bq\nfrom sqlalchemy.orm import Session\n\napp = bq.BeanQueue()\ndelay_retry = bq.DelayRetry(delay=datetime.timedelta(seconds=120))\n\n@app.processor(channel=\"images\", retry_policy=delay_retry)\ndef resize_image(db: Session, task: bq.Task, width: int, height: int):\n    # resize image here ...\n    pass\n```\n\nCurrently, we provide some simple common retry policies such as `DelayRetry` and `ExponentialBackoffRetry`.\nYou can define your retry policy easily by making a function that returns an optional object at the next scheduled time for retry.\n\n```python\ndef my_retry_policy(task: bq.Task) -\u003e typing.Any:\n    # Calculate delay based on task model ...\n    return func.now() + datetime.timedelta(seconds=delay)\n```\n\nTo cap how many attempts are allowed, you can also use `LimitAttempt` like this:\n\n```python\ndelay_retry = bq.DelayRetry(delay=datetime.timedelta(seconds=120))\ncapped_delay_retry = bq.LimitAttempt(3, delay_retry)\n\n@app.processor(channel=\"images\", retry_policy=capped_delay_retry)\ndef resize_image(db: Session, task: bq.Task, width: int, height: int):\n    # Resize image here ...\n    pass\n```\n\nYou can also retry only for specific exception classes with the `retry_exceptions` argument.\n\n```python\n@app.processor(\n    channel=\"images\",\n    retry_policy=delay_retry,\n    retry_exceptions=ValueError,\n)\ndef resize_image(db: Session, task: bq.Task, width: int, height: int):\n    # resize image here ...\n    pass\n```\n\n### Configurations\n\nConfigurations can be modified by setting environment variables with `BQ_` prefix.\nFor example, to set the python packages to scan for processors, you can set `BQ_PROCESSOR_PACKAGES`.\nTo change the PostgreSQL database to connect to, you can set `BQ_DATABASE_URL`.\nThe complete definition of configurations can be found at the [bq/config.py](bq/config.py) module.\n\nIf you want to configure BeanQueue programmatically, you can pass in `Config` object to the `bq.BeanQueue` object when creating.\nFor example:\n\n```python\nimport bq\nfrom .my_config import config\n\nconfig = bq.Config(\n    PROCESSOR_PACKAGES=[\"my_pkgs.processors\"],\n    DATABASE_URL=config.DATABASE_URL,\n    BATCH_SIZE=10,\n)\napp = bq.BeanQueue(config=config)\n```\n\nThen you can pass `--app` argument (or `-a` for short) pointing to the app object to the process command like this:\n\n```bash\nbq -a my_pkgs.bq.app process images\n```\n\nOr if you prefer to define your own process command, you can also call `process_tasks` of the `BeanQueue` object directly like this:\n\n```python\napp.process_tasks(channels=(\"images\",))\n```\n\n### Define your own tables\n\nBeanQueue is designed to be as customizable as much as possible.\nOne of its key features is that you can define your own SQLAlchemy model instead of using the ones we provided.\n\nTo make defining your own `Task`, `Worker` or `Event` model much easier, use bq's mixin classes:\n\n- `bq.TaskModelMixin`: provides task model columns\n- `bq.TaskModelRefWorkerMixin`: provides foreign key column and relationship to `bq.Worker`\n- `bq.TaskModelRefParentMixin`: provides foreign key column and relationship to children `bq.Task` created during processing\n- `bq.TaskModelRefEventMixin`: provides foreign key column and relationship to `bq.Event`\n- `bq.WorkerModelMixin`: provides worker model columns\n- `bq.WorkerRefMixin`: provides relationship to `bq.Task`\n- `bq.EventModelMixin`: provides event model columns\n- `bq.EventModelRefTaskMixin`: provides foreign key column and relationship to `bq.Task`\n\nHere's an example for defining your own Task model:\n\n```python\nimport uuid\n\nfrom sqlalchemy import ForeignKey\nfrom sqlalchemy.dialects.postgresql import UUID\nfrom sqlalchemy.orm import Mapped\nfrom sqlalchemy.orm import mapped_column\nfrom sqlalchemy.orm import relationship\nimport bq\nfrom bq.models.task import listen_events\n\nfrom .base_class import Base\n\n\nclass Task(bq.TaskModelMixin, Base):\n    __tablename__ = \"task\"\n    worker_id: Mapped[uuid.UUID] = mapped_column(\n        UUID(as_uuid=True),\n        ForeignKey(\"worker.id\", onupdate=\"CASCADE\"),\n        nullable=True,\n        index=True,\n    )\n\n    worker: Mapped[\"Worker\"] = relationship(\n        \"Worker\", back_populates=\"tasks\", uselist=False\n    )\n\nlisten_events(Task)\n```\n\nFor task insertion and updates to notify workers, we need to register any custom task types with `bq.models.task.listen_events`.\nIn the example above, this is done right after the Task model definition.\nFor more details and advanced usage, see the definition of `bq.models.task.listen_events`.\n\nYou just see how easy it is to define your Task model. Now, here's an example for defining your own Worker model:\n\n```python\nimport bq\nfrom sqlalchemy.orm import Mapped\nfrom sqlalchemy.orm import relationship\n\nfrom .base_class import Base\n\n\nclass Worker(bq.WorkerModelMixin, Base):\n    __tablename__ = \"worker\"\n\n    tasks: Mapped[list[\"Task\"]] = relationship(\n        \"Task\",\n        back_populates=\"worker\",\n        cascade=\"all,delete\",\n        order_by=\"Task.created_at\",\n    )\n```\n\nWith the model class ready, you only need to change the `TASK_MODEL`, `WORKER_MODEL` and `EVENT_MODEL` of `Config` to the full Python module name plus the class name like this.\n\n```python\nimport bq\nconfig = bq.Config(\n    TASK_MODEL=\"my_pkgs.models.Task\",\n    WORKER_MODEL=\"my_pkgs.models.Worker\",\n    EVENT_MODEL=\"my_pkgs.models.Event\",\n    # ... other configs\n)\napp = bq.BeanQueue(config)\n```\n\n## Why?\n\nThere are countless work queue projects. Why make yet another one?\nThe primary issue with most work queue tools is their reliance on a standalone broker server.\nOur work queue tasks frequently interact with the database, and the atomic nature of database transactions is great for data integrity.\nHowever, integrating an external work queue into the system presents a risk.\nThe work queue and the database don't share the same data view, potentially compromising data integrity and reliability.\n\nFor example, you have a table of `images` to keep the user-uploaded images.\nAnd you have a background work queue for resizing the uploaded images into different thumbnail sizes.\nSo, you will first need to insert a row for the uploaded image about the job into the database before you push the task to the work queue.\n\nSay you push the task to the work queue immediately after you insert the `images` table then commit like this:\n\n```\n1. Insert into the \"images\" table\n2. Push resizing task to the work queue\n3. Commit db changes\n```\n\nWhile this might seem like the right way to do it, there's a hidden bug.\nIf the worker starts too fast before the transaction commits at step 3, it will not be able to see the new row in `images` as it has not been committed yet.\nOne may need to make the task retry a few times to ensure that even if the first attempt failed, it could see the image row in the following attempt.\nBut this adds complexity to the system and also increases the latency if the first attempt fails.\nAlso, if the commit step fails, you will have a failed work queue job trying to fetch a row from the database that will never exist.\n\nAnother approach is to push the resize task after the database changes are committed. It works like this:\n\n```\n1. Insert into the \"images\" table\n2. Commit db changes\n3. Push resizing task to the work queue\n```\n\nWith this approach, we don't need to worry about workers picking up the task too early.\nHowever, there's another drawback.\nIf step 3 for pushing a new task to the work queue fails, the newly inserted `images` row will never be processed.\nThere are many solutions to this problem, but these are all caused by inconsistent data views between the database and the work queue storage.\nThings would be much easier if we had a work queue that shared the same consistent view as the database.\n\nBy using a database as the data storage, all the problems are gone.\nYou can simply do the following:\n\n```\n1. Insert into the \"images\" table\n2. Insert the image resizing task into the `tasks` table\n3. Commit db changes\n```\n\nIt's all or nothing!\nBy doing so, you don't need to maintain another work queue backend.\nYou are probably using a database anyway, so this work queue comes for free.\n\nUsually, a database is inefficient as the work queues data storage because of the potential lock contention and the need for constant querying.\nHowever, things have changed since the [introduction of the SKIP LOCKED](https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/) and [LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) / [NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) features in PostgreSQL or other databases.\n\nThis project is inspired by many of the SKIP-LOCKED-based work queue successors.\nWhy don't we just use those existing tools?\nWell, because while they work great as work queue solutions, they don't take advantage of writing tasks and their relative data into the database in a transaction.\nMany provide an abstraction function or gRPC method for pushing tasks into the database, rather than allowing users to directly insert rows and commit them together.\n\nBeanQueue doesn't overly abstract the logic of publishing a new task into the queue.\nInstead, you insert rows directly, choosing when and what to commit as tasks.\n\n## Sponsor\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://beanhub.io\"\u003e\u003cimg src=\"https://github.com/LaunchPlatform/bq/raw/master/assets/beanhub.svg?raw=true\" alt=\"BeanHub logo\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nA modern accounting book service based on the most popular open source version control system [Git](https://git-scm.com/) and text-based double entry accounting book software [Beancount](https://beancount.github.io/docs/index.html).\n\n## Alternatives\n\n- [solid_queue](https://github.com/rails/solid_queue)\n- [good_job](https://github.com/bensheldon/good_job)\n- [graphile-worker](https://github.com/graphile/worker)\n- [postgres-tq](https://github.com/flix-tech/postgres-tq)\n- [pq](https://github.com/malthe/pq/)\n- [PgQueuer](https://github.com/janbjorge/PgQueuer)\n- [hatchet](https://github.com/hatchet-dev/hatchet)\n- [procrastinate](https://github.com/procrastinate-org/procrastinate)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flaunchplatform%2Fbq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flaunchplatform%2Fbq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flaunchplatform%2Fbq/lists"}