{"id":23380553,"url":"https://github.com/flix-tech/postgres-tq","last_synced_at":"2025-04-10T22:42:44.504Z","repository":{"id":163489120,"uuid":"638868127","full_name":"flix-tech/postgres-tq","owner":"flix-tech","description":"Task queue based on Postgres and compatible with redis-tq","archived":false,"fork":false,"pushed_at":"2025-01-08T12:46:30.000Z","size":156,"stargazers_count":22,"open_issues_count":0,"forks_count":2,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-06T14:14:33.305Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flix-tech.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-10T09:22:00.000Z","updated_at":"2025-03-20T19:34:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"e289d5c0-3024-4407-bf09-35f14fc2349f","html_url":"https://github.com/flix-tech/postgres-tq","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flix-tech%2Fpostgres-tq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flix-tech%2Fpostgres-tq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flix-tech%2Fpostgres-tq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flix-tech%2Fpostgres-tq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flix-tech","download_url":"https://codeload.github.com/flix-tech/postgres-tq/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248312208,"owners_count":21082638,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-21T20:16:44.727Z","updated_at":"2025-04-10T22:42:44.484Z","avatar_url":"https://github.com/flix-tech.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![postgres-tq Actions Status](https://github.com/flix-tech/postgres-tq/workflows/CI/CD%20Pipeline/badge.svg?branch=main)](https://github.com/flix-tech/postgres-tq/actions)\n[![License](https://img.shields.io/github/license/flix-tech/postgres-tq)](https://pypi.org/project/postgres-tq/)\n[![PyPI - Python Version](https://img.shields.io/pypi/v/postgres-tq)](https://pypi.org/project/postgres-tq/)\n\n# Postgres Task Queue\n\nThis library makes it possible to define a list of tasks to be persisted in a Postgres database and executed by multiple workers. Tasks are retried automatically after a given timeout and for a given number of time. Tasks are persisted in the database and executed in order of insertion unless a specific start timestamp is provided.\n\nThis is similar to [our redis-tq](https://github.com/flix-tech/redis-tq) but based on Postgres thanks to the `FOR UPDATE SKIP LOCKED` feature.\n\nSimilar to redis-tq, this package allows for sharing data between multiple processes or hosts.\n\nTasks support a \"lease time\". After that time other workers may consider this client to have crashed or stalled and pick up the item instead. The number of retries can also be configured.\n\nBy default it keeps all the queues and tasks in a single table `task_queue`. If you want to use a different table for different queues for example it could also be configured when instantiating the queue.\n\nYou can set the `create_table=True` when instantiating the queue to have the table created for you. If the table already exist it will not be touched.\n\nWhen defining a task you can provide a `can_start_at` timestamp parameter, and the task will not be executed until then, which can be useful to schedule tasks. By default the current timestamp is used.\n\n## Installation\n\npostgres-tq is available on [PyPI][] so you can simply install via:\n\n```bash\n$ pip install postgres-tq\n```\n\n[PyPI]: https://pypi.org/project/postgres-tq/\n\n## How to use\n\nOn the producing side, populate the queue with tasks and a respective lease timeout:\n\n```py\nfrom datetime import datetime, UTC, timedelta\nfrom postgrestq import TaskQueue\n\ntask_queue = TaskQueue(\n    POSTGRES_CONN_STR,\n    queue_name, # name of the queue as a string\n    reset=True, # delete existing tasks for this queue\n    ttl_zero_callback=handle_failure, # will call handle_failure(task_id, task) when the task failed too many times\n)\n\nfor i in range(10):\n    task_queue.add(\n        some_task,\n        lease_timeout, # in seconds, after this interval it will be assumed to have failed (and the callback is called)\n        ttl=3, # attempts before abandoning\n        can_start_at=datetime.now(UTC) + timedelta(minutes=5), # start it not before than 5 minutes in the future\n    )\n```\n\nOn the consuming side:\n\n```py\nfrom postgrestq import TaskQueue\n\ntask_queue = TaskQueue(POSTGRES_CONN_STR, queue_name, reset=True)\nwhile True:\n    task, task_id, _queue_name = task_queue.get()\n    if task is not None:\n        # do something with task and mark it as complete afterwards\n        task_queue.complete(task_id)\n    if task_queue.is_empty():\n        break\n    # task_queue.get is non-blocking, so you may want to sleep a\n    # bit before the next iteration\n    time.sleep(1)\n```\n\nNotice that `get()` returns the queue name too, in case in future multi-queue is implemented.\nAt the moment it's always the same as the queue_name given to the class.\n\nOr you can even use the \\_\\_iter\\_\\_() method of the class TaskQueue and loop over the queue:\n\n```py\nfrom postgrestq import TaskQueue\n\ntask_queue = TaskQueue(POSTGRES_CONN_STR, queue_name, reset=True)\n\nfor task, id_, queue_name in taskqueue:\n    # do something with task and it's automatically\n    # marked as completed by the iterator at the end\n    # of the iteration\n\n```\n\nIf the consumer crashes (i.e. the task is not marked as completed after lease_timeout seconds), the task will be put back into the task queue. This rescheduling will happen at most ttl times and then the task will be dropped. A callback can be provided if you want to monitor such cases.\n\nAs the tasks are completed, they will remain in the `task_queue`\npostgres table. The table will be deleted of its content if\ninitializing a `TaskQueue` instance with the `reset` flag to `true`\nor if using the `prune_completed_tasks` method:\n\n```py\nfrom postgrestq import TaskQueue\n\n# If reset=True, the full queue content will be deleted\ntask_queue = TaskQueue(POSTGRES_CONN_STR, queue_name, reset=False)\n\n# Prune all tasks from queue completed more than 1 hour (in seconds)\n# ago. Tasks in progress, not started and completed recently will\n# stay in the postgres task_queue table\ntask_queue.prune_completed_tasks(3600)\n\n```\n\n\n## How it works\n\nIt uses row level locks of postgres to mimic the atomic pop and atomic push of redis-tq when getting a new task from the queue:\n\n```sql\nUPDATE task_queue\nSET started_at = current_timestamp\nWHERE id = (\n    SELECT id\n    FROM task_queue\n    WHERE completed_at IS NULL\n        AND started_at IS NULL\n        AND queue_name = \u003cyour_queue_name\u003e\n        AND ttl \u003e 0\n        AND can_start_at \u003c= current_timestamp\n    ORDER BY can_start_at\n    FOR UPDATE SKIP LOCKED\n    LIMIT 1\n)\nRETURNING id, task;\n```\n\nLet's say two workers try to get a new task at the same time, assuming that they will probably see the same task to be picked up using the subquery:\n\n```sql\nSELECT id\nFROM task_queue\nWHERE completed_at IS NULL\n    AND started_at IS NULL\n    AND queue_name = \u003cyour_queue_name\u003e\n    AND ttl \u003e 0\n    AND can_start_at \u003c= current_timestamp\nORDER BY can_start_at\n```\n\nThe first worker locks the row with the `FOR UPDATE` clause until the update is completed and committed. If we hadn't used the `SKIP LOCKED` clause, the second worker would have seen the same row and waited for the first worker to finish the update. However, since the first worker already updated it, the subquery would no longer be valid, and the second worker would return zero rows because `WHERE id = NULL`.\n\nHowever, since we are using `FOR UPDATE SKIP LOCKED` the first worker locks the row for update and the second worker, skips that locked row and chooses another row for itself to update. This way we can avoid the race condition.\n\nThe other methods `complete()` and `reschedule()` work similarly under the hood.\n\n## Running the tests\n\nThe tests will check a presence of an Postgres DB in the port 15432. To initiate one using docker you can run:\n\n```bash\n$ make run-postgres\n```\n\nThen run\n\n```bash\n$ pdm run make test\n```\n\n`pdm run` will ensure that the `make test` command is executed within the context of the virtual environment managed by `pdm`. Make sure you have `pdm` and `Docker` installed for this to work.\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflix-tech%2Fpostgres-tq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflix-tech%2Fpostgres-tq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflix-tech%2Fpostgres-tq/lists"}