{"id":14065344,"url":"https://github.com/malthe/pq","last_synced_at":"2025-05-16T03:04:54.103Z","repository":{"id":10366327,"uuid":"12508068","full_name":"malthe/pq","owner":"malthe","description":"A PostgreSQL job queueing system","archived":false,"fork":false,"pushed_at":"2024-08-03T07:26:01.000Z","size":191,"stargazers_count":378,"open_issues_count":23,"forks_count":41,"subscribers_count":13,"default_branch":"1.x","last_synced_at":"2025-05-12T01:07:02.828Z","etag":null,"topics":["postgresql","python","queue"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/malthe.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGES.rst","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-08-31T16:18:40.000Z","updated_at":"2025-04-28T14:19:44.000Z","dependencies_parsed_at":"2024-02-02T09:25:19.662Z","dependency_job_id":"5d8347d9-81c5-4e0e-9526-2d56545e0c1b","html_url":"https://github.com/malthe/pq","commit_stats":{"total_commits":144,"total_committers":19,"mean_commits":7.578947368421052,"dds":"0.47916666666666663","last_synced_commit":"faf73d3a526c2eb43cc83bfd8cfff149549dc639"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malthe%2Fpq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malthe%2Fpq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malthe%2Fpq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malthe%2Fpq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/malthe","download_url":"https://codeload.github.com/malthe/pq/tar.gz/refs/heads/1.x","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254459088,"owners_count":22074605,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["postgresql","python","queue"],"created_at":"2024-08-13T07:04:26.479Z","updated_at":"2025-05-16T03:04:49.088Z","avatar_url":"https://github.com/malthe.png","language":"Python","readme":"PQ\n**\n\nA transactional queue system for PostgreSQL written in Python.\n\n.. figure:: https://pq.readthedocs.org/en/latest/_static/intro.svg\n   :alt: PQ does the job!\n\nIt allows you to push and pop items in and out of a queue in various\nways and also provides two scheduling options: delayed processing and\nprioritization.\n\nThe system uses a single table that holds all jobs across queues; the\nspecifics are easy to customize.\n\nThe system currently supports only the `psycopg2\n\u003chttps://pypi.python.org/pypi/psycopg2\u003e`_ database driver - or\n`psycopg2cffi \u003chttps://pypi.python.org/pypi/psycopg2cffi\u003e`_ for PyPy.\n\nThe basic queue implementation is similar to Ryan Smith's\n`queue_classic \u003chttps://github.com/ryandotsmith/queue_classic\u003e`_\nlibrary written in Ruby, but uses `SKIP LOCKED\n\u003chttps://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/\u003e`_\nfor concurrency control.\n\nIn terms of performance, the implementation clock in at about 1,000\noperations per second. Using the `PyPy \u003chttp://pypy.org/\u003e`_\ninterpreter, this scales linearly with the number of cores available.\n\n\nGetting started\n===============\n\nAll functionality is encapsulated in a single class ``PQ``.\n\n     ``class PQ(conn=None, pool=None, table=\"queue\", schema=None)``\n\nThe optional ``schema`` argument can be used to qualify the table with\na schema if necessary.\n\nExample usage:\n\n.. code-block:: python\n\n    from psycopg2 import connect\n    from pq import PQ\n\n    conn = connect('dbname=example user=postgres')\n    pq = PQ(conn)\n\nFor multi-threaded operation, use a connection pool such as\n``psycopg2.pool.ThreadedConnectionPool``.\n\nYou probably want to make sure your database is created with the\n``utf-8`` encoding.\n\nTo create and configure the queue table, call the ``create()`` method.\n\n.. code-block:: python\n\n    pq.create()\n\n\nQueues\n======\n\nThe ``pq`` object exposes queues through Python's dictionary\ninterface:\n\n.. code-block:: python\n\n    queue = pq['apples']\n\nThe ``queue`` object provides ``get`` and ``put`` methods as explained\nbelow, and in addition, it also works as a context manager where it\nmanages a transaction:\n\n.. code-block:: python\n\n    with queue as cursor:\n        ...\n\nThe statements inside the context manager are either committed as a\ntransaction or rejected, atomically. This is useful when a queue is\nused to manage jobs because it allows you to retrieve a job from the\nqueue, perform a job and write a result, with transactional\nsemantics.\n\nMethods\n=======\n\nUse the ``put(data)`` method to insert an item into the queue. It\ntakes a JSON-compatible object such as a Python dictionary:\n\n.. code-block:: python\n\n    queue.put({'kind': 'Cox'})\n    queue.put({'kind': 'Arthur Turner'})\n    queue.put({'kind': 'Golden Delicious'})\n\nItems are pulled out of the queue using ``get(block=True)``. The\ndefault behavior is to block until an item is available with a default\ntimeout of one second after which a value of ``None`` is returned.\n\n.. code-block:: python\n\n    def eat(kind):\n        print 'umm, %s apples taste good.' % kind\n\n    job = queue.get()\n    eat(**job.data)\n\nThe ``job`` object provides additional metadata in addition to the\n``data`` attribute as illustrated by the string representation:\n\n    \u003e\u003e\u003e job\n    \u003cpq.Job id=77709 size=1 enqueued_at=\"2014-02-21T16:22:06Z\" schedule_at=None\u003e\n\nThe ``get`` operation is also available through iteration:\n\n.. code-block:: python\n\n    for job in queue:\n        if job is None:\n            break\n\n        eat(**job.data)\n\nThe iterator blocks if no item is available. Again, there is a default\ntimeout of one second, after which the iterator yields a value of\n``None``.\n\nAn application can then choose to break out of the loop, or wait again\nfor an item to be ready.\n\n.. code-block:: python\n\n    for job in queue:\n        if job is not None:\n            eat(**job.data)\n\n        # This is an infinite loop!\n\n\nScheduling\n==========\n\nItems can be scheduled such that they're not pulled until a later\ntime:\n\n.. code-block:: python\n\n    queue.put({'kind': 'Cox'}, '5m')\n\nIn this example, the item is ready for work five minutes later. The\nmethod also accepts ``datetime`` and ``timedelta`` objects.\n\n\nPriority\n========\n\nIf some items are more important than others, a time expectation can\nbe expressed:\n\n.. code-block:: python\n\n    queue.put({'kind': 'Cox'}, expected_at='5m')\n\nThis tells the queue processor to give priority to this item over an\nitem expected at a later time, and conversely, to prefer an item with\nan earlier expected time. Note that items without a set priority are\npulled last.\n\nThe scheduling and priority options can be combined:\n\n.. code-block:: python\n\n    queue.put({'kind': 'Cox'}, '1h', '2h')\n\nThis item won't be pulled out until after one hour, and even then,\nit's only processed subject to it's priority of two hours.\n\n\nEncoding and decoding\n=====================\n\nThe task data is encoded and decoded into JSON using the built-in\n`json` module. If you want to use a different implementation or need\nto configure this, pass `encode` and/or `decode` arguments to the `PQ`\nconstructor.\n\n\nPickles\n=======\n\nIf a queue name is provided as ``\u003cname\u003e/pickle``\n(e.g. ``'jobs/pickle'``), items are automatically pickled and\nunpickled using Python's built-in ``cPickle`` module:\n\n.. code-block:: python\n\n    queue = pq['apples/pickle']\n\n    class Apple(object):\n        def __init__(self, kind):\n           self.kind = kind\n\n    queue.put(Apple('Cox'))\n\nThis allows you to store most objects without having to add any\nfurther serialization code.\n\nThe old pickle protocol ``0`` is used to ensure the pickled data is\nencoded as ``ascii`` which should be compatible with any database\nencoding. Note that the pickle data is still wrapped as a JSON string at the\ndatabase level.\n\nWhile using the pickle protocol is an easy way to serialize objects,\nfor advanced users t might be better to use JSON serialization\ndirectly on the objects, using for example the object hook mechanism\nin the built-in `json` module or subclassing\n`JSONEncoder \u003chttps://docs.python.org/2/library/json.html#json.JSONEncoder\u003e`.\n\n\nTasks\n=====\n\n``pq`` comes with a higher level ``API`` that helps to manage ``tasks``.\n\n\n.. code-block:: python\n\n    from pq.tasks import PQ\n\n    pq = PQ(...)\n\n    queue = pq['default']\n\n    @queue.task(schedule_at='1h')\n    def eat(job_id, kind):\n        print 'umm, %s apples taste good.' % kind\n\n    eat('Cox')\n\n    queue.work()\n\n\n``tasks``'s ``jobs`` can optionally be re-scheduled on failure:\n\n.. code-block:: python\n\n    @queue.task(schedule_at='1h', max_retries=2, retry_in='10s')\n    def eat(job_id, kind):\n        # ...\n\n\nTime expectations can be overriden at ``task`` call:\n\n.. code-block:: python\n\n    eat('Cox', _expected_at='2m', _schedule_at='1m')\n\n\n** NOTE ** First positional argument is id of job. It's PK of record in PostgreSQL.\n\nThread-safety\n=============\n\nAll objects are thread-safe as long as a connection pool is provided\nwhere each thread receives its own database connection.\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmalthe%2Fpq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmalthe%2Fpq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmalthe%2Fpq/lists"}