{"id":23536881,"url":"https://github.com/peter-wangxu/persist-queue","last_synced_at":"2025-05-16T10:06:03.085Z","repository":{"id":13548683,"uuid":"74737878","full_name":"peter-wangxu/persist-queue","owner":"peter-wangxu","description":"A thread-safe disk based persistent queue in Python","archived":false,"fork":false,"pushed_at":"2024-07-09T17:31:40.000Z","size":279,"stargazers_count":350,"open_issues_count":26,"forks_count":49,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-05-12T20:56:10.706Z","etag":null,"topics":["mysql","persistent-queue","python","sqlite","thread-safety"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/peter-wangxu.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-11-25T08:04:47.000Z","updated_at":"2025-04-22T19:45:44.000Z","dependencies_parsed_at":"2024-02-08T04:27:30.536Z","dependency_job_id":"4e497b62-4f24-4e97-9875-e745b528d05f","html_url":"https://github.com/peter-wangxu/persist-queue","commit_stats":{"total_commits":142,"total_committers":23,"mean_commits":6.173913043478261,"dds":0.7887323943661972,"last_synced_commit":"f3f3682778161ff8e38d65372babb34ac07f40b3"},"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peter-wangxu%2Fpersist-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peter-wangxu%2Fpersist-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peter-wangxu%2Fpersist-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peter-wangxu%2Fpersist-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/peter-wangxu","download_url":"https://codeload.github.com/peter-wangxu/persist-queue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254509477,"owners_count":22082891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mysql","persistent-queue","python","sqlite","thread-safety"],"created_at":"2024-12-26T02:37:01.272Z","updated_at":"2025-05-16T10:06:03.026Z","avatar_url":"https://github.com/peter-wangxu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"persist-queue - A thread-safe, disk-based queue for Python\n==========================================================\n\n.. image:: https://img.shields.io/circleci/project/github/peter-wangxu/persist-queue/master.svg?label=Linux%20%26%20Mac\n    :target: https://circleci.com/gh/peter-wangxu/persist-queue\n\n.. image:: https://img.shields.io/appveyor/ci/peter-wangxu/persist-queue/master.svg?label=Windows\n    :target: https://ci.appveyor.com/project/peter-wangxu/persist-queue\n\n.. image:: https://img.shields.io/codecov/c/github/peter-wangxu/persist-queue/master.svg\n    :target: https://codecov.io/gh/peter-wangxu/persist-queue\n\n.. image:: https://img.shields.io/pypi/v/persist-queue.svg\n    :target: https://pypi.python.org/pypi/persist-queue\n\n.. image:: https://img.shields.io/pypi/pyversions/persist-queue\n   :alt: PyPI - Python Version\n\n``persist-queue`` implements a file-based queue and a serial of sqlite3-based queues. The goals is to achieve following requirements:\n\n* Disk-based: each queued item should be stored in disk in case of any crash.\n* Thread-safe: can be used by multi-threaded producers and multi-threaded consumers.\n* Recoverable: Items can be read after process restart.\n* Green-compatible: can be used in ``greenlet`` or ``eventlet`` environment.\n\nWhile *queuelib* and *python-pqueue* cannot fulfil all of above. After some try, I found it's hard to achieve based on their current\nimplementation without huge code change. this is the motivation to start this project.\n\nBy default, *persist-queue* use *pickle* object serialization module to support object instances.\nMost built-in type, like `int`, `dict`, `list` are able to be persisted by `persist-queue` directly, to support customized objects,\nplease refer to `Pickling and unpickling extension types(Python2) \u003chttps://docs.python.org/2/library/pickle.html#pickling-and-unpickling-normal-class-instances\u003e`_\nand `Pickling Class Instances(Python3) \u003chttps://docs.python.org/3/library/pickle.html#pickling-class-instances\u003e`_\n\nThis project is based on the achievements of `python-pqueue \u003chttps://github.com/balena/python-pqueue\u003e`_\nand `queuelib \u003chttps://github.com/scrapy/queuelib\u003e`_\n\nSlack channels\n^^^^^^^^^^^^^^\n\nJoin `persist-queue \u003chttps://join.slack\n.com/t/persist-queue/shared_invite\n/enQtOTM0MDgzNTQ0MDg3LTNmN2IzYjQ1MDc0MDYzMjI4OGJmNmVkNWE3ZDBjYzg5MDc0OWUzZDJkYTkwODdkZmYwODdjNjUzMTk3MWExNDE\u003e`_ channel\n\n\nRequirements\n------------\n* Python 3.5 or newer versions (refer to `Deprecation`_ for older Python versions)\n* Full support for Linux and MacOS.\n* Windows support (with `Caution`_ if ``persistqueue.Queue`` is used).\n\nFeatures\n--------\n\n- Multiple platforms support: Linux, macOS, Windows\n- Pure python\n- Both filed based queues and sqlite3 based queues are supported\n- Filed based queue: multiple serialization protocol support: pickle(default), msgpack, cbor, json\n\nDeprecation\n-----------\n- `persist-queue` drops Python 2 support since version `1.0.0`, no new feature will be developed under Python 2 as `Python 2 was sunset on January 1, 2020 \u003chttps://www.python.org/doc/sunset-python-2/\u003e`_.\n- `Python 3.4 release has reached end of life \u003chttps://www.python.org/downloads/release/python-3410/\u003e`_ and\n  `DBUtils \u003chttps://webwareforpython.github.io/DBUtils/changelog.html\u003e`_ ceased support for `Python 3.4`, `persist queue` drops MySQL based queue for python 3.4 since version 0.8.0.\n  other queue implementations such as file based queue and sqlite3 based queue are still workable.\n\nInstallation\n------------\n\nfrom pypi\n^^^^^^^^^\n\n.. code-block:: console\n\n    pip install persist-queue\n    # for msgpack, cbor and mysql support, use following command\n    pip install \"persist-queue[extra]\"\n\n\nfrom source code\n^^^^^^^^^^^^^^^^\n\n.. code-block:: console\n\n    git clone https://github.com/peter-wangxu/persist-queue\n    cd persist-queue\n    # for msgpack and cbor support, run 'pip install -r extra-requirements.txt' first\n    python setup.py install\n\n\nBenchmark\n---------\n\nHere are the time spent(in seconds) for writing/reading **1000** items to the\ndisk comparing the sqlite3 and file queue.\n\n- Windows\n    - OS: Windows 10\n    - Disk: SATA3 SSD\n    - RAM: 16 GiB\n\n+---------------+---------+-------------------------+----------------------------+\n|               | Write   | Write/Read(1 task_done) | Write/Read(many task_done) |\n+---------------+---------+-------------------------+----------------------------+\n| SQLite3 Queue | 1.8880  | 2.0290                  | 3.5940                     |\n+---------------+---------+-------------------------+----------------------------+\n| File Queue    | 4.9520  | 5.0560                  | 8.4900                     |\n+---------------+---------+-------------------------+----------------------------+\n\n**windows note**\nPerformance of Windows File Queue has dramatic improvement since `v0.4.1` due to the\natomic renaming support(3-4X faster)\n\n- Linux\n    - OS: Ubuntu 16.04 (VM)\n    - Disk: SATA3 SSD\n    - RAM:  4 GiB\n\n+---------------+--------+-------------------------+----------------------------+\n|               | Write  | Write/Read(1 task_done) | Write/Read(many task_done) |\n+---------------+--------+-------------------------+----------------------------+\n| SQLite3 Queue | 1.8282 | 1.8075                  | 2.8639                     |\n+---------------+--------+-------------------------+----------------------------+\n| File Queue    | 0.9123 | 1.0411                  | 2.5104                     |\n+---------------+--------+-------------------------+----------------------------+\n\n- Mac OS\n    - OS: 10.14 (macOS Mojave)\n    - Disk: PCIe SSD\n    - RAM:  16 GiB\n\n+---------------+--------+-------------------------+----------------------------+\n|               | Write  | Write/Read(1 task_done) | Write/Read(many task_done) |\n+---------------+--------+-------------------------+----------------------------+\n| SQLite3 Queue | 0.1879 | 0.2115                  | 0.3147                     |\n+---------------+--------+-------------------------+----------------------------+\n| File Queue    | 0.5158 | 0.5357                  | 1.0446                     |\n+---------------+--------+-------------------------+----------------------------+\n\n**note**\n\n- The value above is in seconds for reading/writing *1000* items, the less\n  the better\n- Above result was got from:\n\n.. code-block:: console\n\n    python benchmark/run_benchmark.py 1000\n\n\nTo see the real performance on your host, run the script under ``benchmark/run_benchmark.py``:\n\n.. code-block:: console\n\n    python benchmark/run_benchmark.py \u003cCOUNT, default to 100\u003e\n\n\nExamples\n--------\n\n\nExample usage with a SQLite3 based queue\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code-block:: python\n\n    \u003e\u003e\u003e import persistqueue\n    \u003e\u003e\u003e q = persistqueue.SQLiteQueue('mypath', auto_commit=True)\n    \u003e\u003e\u003e q.put('str1')\n    \u003e\u003e\u003e q.put('str2')\n    \u003e\u003e\u003e q.put('str3')\n    \u003e\u003e\u003e q.get()\n    'str1'\n    \u003e\u003e\u003e del q\n\n\nClose the console, and then recreate the queue:\n\n.. code-block:: python\n\n   \u003e\u003e\u003e import persistqueue\n   \u003e\u003e\u003e q = persistqueue.SQLiteQueue('mypath', auto_commit=True)\n   \u003e\u003e\u003e q.get()\n   'str2'\n   \u003e\u003e\u003e\n\nNew functions:\n*Available since v0.8.0*\n\n- ``shrink_disk_usage`` perform a ``VACUUM`` against the sqlite, and rebuild the database file, this usually takes long time and frees a lot of disk space after ``get()``\n\n\nExample usage of SQLite3 based ``UniqueQ``\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nThis queue does not allow duplicate items.\n\n.. code-block:: python\n\n   \u003e\u003e\u003e import persistqueue\n   \u003e\u003e\u003e q = persistqueue.UniqueQ('mypath')\n   \u003e\u003e\u003e q.put('str1')\n   \u003e\u003e\u003e q.put('str1')\n   \u003e\u003e\u003e q.size\n   1\n   \u003e\u003e\u003e q.put('str2')\n   \u003e\u003e\u003e q.size\n   2\n   \u003e\u003e\u003e\n\nExample usage of SQLite3 based ``SQLiteAckQueue``/``UniqueAckQ``\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nThe core functions:\n\n- ``put``: add item to the queue. Returns ``id``\n- ``get``: get item from queue and mark as unack.  Returns ``item``, Optional paramaters (``block``, ``timeout``, ``id``, ``next_in_order``, ``raw``)\n- ``update``: update an item. Returns ``id``, Paramaters (``item``), Optional parameter if item not in raw format (``id``)\n- ``ack``: mark item as acked. Returns ``id``, Parameters (``item`` or ``id``)\n- ``nack``: there might be something wrong with current consumer, so mark item as ready and new consumer will get it.  Returns ``id``, Parameters (``item`` or ``id``)\n- ``ack_failed``: there might be something wrong during process, so just mark item as failed. Returns ``id``, Parameters (``item`` or ``id``)\n- ``clear_acked_data``: perform a sql delete agaist sqlite. It removes 1000 items, while keeping 1000 of the most recent, whose status is ``AckStatus.acked`` (note: this does not shrink the file size on disk) Optional paramters (``max_delete``, ``keep_latest``, ``clear_ack_failed``)\n- ``shrink_disk_usage`` perform a ``VACUUM`` against the sqlite, and rebuild the database file, this usually takes long time and frees a lot of disk space after ``clear_acked_data``\n- ``queue``: returns the database contents as a Python List[Dict]\n- ``active_size``: The active size changes when an item is added (put) and completed (ack/ack_failed) unlike ``qsize`` which changes when an item is pulled (get) or returned (nack).\n\n.. code-block:: python\n\n   \u003e\u003e\u003e import persistqueue\n   \u003e\u003e\u003e ackq = persistqueue.SQLiteAckQueue('path')\n   \u003e\u003e\u003e ackq.put('str1')\n   \u003e\u003e\u003e item = ackq.get()\n   \u003e\u003e\u003e # Do something with the item\n   \u003e\u003e\u003e ackq.ack(item) # If done with the item\n   \u003e\u003e\u003e ackq.nack(item) # Else mark item as `nack` so that it can be proceeded again by any worker\n   \u003e\u003e\u003e ackq.ack_failed(item) # Or else mark item as `ack_failed` to discard this item\n\nParameters:\n\n- ``clear_acked_data``\n    - ``max_delete`` (defaults to 1000): This is the LIMIT.  How many items to delete.\n    - ``keep_latest`` (defaults to 1000): This is the OFFSET.  How many recent items to keep.\n    - ``clear_ack_failed`` (defaults to False): Clears the ``AckStatus.ack_failed`` in addition to the ``AckStatus.ack``.\n\n- ``get``\n    - ``raw`` (defaults to False): Returns the metadata along with the record, which includes the id (``pqid``) and timestamp.  On the SQLiteAckQueue, the raw results can be ack, nack, ack_failed similar to the normal return.\n    -  ``id`` (defaults to None): Accepts an `id` or a raw item containing ``pqid``.  Will select the item based on the row id.\n    -  ``next_in_order`` (defaults to False): Requires the ``id`` attribute.  This option tells the SQLiteAckQueue/UniqueAckQ to get the next item based on  ``id``, not the first available.  This allows the user to get, nack, get, nack and progress down the queue, instead of continuing to get the same nack'd item over again.\n\n``raw`` example:\n\n.. code-block:: python\n\n   \u003e\u003e\u003e q.put('val1')\n   \u003e\u003e\u003e d = q.get(raw=True)\n   \u003e\u003e\u003e print(d)\n   \u003e\u003e\u003e {'pqid': 1, 'data': 'val1', 'timestamp': 1616719225.012912}\n   \u003e\u003e\u003e q.ack(d)\n\n``next_in_order`` example:\n\n.. code-block:: python\n\n   \u003e\u003e\u003e q.put(\"val1\")\n   \u003e\u003e\u003e q.put(\"val2\")\n   \u003e\u003e\u003e q.put(\"val3\")\n   \u003e\u003e\u003e item = q.get()\n   \u003e\u003e\u003e id = q.nack(item)\n   \u003e\u003e\u003e item = q.get(id=id, next_in_order=True)\n   \u003e\u003e\u003e print(item)\n   \u003e\u003e\u003e val2\n\n\nNote:\n\n1. The SQLiteAckQueue always uses \"auto_commit=True\".\n2. The Queue could be set in non-block style, e.g. \"SQLiteAckQueue.get(block=False, timeout=5)\".\n3. ``UniqueAckQ`` only allows for unique items\n\nExample usage with a file based queue\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nParameters:\n\n- ``path``: specifies the directory wher enqueued data persisted.\n- ``maxsize``: indicates the maximum size stored in the queue, if maxsize\u003c=0 the queue is unlimited.\n- ``chunksize``: indicates how many entries should exist in each chunk file on disk. When a all entries in a chunk file was dequeued by get(), the file would be removed from filesystem.\n- ``tempdir``: indicates where temporary files should be stored. The tempdir has to be located on the same disk as the enqueued data in order to obtain atomic operations.\n- ``serializer``: controls how enqueued data is serialized.\n- ``auto_save``: `True` or `False`. By default, the change is only persisted when task_done() is called. If autosave is enabled, info data is persisted immediately when get() is called. Adding data to the queue with put() will always persist immediately regardless of this setting.\n\n.. code-block:: python\n\n    \u003e\u003e\u003e from persistqueue import Queue\n    \u003e\u003e\u003e q = Queue(\"mypath\")\n    \u003e\u003e\u003e q.put('a')\n    \u003e\u003e\u003e q.put('b')\n    \u003e\u003e\u003e q.put('c')\n    \u003e\u003e\u003e q.get()\n    'a'\n    \u003e\u003e\u003e q.task_done()\n\n\nClose the python console, and then we restart the queue from the same path,\n\n.. code-block:: python\n\n    \u003e\u003e\u003e from persistqueue import Queue\n    \u003e\u003e\u003e q = Queue('mypath')\n    \u003e\u003e\u003e q.get()\n    'b'\n    \u003e\u003e\u003e q.task_done()\n\nExample usage with an auto-saving file based queue\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n*Available since: v0.5.0*\n\nBy default, items added to the queue are persisted during the ``put()`` call,\nand items removed from a queue are only persisted when ``task_done()`` is\ncalled.\n\n.. code-block:: python\n\n    \u003e\u003e\u003e from persistqueue import Queue\n    \u003e\u003e\u003e q = Queue(\"mypath\")\n    \u003e\u003e\u003e q.put('a')\n    \u003e\u003e\u003e q.put('b')\n    \u003e\u003e\u003e q.get()\n    'a'\n    \u003e\u003e\u003e q.get()\n    'b'\n\nAfter exiting and restarting the queue from the same path, we see the items\nremain in the queue, because ``task_done()`` wasn't called before.\n\n.. code-block:: python\n\n    \u003e\u003e\u003e from persistqueue import Queue\n    \u003e\u003e\u003e q = Queue('mypath')\n    \u003e\u003e\u003e q.get()\n    'a'\n    \u003e\u003e\u003e q.get()\n    'b'\n\nThis can be advantageous. For example, if your program crashes before finishing\nprocessing an item, it will remain in the queue after restarting. You can also\nspread out the ``task_done()`` calls for performance reasons to avoid lots of\nindividual writes.\n\nUsing ``autosave=True`` on a file based queue will automatically save on every\ncall to ``get()``. Calling ``task_done()`` is not necessary, but may still be\nused to ``join()`` against the queue.\n\n.. code-block:: python\n\n    \u003e\u003e\u003e from persistqueue import Queue\n    \u003e\u003e\u003e q = Queue(\"mypath\", autosave=True)\n    \u003e\u003e\u003e q.put('a')\n    \u003e\u003e\u003e q.put('b')\n    \u003e\u003e\u003e q.get()\n    'a'\n\nAfter exiting and restarting the queue from the same path, only the second item\nremains:\n\n.. code-block:: python\n\n    \u003e\u003e\u003e from persistqueue import Queue\n    \u003e\u003e\u003e q = Queue('mypath', autosave=True)\n    \u003e\u003e\u003e q.get()\n    'b'\n\n\nExample usage with a SQLite3 based dict\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code-block:: python\n\n    \u003e\u003e\u003e from persisitqueue import PDict\n    \u003e\u003e\u003e q = PDict(\"testpath\", \"testname\")\n    \u003e\u003e\u003e q['key1'] = 123\n    \u003e\u003e\u003e q['key2'] = 321\n    \u003e\u003e\u003e q['key1']\n    123\n    \u003e\u003e\u003e len(q)\n    2\n    \u003e\u003e\u003e del q['key1']\n    \u003e\u003e\u003e q['key1']\n    Traceback (most recent call last):\n      File \"\u003cstdin\u003e\", line 1, in \u003cmodule\u003e\n      File \"persistqueue\\pdict.py\", line 58, in __getitem__\n        raise KeyError('Key: {} not exists.'.format(item))\n    KeyError: 'Key: key1 not exists.'\n\nClose the console and restart the PDict\n\n\n.. code-block:: python\n\n    \u003e\u003e\u003e from persisitqueue import PDict\n    \u003e\u003e\u003e q = PDict(\"testpath\", \"testname\")\n    \u003e\u003e\u003e q['key2']\n    321\n\n\nMulti-thread usage for **SQLite3** based queue\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code-block:: python\n\n    from persistqueue import FIFOSQLiteQueue\n\n    q = FIFOSQLiteQueue(path=\"./test\", multithreading=True)\n\n    def worker():\n        while True:\n            item = q.get()\n            do_work(item)\n\n    for i in range(num_worker_threads):\n         t = Thread(target=worker)\n         t.daemon = True\n         t.start()\n\n    for item in source():\n        q.put(item)\n\n\nmulti-thread usage for **Queue**\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code-block:: python\n\n    from persistqueue import Queue\n\n    q = Queue()\n\n    def worker():\n        while True:\n            item = q.get()\n            do_work(item)\n            q.task_done()\n\n    for i in range(num_worker_threads):\n         t = Thread(target=worker)\n         t.daemon = True\n         t.start()\n\n    for item in source():\n        q.put(item)\n\n    q.join()       # block until all tasks are done\n\nExample usage with a MySQL based queue\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n*Available since: v0.8.0*\n\n.. code-block:: python\n\n    \u003e\u003e\u003e import persistqueue\n    \u003e\u003e\u003e db_conf = {\n    \u003e\u003e\u003e     \"host\": \"127.0.0.1\",\n    \u003e\u003e\u003e     \"user\": \"user\",\n    \u003e\u003e\u003e     \"passwd\": \"passw0rd\",\n    \u003e\u003e\u003e     \"db_name\": \"testqueue\",\n    \u003e\u003e\u003e     # \"name\": \"\",\n    \u003e\u003e\u003e     \"port\": 3306\n    \u003e\u003e\u003e }\n    \u003e\u003e\u003e q = persistqueue.MySQLQueue(name=\"testtable\", **db_conf)\n    \u003e\u003e\u003e q.put('str1')\n    \u003e\u003e\u003e q.put('str2')\n    \u003e\u003e\u003e q.put('str3')\n    \u003e\u003e\u003e q.get()\n    'str1'\n    \u003e\u003e\u003e del q\n\n\nClose the console, and then recreate the queue:\n\n.. code-block:: python\n\n   \u003e\u003e\u003e import persistqueue\n   \u003e\u003e\u003e q = persistqueue.MySQLQueue(name=\"testtable\", **db_conf)\n   \u003e\u003e\u003e q.get()\n   'str2'\n   \u003e\u003e\u003e\n\n\n\n**note**\n\nDue to the limitation of file queue described in issue `#89 \u003chttps://github.com/peter-wangxu/persist-queue/issues/89\u003e`_,\n`task_done` in one thread may acknowledge items in other threads which should not be. Considering the `SQLiteAckQueue` if you have such requirement.\n\n\nSerialization via msgpack/cbor/json\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n- v0.4.1: Currently only available for file based Queue\n- v0.4.2: Also available for SQLite3 based Queues\n\n.. code-block:: python\n\n    \u003e\u003e\u003e from persistqueue\n    \u003e\u003e\u003e q = persistqueue.Queue('mypath', serializer=persistqueue.serializers.msgpack)\n    \u003e\u003e\u003e # via cbor2\n    \u003e\u003e\u003e # q = persistqueue.Queue('mypath', serializer=persistqueue.serializers.cbor2)\n    \u003e\u003e\u003e # via json\n    \u003e\u003e\u003e # q = Queue('mypath', serializer=persistqueue.serializers.json)\n    \u003e\u003e\u003e q.get()\n    'b'\n    \u003e\u003e\u003e q.task_done()\n\nExplicit resource reclaim\n^^^^^^^^^^^^^^^^^^^^^^^^^\n\nFor some reasons, an application may require explicit reclamation for file\nhandles or sql connections before end of execution. In these cases, user can\nsimply call:\n.. code-block:: python\n\n    q = Queue() # or q = persistqueue.SQLiteQueue('mypath', auto_commit=True)\n    del q\n\n\nto reclaim related file handles or sql connections.\n\nTips\n----\n\n``task_done`` is required both for file based queue and SQLite3 based queue (when ``auto_commit=False``)\nto persist the cursor of next ``get`` to the disk.\n\n\nPerformance impact\n------------------\n\n- **WAL**\n\n  Starting on v0.3.2, the ``persistqueue`` is leveraging the sqlite3 builtin feature\n  `WAL \u003chttps://www.sqlite.org/wal.html\u003e`_ which can improve the performance\n  significantly, a general testing indicates that ``persistqueue`` is 2-4 times\n  faster than previous version.\n\n- **auto_commit=False**\n\n  Since persistqueue v0.3.0, a new parameter ``auto_commit`` is introduced to tweak\n  the performance for sqlite3 based queues as needed. When specify ``auto_commit=False``, user\n  needs to perform ``queue.task_done()`` to persist the changes made to the disk since\n  last ``task_done`` invocation.\n\n- **pickle protocol selection**\n\n  From v0.3.6, the ``persistqueue`` will select ``Protocol version 2`` for python2 and ``Protocol version 4`` for python3\n  respectively. This selection only happens when the directory is not present when initializing the queue.\n\nTests\n-----\n\n*persist-queue* use ``tox`` to trigger tests.\n\n- Unit test\n\n.. code-block:: console\n\n    tox -e \u003cPYTHON_VERSION\u003e\n\nAvailable ``\u003cPYTHON_VERSION\u003e``: ``py27``, ``py34``, ``py35``, ``py36``, ``py37``\n\n\n- PEP8 check\n\n.. code-block:: console\n\n   tox -e pep8\n\n\n`pyenv \u003chttps://github.com/pyenv/pyenv\u003e`_ is usually a helpful tool to manage multiple versions of Python.\n\nCaution\n-------\n\nCurrently, the atomic operation is supported on Windows while still in experimental,\nThat's saying, the data in ``persistqueue.Queue`` could be in unreadable state when an incidental failure occurs during ``Queue.task_done``.\n\n**DO NOT put any critical data on persistqueue.queue on Windows**.\n\n\nContribution\n------------\n\nSimply fork this repo and send PR for your code change(also tests to cover your change), remember to give a title and description of your PR. I am willing to\nenhance this project with you :).\n\n\nLicense\n-------\n\n`BSD \u003cLICENSE\u003e`_\n\nContributors\n------------\n\n`Contributors \u003chttps://github.com/peter-wangxu/persist-queue/graphs/contributors\u003e`_\n\nFAQ\n---\n\n* ``sqlite3.OperationalError: database is locked`` is raised.\n\npersistqueue open 2 connections for the db if ``multithreading=True``, the\nSQLite database is locked until that transaction is committed. The ``timeout``\nparameter specifies how long the connection should wait for the lock to go away\nuntil raising an exception. Default time is **10**, increase ``timeout``\nwhen creating the queue if above error occurs.\n\n* sqlite3 based queues are not thread-safe.\n\nThe sqlite3 queues are heavily tested under multi-threading environment, if you find it's not thread-safe, please\nmake sure you set the ``multithreading=True`` when initializing the queue before submitting new issue:).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpeter-wangxu%2Fpersist-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpeter-wangxu%2Fpersist-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpeter-wangxu%2Fpersist-queue/lists"}