{"id":13414845,"url":"https://github.com/scrapy/queuelib","last_synced_at":"2026-01-29T13:02:59.353Z","repository":{"id":8198347,"uuid":"9629438","full_name":"scrapy/queuelib","owner":"scrapy","description":"Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python","archived":false,"fork":false,"pushed_at":"2025-03-31T12:17:12.000Z","size":143,"stargazers_count":277,"open_issues_count":5,"forks_count":55,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-04-20T15:44:52.839Z","etag":null,"topics":["hacktoberfest","non-persistent","persistent","python","python3","queues"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scrapy.png","metadata":{"files":{"readme":"README.rst","changelog":"NEWS","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2013-04-23T17:51:41.000Z","updated_at":"2025-04-17T16:46:33.000Z","dependencies_parsed_at":"2023-11-19T17:24:06.602Z","dependency_job_id":"102d91d4-d4ec-4b6f-bc72-303149940ada","html_url":"https://github.com/scrapy/queuelib","commit_stats":{"total_commits":108,"total_committers":13,"mean_commits":8.307692307692308,"dds":0.7037037037037037,"last_synced_commit":"eaf206dbb3010d766fbef702cdc5468bae67db4d"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrapy%2Fqueuelib","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrapy%2Fqueuelib/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrapy%2Fqueuelib/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrapy%2Fqueuelib/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scrapy","download_url":"https://codeload.github.com/scrapy/queuelib/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254140768,"owners_count":22021220,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest","non-persistent","persistent","python","python3","queues"],"created_at":"2024-07-30T21:00:37.924Z","updated_at":"2026-01-29T13:02:59.347Z","avatar_url":"https://github.com/scrapy.png","language":"Python","funding_links":[],"categories":["Python","utils"],"sub_categories":[],"readme":"========\nqueuelib\n========\n\n.. image:: https://img.shields.io/pypi/v/queuelib.svg\n   :target: https://pypi.python.org/pypi/queuelib\n\n.. image:: https://img.shields.io/pypi/pyversions/queuelib.svg\n   :target: https://pypi.python.org/pypi/queuelib\n\n.. image:: https://github.com/scrapy/queuelib/actions/workflows/tests-ubuntu.yml/badge.svg\n   :target: https://github.com/scrapy/queuelib/actions/workflows/tests-ubuntu.yml\n\n.. image:: https://img.shields.io/codecov/c/github/scrapy/queuelib/master.svg\n   :target: http://codecov.io/github/scrapy/queuelib?branch=master\n   :alt: Coverage report\n\n\nQueuelib is a Python library that implements object collections which are stored\nin memory or persisted to disk, provide a simple API, and run fast.\n\nQueuelib provides collections for queues_ (FIFO), stacks_ (LIFO), queues\nsorted by priority and queues that are emptied in a round-robin_ fashion.\n\n.. note:: Queuelib collections are not thread-safe.\n\nQueuelib supports Python 3.10+ and has no dependencies.\n\n.. _queues: https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)\n.. _round-robin: https://en.wikipedia.org/wiki/Round-robin_scheduling\n.. _stacks: https://en.wikipedia.org/wiki/Stack_(abstract_data_type)\n\nInstallation\n============\n\nYou can install Queuelib either via the Python Package Index (PyPI) or from\nsource.\n\nTo install using pip::\n\n    $ pip install queuelib\n\nTo install using easy_install::\n\n    $ easy_install queuelib\n\nIf you have downloaded a source tarball you can install it by running the\nfollowing (as root)::\n\n    # python setup.py install\n\nFIFO/LIFO disk queues\n=====================\n\nQueuelib provides FIFO and LIFO queue implementations.\n\nHere is an example usage of the FIFO queue::\n\n    \u003e\u003e\u003e from queuelib import FifoDiskQueue\n    \u003e\u003e\u003e q = FifoDiskQueue(\"queuefile\")\n    \u003e\u003e\u003e q.push(b'a')\n    \u003e\u003e\u003e q.push(b'b')\n    \u003e\u003e\u003e q.push(b'c')\n    \u003e\u003e\u003e q.pop()\n    b'a'\n    \u003e\u003e\u003e q.close()\n    \u003e\u003e\u003e q = FifoDiskQueue(\"queuefile\")\n    \u003e\u003e\u003e q.pop()\n    b'b'\n    \u003e\u003e\u003e q.pop()\n    b'c'\n    \u003e\u003e\u003e q.pop()\n    \u003e\u003e\u003e\n\nThe LIFO queue is identical (API-wise), but importing ``LifoDiskQueue``\ninstead.\n\nPriorityQueue\n=============\n\nA discrete-priority queue implemented by combining multiple FIFO/LIFO queues\n(one per priority).\n\nFirst, select the type of queue to be used per priority (FIFO or LIFO)::\n\n    \u003e\u003e\u003e from queuelib import FifoDiskQueue\n    \u003e\u003e\u003e qfactory = lambda priority: FifoDiskQueue('queue-dir-%s' % priority)\n\nThen instantiate the Priority Queue with it::\n\n    \u003e\u003e\u003e from queuelib import PriorityQueue\n    \u003e\u003e\u003e pq = PriorityQueue(qfactory)\n\nAnd use it::\n\n    \u003e\u003e\u003e pq.push(b'a', 3)\n    \u003e\u003e\u003e pq.push(b'b', 1)\n    \u003e\u003e\u003e pq.push(b'c', 2)\n    \u003e\u003e\u003e pq.push(b'd', 2)\n    \u003e\u003e\u003e pq.pop()\n    b'b'\n    \u003e\u003e\u003e pq.pop()\n    b'c'\n    \u003e\u003e\u003e pq.pop()\n    b'd'\n    \u003e\u003e\u003e pq.pop()\n    b'a'\n\nRoundRobinQueue\n===============\n\nHas nearly the same interface and implementation as a Priority Queue except\nthat each element must be pushed with a (mandatory) key.  Popping from the\nqueue cycles through the keys \"round robin\".\n\nInstantiate the Round Robin Queue similarly to the Priority Queue::\n\n    \u003e\u003e\u003e from queuelib import RoundRobinQueue\n    \u003e\u003e\u003e rr = RoundRobinQueue(qfactory)\n\nAnd use it::\n\n    \u003e\u003e\u003e rr.push(b'a', '1')\n    \u003e\u003e\u003e rr.push(b'b', '1')\n    \u003e\u003e\u003e rr.push(b'c', '2')\n    \u003e\u003e\u003e rr.push(b'd', '2')\n    \u003e\u003e\u003e rr.pop()\n    b'a'\n    \u003e\u003e\u003e rr.pop()\n    b'c'\n    \u003e\u003e\u003e rr.pop()\n    b'b'\n    \u003e\u003e\u003e rr.pop()\n    b'd'\n\n\nMailing list\n============\n\nUse the `scrapy-users`_ mailing list for questions about Queuelib.\n\nBug tracker\n===========\n\nIf you have any suggestions, bug reports or annoyances please report them to\nour issue tracker at: http://github.com/scrapy/queuelib/issues/\n\nContributing\n============\n\nDevelopment of Queuelib happens at GitHub: http://github.com/scrapy/queuelib\n\nYou are highly encouraged to participate in the development. If you don't like\nGitHub (for some reason) you're welcome to send regular patches.\n\nAll changes require tests to be merged.\n\nTests\n=====\n\nTests are located in `queuelib/tests` directory. They can be run using\n`nosetests`_ with the following command::\n\n    nosetests\n\nThe output should be something like the following::\n\n    $ nosetests\n    .............................................................................\n    ----------------------------------------------------------------------\n    Ran 77 tests in 0.145s\n\n    OK\n\nLicense\n=======\n\nThis software is licensed under the BSD License. See the LICENSE file in the\ntop distribution directory for the full license text.\n\nVersioning\n==========\n\nThis software follows `Semantic Versioning`_\n\n.. _Scrapy framework: http://scrapy.org\n.. _scrapy-users: http://groups.google.com/group/scrapy-users\n.. _Semantic Versioning: http://semver.org/\n.. _nosetests: https://nose.readthedocs.org/en/latest/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscrapy%2Fqueuelib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscrapy%2Fqueuelib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscrapy%2Fqueuelib/lists"}