Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/coleifer/huey

a little task queue for python
https://github.com/coleifer/huey

dank django python queue redis task-queue

Last synced: 16 days ago
JSON representation

a little task queue for python

Awesome Lists containing this project

README 

          
.. image:: https://media.charlesleifer.com/blog/photos/huey3-logo.png



*a lightweight alternative*.



huey is:



* a task queue

* written in python

* clean and simple API



huey has:



* support for redis (or valkey/redict), postgres, sqlite, file-system, or in-memory storage

* zero dependencies (``redis-py`` required to use redis-like brokers,

  ``psycopg`` for postgres).

* `example code `_.

* `django `_ integration (native or via django.tasks)

  with admin integration for visibility and management.

* `documentation `_.



huey supports:



* multi-process, multi-thread or greenlet task execution models

* schedule tasks to execute at a given time, or after a given delay

* schedule recurring tasks, like a crontab

* automatically retry tasks that fail

* task prioritization

* task result storage

* task expiration

* task locking, rate-limits and timeouts

* task pipelines and chains

* groups (fan-out), chords (map / reduce)



.. image:: https://i.imgur.com/2EpRs.jpg



At a glance

-----------



.. code-block:: python



    from huey import RedisHuey, crontab



    # Or PostgresHuey, SqliteHuey, FileHuey, etc...

    huey = RedisHuey('my-app', host='redis.myapp.com')



    @huey.task()

    def add_numbers(a, b):

        return a + b



    @huey.task(retries=2, retry_delay=60)

    def flaky_task(url):

        # This task might fail, in which case it will be retried up to 2 times

        # with a delay of 60s between retries.

        return this_might_fail(url)



    @huey.periodic_task(crontab(minute='0', hour='3'))

    def nightly_backup():

        sync_all_data()



Calling a ``task``-decorated function will enqueue the function call for

execution by the consumer. A special result handle is returned immediately,

which can be used to fetch the result once the task is finished:



.. code-block:: pycon



    >>> from demo import add_numbers

    >>> res = add_numbers(1, 2)

    >>> res

    



    >>> res()

    3



Tasks can be scheduled to run in the future:



.. code-block:: pycon



    >>> res = add_numbers.schedule((2, 3), delay=10)  # Will be run in ~10s.

    >>> res(blocking=True)  # Will block until task finishes, in ~10s.

    5



For much more, check out the `guide `_

or take a look at the `example code `_.



Running the consumer

^^^^^^^^^^^^^^^^^^^^



Run the consumer with four worker processes:



.. code-block:: console



    $ huey_consumer my_app.huey -k process -w 4



To run the consumer with a single worker thread (default):



.. code-block:: console



    $ huey_consumer my_app.huey



If your work-loads are mostly IO-bound, you can run the consumer with threads

or greenlets instead. Because greenlets are so lightweight, you can run quite a

few of them efficiently:



.. code-block:: console



    $ huey_consumer my_app.huey -k greenlet -w 32



Storage

-------



Huey's design and feature-set were informed by the capabilities of the

`Redis `_ database. Redis is a fantastic fit for a

lightweight task queueing library like Huey: it's self-contained, versatile,

and can be a multi-purpose solution for other web-application tasks like

caching, event publishing, analytics, rate-limiting, and more.



Although Huey was designed with Redis in mind, the storage system implements a

simple API and many other tools could be used instead of Redis if that's your

preference.



Huey comes with builtin support for Redis, Postgres, Sqlite, File-system, and

in-memory storage.



Frameworks

----------



Huey provides Django integration either natively or via ``django.tasks``. Huey

also provides an optional admin integration for Django:



.. image:: https://huey.readthedocs.io/en/latest/_images/django-admin.png



Huey also provides `flask-peewee `_ admin

integration based on the same underlying stat-tracking system:



.. image:: https://huey.readthedocs.io/en/latest/_images/flask-admin-panel.png



Other frameworks can use the `stats `_

extension to collect and display this information.



Documentation

----------------



`See Huey documentation `_.



Project page

---------------



`See source code and issue tracker on Github `_.



Huey is named in honor of my cat:



.. image:: https://m.charlesleifer.com/t/800x-/blog/photos/p1473037658.76.jpg?key=mD9_qMaKBAuGPi95KzXYqg