Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/coleifer/huey

a little task queue for python
https://github.com/coleifer/huey

dank python queue redis task-queue

Last synced: 5 days ago
JSON representation

a little task queue for python

Awesome Lists containing this project

README 

        
.. image:: http://media.charlesleifer.com/blog/photos/huey2-logo.png



*a lightweight alternative*.



huey is:



* a task queue

* written in python

* clean and simple API

* redis, sqlite, file-system, or in-memory storage

* `example code `_.

* `read the documentation `_.



huey supports:



* multi-process, multi-thread or greenlet task execution models

* schedule tasks to execute at a given time, or after a given delay

* schedule recurring tasks, like a crontab

* automatically retry tasks that fail

* task prioritization

* task result storage

* task expiration

* task locking

* task pipelines and chains



.. image:: http://i.imgur.com/2EpRs.jpg



At a glance

-----------



.. code-block:: python



    from huey import RedisHuey, crontab



    huey = RedisHuey('my-app', host='redis.myapp.com')



    @huey.task()

    def add_numbers(a, b):

        return a + b



    @huey.task(retries=2, retry_delay=60)

    def flaky_task(url):

        # This task might fail, in which case it will be retried up to 2 times

        # with a delay of 60s between retries.

        return this_might_fail(url)



    @huey.periodic_task(crontab(minute='0', hour='3'))

    def nightly_backup():

        sync_all_data()



Calling a ``task``-decorated function will enqueue the function call for

execution by the consumer. A special result handle is returned immediately,

which can be used to fetch the result once the task is finished:



.. code-block:: pycon



    >>> from demo import add_numbers

    >>> res = add_numbers(1, 2)

    >>> res

    



    >>> res()

    3



Tasks can be scheduled to run in the future:



.. code-block:: pycon



    >>> res = add_numbers.schedule((2, 3), delay=10)  # Will be run in ~10s.

    >>> res(blocking=True)  # Will block until task finishes, in ~10s.

    5



For much more, check out the `guide `_

or take a look at the `example code `_.



Running the consumer

^^^^^^^^^^^^^^^^^^^^



Run the consumer with four worker processes:



.. code-block:: console



    $ huey_consumer.py my_app.huey -k process -w 4



To run the consumer with a single worker thread (default):



.. code-block:: console



    $ huey_consumer.py my_app.huey



If your work-loads are mostly IO-bound, you can run the consumer with threads

or greenlets instead. Because greenlets are so lightweight, you can run quite a

few of them efficiently:



.. code-block:: console



    $ huey_consumer.py my_app.huey -k greenlet -w 32



Storage

-------



Huey's design and feature-set were informed by the capabilities of the

`Redis `_ database. Redis is a fantastic fit for a

lightweight task queueing library like Huey: it's self-contained, versatile,

and can be a multi-purpose solution for other web-application tasks like

caching, event publishing, analytics, rate-limiting, and more.



Although Huey was designed with Redis in mind, the storage system implements a

simple API and many other tools could be used instead of Redis if that's your

preference.



Huey comes with builtin support for Redis, Sqlite and in-memory storage.



Documentation

----------------



`See Huey documentation `_.



Project page

---------------



`See source code and issue tracker on Github `_.



Huey is named in honor of my cat:



.. image:: http://m.charlesleifer.com/t/800x-/blog/photos/p1473037658.76.jpg?key=mD9_qMaKBAuGPi95KzXYqg