https://github.com/flix-tech/redis-tq

redis-based task queue written in Python
https://github.com/flix-tech/redis-tq

python redis task-queue

Last synced: 26 days ago
JSON representation

redis-based task queue written in Python

• Host: GitHub
• URL: https://github.com/flix-tech/redis-tq
• Owner: flix-tech
• License: mit
• Created: 2021-04-09T10:20:00.000Z (almost 4 years ago)
• Default Branch: master
• Last Pushed: 2023-01-24T17:02:50.000Z (almost 2 years ago)
• Last Synced: 2024-11-28T11:07:23.515Z (about 2 months ago)
• Topics: python, redis, task-queue
• Language: Python
• Homepage:
• Size: 63.5 KB
• Stars: 7
• Watchers: 3
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# redis-tq

redis-tq is a [Redis-based][redis] multi-producer, multi-consumer Queue.

Allows for sharing data between multiple processes or hosts.

Tasks support a "lease time". After that time other workers may consider this

client to have crashed or stalled and pick up the item instead. The number of

retries can be configured as well.

Based on [this example][source] but with many improvements added.

[source]: http://peter-hoffmann.com/2012/python-simple-queue-redis-queue.html

[redis]: https://redis.io/

## Installing

redis-tq is available on [PyPI][] so you can simply install via:

```sh

$ pip install redis-tq

```

[PyPI]: https://pypi.org/project/redis-tq/

## How to use

On the producing side, populate the queue with tasks and a respective lease

timeout:

```python

from redistq import TaskQueue

tq = TaskQueue('localhost', 'myqueue')

for i in range(10):

    tq.add(some task, lease_timeout, ttl=3)

```

On the consuming side:

```python

from redistq import TaskQueue

tq = TaskQueue('localhost', 'myqueue')

while True:

    task, task_id = tq.get()

    if task is not None:

        # do something with task and mark it as complete afterwards

        tq.complete(task_id)

    if tq.is_empty():

        break

    # tq.get is non-blocking, so you may want to sleep a

    # bit before the next iteration

    time.sleep(1)

```

If the consumer crashes (i.e. the task is not marked as completed after

`lease_timeout` seconds), the task will be put back into the task queue. This

rescheduling will happen at most `ttl` times and then the task will be

dropped. A callback can be provided if you want to monitor such cases.

## Running the tests

The tests will check the presence of a Redis instance on localhost, you can

use

    docker run --rm -d -p 6379:6379 redis

to get one. Then use `make test`, it will take care of creating an appropriate

virtualenv and use it to run the tests.