https://github.com/eduzen/task_executor
Task runner
https://github.com/eduzen/task_executor
celery docker docker-compose python3 rabbitmq redis
Last synced: 3 months ago
JSON representation
Task runner
- Host: GitHub
- URL: https://github.com/eduzen/task_executor
- Owner: eduzen
- Created: 2018-09-13T18:07:56.000Z (almost 8 years ago)
- Default Branch: master
- Last Pushed: 2022-07-15T18:40:58.000Z (almost 4 years ago)
- Last Synced: 2025-03-06T04:17:41.698Z (over 1 year ago)
- Topics: celery, docker, docker-compose, python3, rabbitmq, redis
- Language: Python
- Homepage:
- Size: 15.6 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Task executor [](https://travis-ci.org/eduzen/task_executor) [](https://codecov.io/gh/eduzen/task_executor)
For this exercise, we chose [Celery](http://www.celeryproject.org/) that is an asynchronous
task queue/job queue based on distributed message passing. Tasks can execute asynchronously
(in the background) or synchronously (wait until ready). Celery requires a message transport
to send and receive message. We chose [Rabbitmq](https://www.rabbitmq.com/) because it works well with celery.
Other broker could be [Redis](https://redis.io/), but for this exercise we used it as a memcachedb, a
kind of persistent key-value store, for managing locks through all the tasks. We can use this
distributed lock to have our tasks try to acquire a non-blocking lock, and exit if the lock isn’t acquired.
## Installation
This project runs with `docker` (you can use traditional `virtualenv` but it's prepared out of the box for `docker`).
We choose `python 3.6.4` and not `python 3.7` because `Celery` doesn't support it yet.
Also to manage docker, we are using [docker-compose](https://docs.docker.com/compose/).
## Usage
If you already have `docker` and `docker-compose`, just run:
```bash
make start
# only test and flake8
make test
```
This command will download the images and build them in a container.
## Workers
Right now the celery worker is configure to execute __1) no more than 3 tasks in parallel__.
We can change this changing the number of concurrency of the worker:
```bash
celery worker --app=task_manager.celeryapp:app --concurrency=3 --loglevel=info
```
Other option is to run several workers:
```bash
docker-compose scale worker=5
```
## Tasks
Celery tasks have a custom decorator in order to achieve: __2) each target (dave for example) can only execute one task at once__.
To do this, we choose Redis instead of django cache though this last option is recommended [here](http://docs.celeryproject.org/en/latest/tutorials/task-cookbook.html#ensuring-a-task-is-only-executed-one-at-a-time)
because if memcached (or some other non- persistent cache) is used and (1) the cache daemon crashes or
(2) the cache key is culled before the appropriate expiration time / lock release,
then you have a race condition where two or more tasks could simultaneously acquire the task lock. We follow this recommendation:
[this](http://loose-bits.com/2010/10/distributed-task-locking-in-celery.html)) but with some changes.
So, in case that the target is busy (locked), we will retry the task with a custom delay. Also the task has a rate limit configured.