https://github.com/aosingh/celery_docker_sample
Sample celery project setup using docker. Redis is used as celery broker and Postgres as celery backend
https://github.com/aosingh/celery_docker_sample
async celery distributed docker postgres redis task-queue
Last synced: about 2 months ago
JSON representation
Sample celery project setup using docker. Redis is used as celery broker and Postgres as celery backend
- Host: GitHub
- URL: https://github.com/aosingh/celery_docker_sample
- Owner: aosingh
- Created: 2019-03-28T15:15:33.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2022-12-08T04:55:04.000Z (over 3 years ago)
- Last Synced: 2025-02-17T11:44:01.318Z (over 1 year ago)
- Topics: async, celery, distributed, docker, postgres, redis, task-queue
- Language: Python
- Homepage:
- Size: 6.84 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
This is a sample project-structure which you can follow if you plan to
use Celery as an Asynchronous distributed Task Queue.
## Pre-requisites
Please install the following for project setup
1. docker
2. docker-compose
3. python3
4. virtualenv
## Breakdown of services.
### Task
A celery task is any callable. You can define a celery task using `@task` decorator.
For this sample project, please refer the `tasks/__init__py` file for an example of a task.
Tasks should never be executed in a blocking manner. They are queued and then executed by a celery
worker instance.
If your task is I/O bound then you can add a timeout period to avoid permanently blocking the worker executing the task.
If your task is CPU intensive then adding more number of workers will help improve the throughput.
In the `docker-compose.yaml` we schedule a task using the following service.
```
task_scheduler:
image: celery_poc:test
command: python scheduler.py
env_file: env.sh
depends_on:
- celery_broker
- celery_backend
```
### Worker
A worker is an instance of your application which has access to the resources to execute the task.
When you start a celery worker by default the concurrency factor is number of cores on your system.
You can limit the concurrency factor using the `-c` option.
In the `docker-compose.yaml` we start a worker instance using the following command.
```
worker:
image: celery_poc:test
command: celery worker -A tasks -Q url -l info -Ofair
env_file: env.sh
depends_on:
- celery_backend
- celery_broker
```
### Message Broker
Tasks are queued and an instance of a celery worker monitors the queue and executes the tasks in that queue.
This queue is called the Message Broker. We use `redis` as a message queue which will be in-memory.
However, with docker volumes the broker state will be persisted in between restarts.
```
celery_broker:
image: redis:3-alpine
command: redis-server --appendonly yes
ports:
- "6379:6379"
volumes:
- "celery_broker_data:/data"
```
### Tasks Result Backend
Tasks results, stats and tracebacks are saved in the result backend. Celery gives multiple options.
We use `postgres`
```
celery_backend:
image: postgres:latest
ports:
- "5432:5432"
environment:
POSTGRES_USER: docker
POSTGRES_PASSWORD: docker
volumes:
- "celery_backend_data:/var/lib/postgresql/data"
```