{"id":18557724,"url":"https://github.com/aosingh/celery_docker_sample","last_synced_at":"2026-04-16T12:40:22.442Z","repository":{"id":45123482,"uuid":"178229415","full_name":"aosingh/celery_docker_sample","owner":"aosingh","description":"Sample celery project setup using docker. Redis is used as celery broker and Postgres as celery backend","archived":false,"fork":false,"pushed_at":"2022-12-08T04:55:04.000Z","size":7,"stargazers_count":1,"open_issues_count":3,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-17T11:44:01.318Z","etag":null,"topics":["async","celery","distributed","docker","postgres","redis","task-queue"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aosingh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-28T15:15:33.000Z","updated_at":"2022-03-04T10:03:37.000Z","dependencies_parsed_at":"2023-01-24T13:01:03.360Z","dependency_job_id":null,"html_url":"https://github.com/aosingh/celery_docker_sample","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aosingh%2Fcelery_docker_sample","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aosingh%2Fcelery_docker_sample/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aosingh%2Fcelery_docker_sample/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aosingh%2Fcelery_docker_sample/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aosingh","download_url":"https://codeload.github.com/aosingh/celery_docker_sample/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254355411,"owners_count":22057356,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","celery","distributed","docker","postgres","redis","task-queue"],"created_at":"2024-11-06T21:37:50.438Z","updated_at":"2026-04-16T12:40:17.373Z","avatar_url":"https://github.com/aosingh.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"This is a sample project-structure which you can follow if you plan to \nuse Celery as an Asynchronous distributed Task Queue. \n\n\n## Pre-requisites\n\nPlease install the following for project setup\n\n1. docker\n2. docker-compose\n3. python3\n4. virtualenv\n\n## Breakdown of services.\n\n### Task\nA celery task is any callable. You can define a celery task using `@task` decorator.\nFor this sample project, please refer the `tasks/__init__py` file for an example of a task.\nTasks should never be executed in a blocking manner. They are queued and then executed by a celery\nworker instance. \n\nIf your task is I/O bound then you can add a timeout period to avoid permanently blocking the worker executing the task. \n\nIf your task is CPU intensive then adding more number of workers will help improve the throughput.\nIn the `docker-compose.yaml` we schedule a task using the following service.\n\n```\ntask_scheduler:\n    image: celery_poc:test\n    command: python scheduler.py\n    env_file: env.sh\n    depends_on:\n      - celery_broker\n      - celery_backend\n```\n\n### Worker\nA worker is an instance of your application which has access to the resources to execute the task.\nWhen you start a celery worker by default the concurrency factor is number of cores on your system.\nYou can limit the concurrency factor using the `-c` option. \nIn the `docker-compose.yaml` we start a worker instance using the following command. \n\n```\nworker:\n    image: celery_poc:test\n    command: celery worker -A tasks -Q url -l info -Ofair\n    env_file: env.sh\n    depends_on:\n      - celery_backend\n      - celery_broker\n```\n\n\n### Message Broker\nTasks are queued and an instance of a celery worker monitors the queue and executes the tasks in that queue. \nThis queue is called the Message Broker. We use `redis` as a message queue which will be in-memory. \nHowever, with docker volumes the broker state will be persisted in between restarts.\n\n```\ncelery_broker:\n    image: redis:3-alpine\n    command: redis-server --appendonly yes\n    ports:\n      - \"6379:6379\"\n    volumes:\n      - \"celery_broker_data:/data\"\n```\n\n\n### Tasks Result Backend\nTasks results, stats and tracebacks are saved in the result backend. Celery gives multiple options. \nWe use `postgres`\n\n```\n  celery_backend:\n    image: postgres:latest\n    ports:\n      - \"5432:5432\"\n    environment:\n      POSTGRES_USER: docker\n      POSTGRES_PASSWORD: docker\n    volumes:\n      - \"celery_backend_data:/var/lib/postgresql/data\"\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faosingh%2Fcelery_docker_sample","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faosingh%2Fcelery_docker_sample","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faosingh%2Fcelery_docker_sample/lists"}