{"id":26287326,"url":"https://github.com/bothub-it/celery-worker-on-demand","last_synced_at":"2025-07-19T04:33:22.712Z","repository":{"id":126912072,"uuid":"156561010","full_name":"bothub-it/celery-worker-on-demand","owner":"bothub-it","description":"Up and down Celery workers on demand.","archived":false,"fork":false,"pushed_at":"2020-10-02T14:57:30.000Z","size":49,"stargazers_count":8,"open_issues_count":1,"forks_count":1,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-05-07T18:42:41.500Z","etag":null,"topics":["celery","docker","hacktoberfest","hacktoberfest-accepted","kubernetes","on-demand"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bothub-it.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-11-07T14:45:08.000Z","updated_at":"2024-01-19T10:25:51.000Z","dependencies_parsed_at":"2023-06-18T21:33:51.661Z","dependency_job_id":null,"html_url":"https://github.com/bothub-it/celery-worker-on-demand","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/bothub-it/celery-worker-on-demand","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bothub-it%2Fcelery-worker-on-demand","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bothub-it%2Fcelery-worker-on-demand/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bothub-it%2Fcelery-worker-on-demand/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bothub-it%2Fcelery-worker-on-demand/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bothub-it","download_url":"https://codeload.github.com/bothub-it/celery-worker-on-demand/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bothub-it%2Fcelery-worker-on-demand/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265888902,"owners_count":23844529,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["celery","docker","hacktoberfest","hacktoberfest-accepted","kubernetes","on-demand"],"created_at":"2025-03-14T21:19:55.034Z","updated_at":"2025-07-19T04:33:22.703Z","avatar_url":"https://github.com/bothub-it.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Celery Worker On Demand\n\nUp and down Celery workers on demand.\n\n## Examples\n\nCheck the [example](https://github.com/bothub-it/celery-worker-on-demand/tree/master/example_docker) using a Docker container to up and down Celery workers.\n\n## Usage\n\n### Dependencies\n\n- Python 3.6\n- Celery 4.3.0 or higher\n- Your [Celery Application](http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html#application)\n- celery_worker_on_demand Python package\n  ```bash\n  pip install celery-worker-on-demand\n  ```\n\n### Create a custom Class extended from CeleryWorkerOnDemand\n\nCreate a custom Class extended from CeleryWorkerOnDemand to customize Agent, UpWorker and DownWorker classes.\n\n```python\nfrom celery_worker_on_demand import CeleryWorkerOnDemand\nfrom celery_worker_on_demand import Agent\nfrom celery_worker_on_demand import UpWorker\nfrom celery_worker_on_demand import DownWorker\n\n\nclass MyDemand(CeleryWorkerOnDemand):\n    # Customize Agent methods to apply new rules to up (flag_up) and down (flag_down) Celery workers\n    Agent = Agent\n    # Overide UpWorker run method with your script to up new Celery worker\n    UpWorker = UpWorker\n    # Overide DownWorker run method with your script to up new Celery worker\n    DownWorker = DownWorker\n```\n\n### Implement your up and down Celery worker script\n\nYou need overide UpWorker and DownWorker run method with your custom script to up and down Celery worker.\n\n```python\nfrom celery_worker_on_demand import CeleryWorkerOnDemand\nfrom celery_worker_on_demand import UpWorker\nfrom celery_worker_on_demand import DownWorker\n\n\nclass MyUpWorker(UpWorkerU):\n    def run(self):\n        # Use self.cwod to retrieve CeleryWorkerOnDemand instance;\n        # Use self.queue to retrieve QueueStatus instance with all queue information;\n        ... Your script here\n\n\nclass MyDownWorker(DownWorker):\n    def run(self):\n        # Use self.cwod to retrieve CeleryWorkerOnDemand instance;\n        # Use self.queue to retrieve QueueStatus instance with all queue information;\n        ... Your script here\n\n\nclass MyDemand(CeleryWorkerOnDemand):\n    UpWorker = MyUpWorker\n    DownWorker = MyDownWorker\n```\n\n### Create a runnable Python file\n\nCreate a runnable Python file like below, to run like a service.\n\n```python\n#!/bin/env/python\nfrom .cwod import MyDemand\nfrom .celery_app import celery_app\n\n\n# Run Celery Worker On Demand service\nMyDemand(celery_app).run()\n```\n\n## Customize up and down rules\n\nTo customize up and down flag you just need overide Agent methods `flag_up` and `flag_down`.\n\n```python\nfrom celery_worker_on_demand import CeleryWorkerOnDemand\nfrom celery_worker_on_demand import Agent\n\n\nclass MyAgent(Agent):\n    # When returns True, the UpWorker.run() method is executed\n    def flag_up(self, queue):\n        # queue is a QueueStatus instance\n        return super().flag_up(queue)\n\n    # When returns True, the DownWorker.run() method is executed\n    def flag_down(self, queue):\n        # queue is a QueueStatus instance\n        return super().flag_down(queue)\n\n\nclass MyDemand(CeleryWorkerOnDemand):\n    Agent = MyAgent\n```\n\n## API\n\n### class CeleryWorkerOnDemand\n\n#### constructor\n\n- **celery_app** (arg): Your [Celery Application](http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html#application) instance.\n- **queue_updater_fill_rate** (kwarg): Times to update per second queue status. Default is `2`.\n- **api_server_address** (kwarg): Tuple with address and port to serve API. Default is `('', 8000)`.\n- **api_basic_authorization** (kwarg): Add basic authorization in API. Set authorization header value to enable. Use [this generator](https://www.blitter.se/utils/basic-authentication-header-generator/).\n\n#### attributes\n\n- **celery_app**: Your Celery Application instance\n- **queue_updater_fill_rate**: Times to update per second queue status.\n- **api_server_address**: Tuple with address and port to serve API.\n- **api_basic_authorization**: Authorization header value.\n- **queues**: Queue dictionary - key is the queue name and value is a QueueStatus instance.\n- **queue_updater**: Thread instance of QueueUpdater\n- **worker_monitor**: Thread instance of WorkerMonitor\n- **api_server**: Thread instance of APIServer\n- **agent**: Thread instance of Agent\n\n#### properties\n\n- **connection**: Cached property, returns main broker connection without heartbeat.\n- **channel**: Cached property, returns default channel.\n\n#### .add_queue(queue_name) method\n\n- **queue_name** (arg): Queue name.\n\nAdd new queue to observer.\n\n#### .run() method\n\nStart the Celery Worker On Demand service.\n\n#### .serializer() method\n\nReturns a JSON serializable dictionary that represents the current state of instance.\n\n### class Agent\n\n#### constructor\n\n- **cwod** (arg): CeleryWorkerOnDemand instance.\n\n#### .run() method\n\nWatch current state of application to up and down workers.\n\n#### flag_up(queue) method\n\n- **queue** (arg): QueueStatus instance.\n\nWhen returns True, the UpWorker.run() method is executed.\n\n#### flag_down(queue) method\n\n- **queue** (arg): QueueStatus instance.\n\nWhen returns True, the DownWorker.run() method is executed\n\n### class UpWorker\n\n#### constructor\n\n- **agent** (arg): Agent instance.\n- **queue** (arg): QueueStatus instance.\n\n#### .run() method\n\nPython script to up new worker to queue.\n\n### class DownWorker\n\n#### constructor\n\n- **agent** (arg): Agent instance.\n- **queue** (arg): QueueStatus instance.\n\n#### .run() method\n\nPython script to down queue's worker(s).\n\n### class QueueStatus\n\n#### constructor\n\n- **name** (arg): Queue name.\n- **size** (kwarg): Current queue size, default is `0`.\n- **workers** (kwarg): List of WorkerStatus instance, default is a empty list.\n\n#### attributes\n\n- **name**: Queue name.\n- **size**: Current queue size.\n- **workers**: List of WorkerStatus instance.\n\n#### properties\n\n- **has_worker**: Returns True when queue has worker.\n- **many_workers**: Returns how many workers the queue have.\n\n#### .serializer() method\n\nReturns a JSON serializable dictionary that represents the current state of instance.\n\n### class WorkerStatus\n\n#### .get(hostname, *, **) class method\n\n- **hostname** (arg): Worker hostname.\n- **\\***: args passed to WorkerStatus constructor.\n- **\\*\\***: kwargs passed to WorkerStatus constructor.\n\nReturns a cached WorkerStatus instance relative to hostname.\n\n#### constructor\n\n- **hostname** (arg): Worker hostname.\n- **last_heartbeat_at** (kwarg): Timestamp of last hearbeat. Default is `None`.\n- **last_task_received_at** (kwarg): Timestamp of last task received. Default is `None`.\n- **last_task_started_at** (kwarg): Timestamp of last task started. Default is `None`.\n- **last_task_succeeded_at** (kwarg): Timestamp of last task succeeded. Default is `None`.\n\n#### attributes\n\n- **hostname** (arg): Worker hostname.\n- **last_heartbeat_at** (kwarg): Timestamp of last hearbeat.\n- **last_task_received_at** (kwarg): Timestamp of last task received.\n- **last_task_started_at** (kwarg): Timestamp of last task started.\n- **last_task_succeeded_at** (kwarg): Timestamp of last task succeeded.\n\n#### .serializer() method\n\nReturns a JSON serializable dictionary that represents the current state of instance.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbothub-it%2Fcelery-worker-on-demand","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbothub-it%2Fcelery-worker-on-demand","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbothub-it%2Fcelery-worker-on-demand/lists"}