{"id":18995811,"url":"https://github.com/klippa-app/keda-celery-scaler","last_synced_at":"2025-04-22T13:48:36.104Z","repository":{"id":65183995,"uuid":"528830827","full_name":"klippa-app/keda-celery-scaler","owner":"klippa-app","description":"Autoscale your Celery workers based on your actual load with KEDA","archived":false,"fork":false,"pushed_at":"2024-02-20T10:46:36.000Z","size":124,"stargazers_count":9,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-29T15:41:42.773Z","etag":null,"topics":["autoscaling","celery","docker","keda","kubernetes","redis"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/klippa-app.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-25T12:00:15.000Z","updated_at":"2025-03-25T12:13:15.000Z","dependencies_parsed_at":"2024-06-21T12:06:46.639Z","dependency_job_id":null,"html_url":"https://github.com/klippa-app/keda-celery-scaler","commit_stats":{"total_commits":40,"total_committers":3,"mean_commits":"13.333333333333334","dds":0.07499999999999996,"last_synced_commit":"02b8986ff073161b7862c163d1e5246dcb72f3aa"},"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/klippa-app%2Fkeda-celery-scaler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/klippa-app%2Fkeda-celery-scaler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/klippa-app%2Fkeda-celery-scaler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/klippa-app%2Fkeda-celery-scaler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/klippa-app","download_url":"https://codeload.github.com/klippa-app/keda-celery-scaler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249268562,"owners_count":21240945,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autoscaling","celery","docker","keda","kubernetes","redis"],"created_at":"2024-11-08T17:32:54.727Z","updated_at":"2025-04-16T19:32:24.811Z","avatar_url":"https://github.com/klippa-app.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# KEDA Celery Scaler\n\n[![Build Status][build-status]][build-url]\n[![Docker](https://img.shields.io/docker/cloud/build/jerbob92/keda-celery-scaler?label=Docker\u0026style=flat)](https://hub.docker.com/r/jerbob92/keda-celery-scaler/builds)\n\n[build-url]:https://github.com/klippa-app/keda-celery-scaler/actions\n\n[build-status]:https://github.com/klippa-app/keda-celery-scaler/workflows/go/badge.svg\n\nAn [External Scaler](https://keda.sh/docs/2.9/concepts/external-scalers/) for KEDA.\n\n**Only supports Celery Redis Broker for now, PRs for other brokers that have heartbeats are welcome!**\n\n## Why?\n\nKEDA allows for autoscaling based on the length of queues, but it's application it not practical for time critical\ntasks. The drawback of queue length autoscaling is that scaling will only occur once tasks are queued. These queued\ntasks have a longer time to completion than desired.\n\nWhat you want is scaling based on the load `(active tasks + queue length) / available workers` so that you're able to\nscale _before_ a queue starts building up. For example, you could start to scale up when the load is at 50%. This way\nyou're able to handle peak load way better than with a queue length based scaler.\n\n## Implementation\n\nSince Celery is quite unstable in a production situation where you have a lot of workers that scale up and down due to\nthe way Celery workers communicate (we would advise everyone to set the Env\nvariables `CELERY_ENABLE_REMOTE_CONTROL=False` and `CELERY_SEND_EVENTS=False` to prevent the pidbox madness), we had to\nfigure out a way to get the current load of a queue.\n\nLuckily, Celery with a Redis broker sends heartbeats on a Pub/Sub channel called `/{db}.celeryev/worker.heartbeat`,\nwhere `{db}` is the used Redis database (since Pub/Sub channels are not scoped to a database).\n\nThis heartbeat contains the hostname of the worker and the current active tasks. It does not include the queue and\nconcurrency information of a worker. See the section \"Determining the queue(s) and concurrency of a worker\" for\ninformation on how to pass in those details.\n\n## Determining the queue(s) and concurrency of a worker\n\n### Automatically\n\nIt's possible to use a Celery hook called `@celeryd_init.connect` to insert the information that we need into the\nheartbeat.\n\nThe code that is needed for this is:\n\n```python\nimport celery\nfrom celery.signals import celeryd_init\nfrom celery.worker.state import SOFTWARE_INFO\n\n@celeryd_init.connect\ndef on_celeryd_init(options: Dict[str, Any], **_):\n    \"\"\"\n    Updates the celery SOFTWARE_INFO dict to contain worker specific settings\n    that will be published with each celery heartbeat. These settings can then\n    be used for load-based autoscaling.\n\n    The SOFTWARE_INFO key is defined in celery/worker/state.py, and is inserted\n    into heartbeats in celery/worker/heartbeat.py in the _send() method.\n\n    Args:\n        options: the options using which the worker is initialized.\n        **_: Other keyword parameters are given by the signal, but these\n             are ignored.\n    \"\"\"\n    queues = options[\"queues\"]\n\n    if queues is None or len(queues) == 0:\n        queues = [\"celery\"]\n\n    SOFTWARE_INFO[\"x_worker_info\"] = {\"queues\": queues, \"concurrency\": options[\"concurrency\"]}\n```\n\nThe scaler will automatically pick up the information from the heartbeat.\n\n### Manual mapping\n\nIf you do not have the ability to change your Celery implementation, or you do not want it (for now), you can also use\nour manual mapping.\n\nThis works with the environment variable `KCS_WORKER_QUEUE_MAP`. The format\nis `{identifier1}:{queue1},{queue2}:{concurrency1};{identifier2}:{queue3},{queue4}:{concurrency2}`.\n\nIf you leave out the concurrency part, it will default to 1.\n\nThis means that if `{identifier1}` is found in the worker hostname, it will be assigned the queues `{queue1}`\nand `{queue2}`. If `{identifier2}` is found in the worker hostname, it will be assigned the queues `{queue3}`\nand `{queue4}`.\n\nAn example mapping could look like `email-worker:emails:1;notifications-worker:notifications:2`.\n\n## Configuration Env variables\n\n| Name                        | Description                                                                                                                                                                                                                                                                                     | Default                      |\n|-----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------|\n| KCS_LOG_LEVEL               | The log level to print (panic, fatal, error, warn(ing), info, debug, trace)                                                                                                                                                                                                                     | info                         |\n| KCS_WORKER_STALE_TIME       | When we should mark a worker as lost in seconds.                                                                                                                                                                                                                                                | 10                           |\n| KCS_WORKER_CLEANUP_INTERVAL | How often to clean up lost workers in seconds.                                                                                                                                                                                                                                                  | 5                            |\n| KCS_WORKER_QUEUE_MAP        | When your Celery apps don't send the queue in the heartbeat, you can use this to map the workers by hostname. The format is {identifier1}:{queue1},{queue2}:{concurrency1};{identifier2}:{queue3},{queue4}:{concurrency2}. When the identifier is found in the worker hostname, it will map it. |                              |\n| KCS_REDIS_TYPE              | What type of redis to use, standalone or sentinel.                                                                                                                                                                                                                                              | standalone                   |\n| KCS_REDIS_SERVER            | host:port list, seperated by a comma (redis in standalone uses only the first in case of multiple given)                                                                                                                                                                                        | localhost:6379               |\n| KCS_REDIS_DB                | The DB to use for the connection.                                                                                                                                                                                                                                                               | 0                            |\n| KCS_REDIS_USERNAME          | The username to use for the connection.                                                                                                                                                                                                                                                         |                              |\n| KCS_REDIS_PASSWORD          | The password to use for the connection.                                                                                                                                                                                                                                                         |                              |\n| KCS_REDIS_TIMEOUTS_DIAL     | The dial timeout in seconds to use for the connection.                                                                                                                                                                                                                                          | 5                            |\n| KCS_REDIS_TIMEOUTS_READ     | The read timeout in seconds to use for the connection.                                                                                                                                                                                                                                          | 3                            |\n| KCS_REDIS_TIMEOUTS_WRITE    | The write timeout in seconds to use for the connection.                                                                                                                                                                                                                                         | 3                            |\n| KCS_REDIS_CONNECTIONS_MIN   | The minimum amount of connections to keep around in the pool.                                                                                                                                                                                                                                   | 0                            |\n| KCS_REDIS_CONNECTIONS_MAX   | The maximum amount of connections to have in the pool.                                                                                                                                                                                                                                          | 10 * CPU COUNT               |\n| KCS_REDIS_TLS_ENABLED       | Whether to use TLS in the connection.                                                                                                                                                                                                                                                           | False                        |\n| KCS_REDIS_TLS_SERVERNAME    | The hostname to validate the certificate with.                                                                                                                                                                                                                                                  | Parsed from KCS_REDIS_SERVER |\n| KCS_REDIS_MASTER            | The Sentinel master name in case of Sentinel.                                                                                                                                                                                                                                                   |                              |\n| KCS_REDIS_SENTINELUSERNAME  | The username for the sentinel server.                                                                                                                                                                                                                                                           |                              |\n| KCS_REDIS_SENTINELPASSWORD  | The password for the sentinel server.                                                                                                                                                                                                                                                           |                              |\n\n## Docker\n\nThis Docker image is stored on the Docker registry\nat [jerbob92/keda-celery-scaler](https://hub.docker.com/r/jerbob92/keda-celery-scaler).\n\n## Kubernetes\n\nAn example deployment could look like this:\n\nAutoscaler K8S deployment:\n\n```yaml\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: keda-celery-autoscaler\n  labels:\n    component: keda-celery-autoscaler\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      component: keda-celery-autoscaler\n  template:\n    metadata:\n      labels:\n        component: keda-celery-autoscaler\n    spec:\n      containers:\n        - name: keda-celery-autoscaler\n          image: \"jerbob92/keda-celery-scaler:latest\"\n          env:\n            - name: KCS_LOG_LEVEL\n              value: \"info\"\n            - name: KCS_WORKER_STALE_TIME\n              value: \"10\"\n            - name: KCS_WORKER_CLEANUP_INTERVAL\n              value: \"5\"\n            - name: KCS_WORKER_QUEUE_MAP\n              value: \"email-worker:emails;notifications-worker:notifications\"\n            - name: KCS_REDIS_TYPE\n              value: \"standalone\"\n            - name: KCS_REDIS_SERVER\n              value: \"redis-service.default.svc.cluster.local\"\n          ports:\n            - containerPort: 6000\n              name: keda-celery-autoscaler\n```\n\nAutoscaler K8S service:\n\n```yaml\napiVersion: v1\nkind: Service\nmetadata:\n  name: keda-celery-autoscaler\nspec:\n  ports:\n    - port: 6000\n      protocol: TCP\n      name: keda-celery-autoscaler\n  selector:\n    component: keda-celery-autoscaler\n```\n\nKEDA ScaledObjects:\n\n```yaml\napiVersion: keda.sh/v1alpha1\nkind: ScaledObject\nmetadata:\n  name: email-worker-scaler\nspec:\n  scaleTargetRef:\n    name: email-worker\n  pollingInterval: 5\n  cooldownPeriod: 300\n  minReplicaCount: 5\n  maxReplicaCount: 15\n\n  triggers:\n    - type: external\n      metadata:\n        scalerAddress: \"keda-celery-autoscaler.default.svc.cluster.local:6000\"\n        queue: \"emails\"\n        scaleLoadValue: \"70\" # Scale at a load of 70%.\n      metricType: Value\n```\n\n```yaml\napiVersion: keda.sh/v1alpha1\nkind: ScaledObject\nmetadata:\n  name: notifications-worker-scaler\nspec:\n  scaleTargetRef:\n    name: notifications-worker\n  pollingInterval: 5\n  cooldownPeriod: 300\n  minReplicaCount: 5\n  maxReplicaCount: 15\n\n  triggers:\n    - type: external\n      metadata:\n        scalerAddress: \"keda-celery-autoscaler.default.svc.cluster.local:6000\"\n        queue: \"notifications\"\n        scaleLoadValue: \"70\" # Scale at a load of 70%.\n      metricType: Value\n```\n\nThis is all just an example, you will need to change this to your own setup.\n\n## About Klippa\n\n[Klippa](https://www.klippa.com/en) is a scale-up\nfrom [Groningen, The Netherlands](https://goo.gl/maps/CcCGaPTBz3u8noSd6) and was founded in 2015 by six Dutch IT\nspecialists with the goal to digitize paper processes with modern technologies.\n\nWe help clients enhance the effectiveness of their organization by using machine learning and OCR. Since 2015 more than\na 1000 happy clients have been served with a variety of the software solutions that Klippa offers. Our passion is to\nhelp our clients to digitize paper processes by using smart apps, accounts payable software and data extraction by using\nOCR.\n\n## License\n\nThe MIT License (MIT)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fklippa-app%2Fkeda-celery-scaler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fklippa-app%2Fkeda-celery-scaler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fklippa-app%2Fkeda-celery-scaler/lists"}