{"id":13416023,"url":"https://github.com/danihodovic/celery-exporter","last_synced_at":"2025-05-16T14:06:17.396Z","repository":{"id":37581017,"uuid":"332823353","full_name":"danihodovic/celery-exporter","owner":"danihodovic","description":"A Prometheus exporter for Celery metrics","archived":false,"fork":false,"pushed_at":"2024-05-29T02:30:55.000Z","size":1182,"stargazers_count":366,"open_issues_count":25,"forks_count":80,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-05-29T11:41:36.498Z","etag":null,"topics":["celery","django","exporter","metrics","prometheus","rabbitmq","redis"],"latest_commit_sha":null,"homepage":"","language":"Jsonnet","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/danihodovic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-25T17:14:09.000Z","updated_at":"2024-06-05T21:54:51.021Z","dependencies_parsed_at":"2024-03-01T17:44:20.758Z","dependency_job_id":"adae5ab8-7385-4177-9c79-6beec9cd4d5e","html_url":"https://github.com/danihodovic/celery-exporter","commit_stats":null,"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danihodovic%2Fcelery-exporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danihodovic%2Fcelery-exporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danihodovic%2Fcelery-exporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danihodovic%2Fcelery-exporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/danihodovic","download_url":"https://codeload.github.com/danihodovic/celery-exporter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248590532,"owners_count":21129840,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["celery","django","exporter","metrics","prometheus","rabbitmq","redis"],"created_at":"2024-07-30T21:00:53.723Z","updated_at":"2025-04-12T15:37:33.389Z","avatar_url":"https://github.com/danihodovic.png","language":"Jsonnet","funding_links":[],"categories":["Third-Party Packages","Administration and Monitoring","Python"],"sub_categories":["Task Queues"],"readme":"# celery-exporter ![Build Status](https://github.com/danihodovic/celery-exporter/actions/workflows/.github/workflows/ci.yml/badge.svg) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\n![celery-tasks-by-task](images/celery-tasks-by-task.png)\n\n##### Table of Contents\n\n* [Why another exporter?](#why-another-exporter)\n  * [Features](#features)\n* [Usage](#usage)\n  * [Enable events using the CLI](#enable-events-using-the-cli)\n  * [Running the exporter](#running-the-exporter)\n* [Metrics](#metrics)\n* [Development](#development)\n* [Contributors](#contributors)\n\n### Why another exporter?\n\nWhile I was adding Celery monitoring to a client site I realized that the\nexisting brokers either didn't work, exposed incorrect metric values or didn't\nexpose the metrics I needed. So I wrote this exporter which essentially wraps\nthe built-in Celery monitoring API and exposes all of the event metrics to\nPrometheus in real-time.\n\n## Features\n\n- Tested for both Redis and RabbitMQ\n- Uses the built in [real-time monitoring component in Celery](https://docs.celeryproject.org/en/latest/userguide/monitoring.html#real-time-processing) to expose Prometheus metrics\n- Tracks task status (task-started, task-succeeded, task-failed etc)\n- Tracks which workers are running and the number of active tasks\n- Follows the Prometheus exporter [best practises](https://prometheus.io/docs/instrumenting/writing_exporters/)\n- Deployed as a Docker image or Python single-file binary (via PyInstaller)\n- Exposes a health check endpoint at /health\n- Grafana dashboards provided by the Celery-mixin\n- Prometheus alerts provided by the Celery-mixin\n\n## Dashboards and alerts\n\nAlerting rules can be found [here](./celery-mixin/prometheus-alerts.yaml). By\ndefault we alert if:\n\n- A task failed in the last 10 minutes.\n- No Celery workers are online.\n\nTweak these to suit your use-case.\n\nThe Grafana dashboard (seen in the image above) is\n[here](https://grafana.com/grafana/dashboards/17508). You can import it\ndirectly into your Grafana instance.\n\nThere's another Grafana dashboards that shows an overview of Celery tasks. An image can be found in `./images/celery-tasks-overview.png`. It can also be found\n[here](https://grafana.com/grafana/dashboards/17509).\n\n## Usage\n\nCelery needs to be configured to send events to the broker which the exporter\nwill collect. You can either enable this via Celery configuration or via the\nCelery CLI.\n\n##### Enable events using the CLI\n\nTo enable events in the CLI run the below command. Note that by default it\ndoesn't send the `task-sent` event which needs to be [configured](https://docs.celeryproject.org/en/latest/userguide/configuration.html#std-setting-task_send_sent_event) in the\nconfiguration. The other events work out of the box.\n\n```sh\n$ celery -A \u003cmyproject\u003e control enable_events\n```\n\n**Enable events using the configuration:**\n\n```python\n# In celeryconfig.py\nworker_send_task_events = True\ntask_send_sent_event = True\n```\n\n**Configuration in Django:**\n```python\n# In settings.py\nCELERY_WORKER_SEND_TASK_EVENTS = True\nCELERY_TASK_SEND_SENT_EVENT = True\n```\n\n##### Running the exporter\n\nUsing Docker:\n\n```sh\ndocker run -p 9808:9808 danihodovic/celery-exporter --broker-url=redis://redis.service.consul/1\n```\n\nUsing the Python binary (for-non Docker environments):\n```sh\ncurl -L https://github.com/danihodovic/celery-exporter/releases/download/latest/celery-exporter -o ./celery-exporter\nchmod+x ./celery-exporter\n./celery-exporter --broker-url=redis://redis.service.consul/1\n```\n\n###### Kubernetes\n\nThere's a Helm in the directory `charts/celery-exporter` for deploying the Celery-exporter to Kubernetes using Helm.\n\n###### Environment variables\n\nAll arguments can be specified using environment variables with a `CE_` prefix:\n\n```sh\ndocker run -p 9808:9808 -e CE_BROKER_URL=redis://redis danihodovic/celery-exporter\n```\n\n###### Specifying optional broker transport options\n\nWhile the default options may be fine for most cases,\nthere may be a need to specify optional broker transport options. This can be done by specifying\none or more --broker-transport-option parameters as follows:\n\n```sh\ndocker run -p 9808:9808 danihodovic/celery-exporter --broker-url=redis://redis.service.consul/1 \\\n  --broker-transport-option global_keyprefix=danihodovic \\\n  --broker-transport-option visibility_timeout=7200\n```\n\nIn case of extended transport options, such as `sentinel_kwargs` you can pass JSON string:,\nfor example:\n\n```sh\ndocker run -p 9808:9808 danihodovic/celery-exporter --broker-url=sentinel://sentinel.service.consul/1 \\\n  --broker-transport-option master_name=my_master \\\n  --broker-transport-option sentinel_kwargs=\"{\\\"password\\\": \\\"sentinelpass\\\"}\"\n```\n\nThe list of available broker transport options can be found here:\nhttps://docs.celeryq.dev/projects/kombu/en/stable/reference/kombu.transport.redis.html\n\n###### Specifying an optional retry interval\n\nBy default, celery-exporter will raise an exception and exit if there\nare any errors communicating with the broker. If preferred, one can\nhave the celery-exporter retry connecting to the broker after a certain\nperiod of time in seconds via the `--retry-interval` parameter as follows:\n\n```sh\ndocker run -p 9808:9808 danihodovic/celery-exporter --broker-url=redis://redis.service.consul/1 \\\n  --retry-interval=5\n```\n\n##### Grafana Dashboards \u0026 Prometheus Alerts\n\nHead over to the [Celery-mixin in this subdirectory](https://github.com/danihodovic/celery-exporter/tree/master/celery-mixin) to generate rules and dashboards suited to your Prometheus setup.\n\n### Metrics\nName     | Description | Type\n---------|-------------|----\ncelery_task_sent_total | Sent when a task message is published. | Counter\ncelery_task_received_total | Sent when the worker receives a task. | Counter\ncelery_task_started_total | Sent just before the worker executes the task. | Counter\ncelery_task_succeeded_total | Sent if the task executed successfully. | Counter\ncelery_task_failed_total | Sent if the execution of the task failed. | Counter\ncelery_task_rejected_total | The task was rejected by the worker, possibly to be re-queued or moved to a dead letter queue. | Counter\ncelery_task_revoked_total | Sent if the task has been revoked. | Counter\ncelery_task_retried_total | Sent if the task failed, but will be retried in the future. | Counter\ncelery_worker_up | Indicates if a worker has recently sent a heartbeat. | Gauge\ncelery_worker_tasks_active | The number of tasks the worker is currently processing | Gauge\ncelery_task_runtime_bucket | Histogram of runtime measurements for each task | Histogram\ncelery_queue_length | The number of message in broker queue | Gauge\ncelery_active_consumer_count | The number of active consumer in broker queue **(Only work for [RabbitMQ and Qpid](https://qpid.apache.org/) broker, more details at [here](https://github.com/danihodovic/celery-exporter/pull/118#issuecomment-1169870481))** | Gauge\ncelery_active_worker_count | The number of active workers in broker queue | Gauge\ncelery_active_process_count | The number of active process in broker queue. Each worker may have more than one process. | Gauge\n\nUsed in production at [https://findwork.dev](https://findwork.dev) and [https://django.wtf](https://django.wtf).\n\n\n## Development\nPull requests are welcome here!\n\nTo start developing run commands below to prepare your environment after the `git clone` command:\n```shell\n# Install dependencies and pre-commit hooks\npoetry install\npre-commit install\n\n# Test everything works fine\npre-commit run --all-files\ndocker-compose up -d\npytest --broker=memory      --log-level=DEBUG\npytest --broker=redis       --log-level=DEBUG\npytest --broker=rabbitmq    --log-level=DEBUG\n```\n\n## Contributors\n\n\u003ca href=\"https://github.com/danihodovic/celery-exporter/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=danihodovic/celery-exporter\" /\u003e\n\u003c/a\u003e\n\nMade with [contrib.rocks](https://contrib.rocks).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanihodovic%2Fcelery-exporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanihodovic%2Fcelery-exporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanihodovic%2Fcelery-exporter/lists"}