{"id":16713965,"url":"https://github.com/tktech/celery-heimdall","last_synced_at":"2025-07-17T14:07:48.688Z","repository":{"id":56870380,"uuid":"526700647","full_name":"TkTech/celery-heimdall","owner":"TkTech","description":"Helpful celery task queue extensions.","archived":false,"fork":false,"pushed_at":"2024-12-10T16:18:52.000Z","size":54,"stargazers_count":28,"open_issues_count":3,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-10T01:09:16.827Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TkTech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"TkTech","patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2022-08-19T17:43:34.000Z","updated_at":"2025-03-08T23:12:30.000Z","dependencies_parsed_at":"2024-10-18T22:52:52.492Z","dependency_job_id":"fa5f74dd-30d2-46a7-b144-a02000fd5017","html_url":"https://github.com/TkTech/celery-heimdall","commit_stats":{"total_commits":35,"total_committers":1,"mean_commits":35.0,"dds":0.0,"last_synced_commit":"43727fa88a3e0b57b778f55531884c3999854199"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/TkTech/celery-heimdall","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TkTech%2Fcelery-heimdall","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TkTech%2Fcelery-heimdall/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TkTech%2Fcelery-heimdall/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TkTech%2Fcelery-heimdall/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TkTech","download_url":"https://codeload.github.com/TkTech/celery-heimdall/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TkTech%2Fcelery-heimdall/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265614332,"owners_count":23798427,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-12T20:48:40.568Z","updated_at":"2025-07-17T14:07:48.672Z","avatar_url":"https://github.com/TkTech.png","language":"Python","funding_links":["https://github.com/sponsors/TkTech"],"categories":[],"sub_categories":[],"readme":"# celery-heimdall\n\n[![codecov](https://codecov.io/gh/TkTech/celery-heimdall/branch/main/graph/badge.svg?token=1A2CVHQ25Q)](https://codecov.io/gh/TkTech/celery-heimdall)\n![GitHub](https://img.shields.io/github/license/tktech/celery-heimdall)\n![PyPI - Python Version](https://img.shields.io/pypi/pyversions/celery-heimdall)\n\nCelery Heimdall is a set of common utilities useful for the Celery background\nworker framework, built on top of Redis. It's not trying to handle every use\ncase, but to be an easy, modern, and maintainable drop-in solution for 90% of\nprojects.\n\n## Features\n\n- Globally unique tasks, allowing only 1 copy of a task to execute at a time, or\n  within a time period (ex: \"Don't allow queuing until an hour has passed\")\n- Global rate limiting. Celery has built-in rate limiting, but it's a rate limit\n  _per worker_, making it unsuitable for purposes such as limiting requests to\n  an API.\n\n## Installation\n\n`pip install celery-heimdall`\n\n## Usage\n\n### Unique Tasks\n\nImagine you have a task that starts when a user presses a button. This task\ntakes a long time and a lot of resources to generate a report. You don't want\nthe user to press the button 10 times and start 10 tasks. In this case, you\nwant what Heimdall calls a unique task:\n\n```python\nfrom celery import shared_task\nfrom celery_heimdall import HeimdallTask\n\n@shared_task(base=HeimdallTask, heimdall={'unique': True})\ndef generate_report(customer_id):\n    pass\n```\n\nAll we've done here is change the base Task class that Celery will use to run\nthe task, and passed in some options for Heimdall to use. This task is now\nunique - for the given arguments, only 1 will ever run at the same time.\n\n#### Expiry\n\nWhat happens if our task dies, or something goes wrong? We might end up in a\nsituation where our lock never gets cleared, called [deadlock][]. To work around\nthis, we add a maximum time before the task is allowed to be queued again:\n\n\n```python\nfrom celery import shared_task\nfrom celery_heimdall import HeimdallTask\n\n@shared_task(\n  base=HeimdallTask,\n  heimdall={\n    'unique': True,\n    'unique_timeout': 60 * 60\n  }\n)\ndef generate_report(customer_id):\n  pass\n```\n\nNow, `generate_report` will be allowed to run again in an hour even if the\ntask got stuck, the worker ran out of memory, the machine burst into flames,\netc...\n\n#### Custom Keys\n\nBy default, a hash of the task name and its arguments is used as the lock key.\nBut this often might not be what you want. What if you only want one report at\na time, even for different customers? Ex:\n\n```python\nfrom celery import shared_task\nfrom celery_heimdall import HeimdallTask\n\n@shared_task(\n  base=HeimdallTask,\n  heimdall={\n    'unique': True,\n    'key': lambda args, kwargs: 'generate_report'\n  }\n)\ndef generate_report(customer_id):\n  pass\n```\nBy specifying our own key function, we can completely customize how we determine\nif a task is unique.\n\n#### The Existing Task\n\nBy default, if you try to queue up a unique task that is already running,\nHeimdall will return the existing task's `AsyncResult`. This lets you write\nsimple code that doesn't need to care if a task is unique or not. Imagine a\nsimple API endpoint that starts a report when it's hit, but we only want it\nto run one at a time. The below is all you need:\n\n```python\nimport time\nfrom celery import shared_task\nfrom celery_heimdall import HeimdallTask\n\n@shared_task(base=HeimdallTask, heimdall={'unique': True})\ndef generate_report(customer_id):\n  time.sleep(10)\n\ndef my_api_call(customer_id: int):\n  return {\n    'status': 'RUNNING',\n    'task_id': generate_report.delay(customer_id).id\n  }\n```\n\nEverytime `my_api_call` is called with the same `customer_id`, the same\n`task_id` will be returned by `generate_report.delay()` until the original task\nhas completed.\n\nSometimes you'll want to catch that the task was already running when you tried\nto queue it again. We can tell Heimdall to raise an exception in this case:\n\n```python\nimport time\nfrom celery import shared_task\nfrom celery_heimdall import HeimdallTask, AlreadyQueuedError\n\n\n@shared_task(\n  base=HeimdallTask,\n  heimdall={\n    'unique': True,\n    'unique_raises': True\n  }\n)\ndef generate_report(customer_id):\n  time.sleep(10)\n\n\ndef my_api_call(customer_id: int):\n  try:\n    task = generate_report.delay(customer_id)\n    return {'status': 'STARTED', 'task_id': task.id}\n  except AlreadyQueuedError as exc:\n    return {'status': 'ALREADY_RUNNING', 'task_id': exc.likely_culprit}\n```\n\nBy setting `unique_raises` to `True` when we define our task, an\n`AlreadyQueuedError` will be raised when you try to queue up a unique task\ntwice. The `AlreadyQueuedError` has two properties:\n\n- `likely_culprit`, which contains the task ID of the already-running task,\n- `expires_in`, which is the time remaining (in seconds) before the \n  already-running task is considered to be expired.\n\n#### Unique Interval Task\n\nWhat if we want the task to only run once in an hour, even if it's finished?\nIn those cases, we want it to run, but not clear the lock when it's finished:\n\n```python\nfrom celery import shared_task\nfrom celery_heimdall import HeimdallTask\n\n@shared_task(\n  base=HeimdallTask,\n  heimdall={\n    'unique': True,\n    'unique_timeout': 60 * 60,\n    'unique_wait_for_expiry': True\n  }\n)\ndef generate_report(customer_id):\n  pass\n```\n\nBy setting `unique_wait_for_expiry` to `True`, the task will finish, and won't\nallow another `generate_report()` to be queued until `unique_timeout` has\npassed.\n\n### Rate Limiting\n\nCelery offers rate limiting out of the box. However, this rate limiting applies\non a per-worker basis. There's no reliable way to rate limit a task across all\nyour workers. Heimdall makes this easy:\n\n```python\nfrom celery import shared_task\nfrom celery_heimdall import HeimdallTask, RateLimit\n\n@shared_task(\n  base=HeimdallTask,\n  heimdall={\n    'rate_limit': RateLimit((2, 60))\n  }\n)\ndef download_report_from_amazon(customer_id):\n  pass\n```\n\nThis says \"every 60 seconds, only allow this task to run 2 times\". If a task\ncan't be run because it would violate the rate limit, it'll be rescheduled.\n\nIt's important to note this does not guarantee that your task will run _exactly_\ntwice a second, just that it won't run _more_ than twice a second. Tasks are\nrescheduled with a random jitter to prevent the [thundering herd][] problem.\n\n\n#### Dynamic Rate Limiting\n\nJust like you can dynamically provide a key for a task, you can also\ndynamically provide a rate limit based off that key.\n\n\n```python\nfrom celery import shared_task\nfrom celery_heimdall import HeimdallTask, RateLimit\n\n\n@shared_task(\n  base=HeimdallTask,\n  heimdall={\n    # Provide a lower rate limit for the customer with the ID 10, for everyone\n    # else provide a higher rate limit.\n    'rate_limit': RateLimit(lambda args: (1, 30) if args[0] == 10 else (2, 30)),\n    'key': lambda args, kwargs: f'customer_{args[0]}'\n  }\n)\ndef download_report_from_amazon(customer_id):\n  pass\n```\n\n\n## Inspirations\n\nThese are more mature projects which inspired this library, and which may\nsupport older versions of Celery \u0026 Python then this project.\n\n- [celery_once][], which is unfortunately abandoned and the reason this project\n  exists.\n- [celery_singleton][]\n- [This snippet][snip] by Vigrond, and subsequent improvements by various\n  contributors.\n\n\n[celery_once]: https://github.com/cameronmaske/celery-once\n[celery_singleton]: https://github.com/steinitzu/celery-singleton\n[deadlock]: https://en.wikipedia.org/wiki/Deadlock\n[thundering herd]: https://en.wikipedia.org/wiki/Thundering_herd_problem\n[snip]: https://gist.github.com/Vigrond/2bbea9be6413415e5479998e79a1b11a","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftktech%2Fcelery-heimdall","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftktech%2Fcelery-heimdall","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftktech%2Fcelery-heimdall/lists"}