{"id":16544493,"url":"https://github.com/steinitzu/celery-singleton","last_synced_at":"2025-05-16T19:03:53.774Z","repository":{"id":20704048,"uuid":"90670580","full_name":"steinitzu/celery-singleton","owner":"steinitzu","description":"Seamlessly prevent duplicate executions of celery tasks","archived":false,"fork":false,"pushed_at":"2023-06-09T15:34:01.000Z","size":69,"stargazers_count":239,"open_issues_count":29,"forks_count":36,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-10-18T22:23:28.378Z","etag":null,"topics":["celery","celery-tasks","distributed-locks","python","rate-limiting","redis","singleton-task"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/steinitzu.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-08T20:41:03.000Z","updated_at":"2024-10-12T06:40:05.000Z","dependencies_parsed_at":"2024-06-18T18:38:05.574Z","dependency_job_id":"78400688-dd65-4989-91bd-836839644ea0","html_url":"https://github.com/steinitzu/celery-singleton","commit_stats":{"total_commits":54,"total_committers":7,"mean_commits":7.714285714285714,"dds":"0.20370370370370372","last_synced_commit":"4ee161853d82f71d4565be2bd7d81fc6d8bcbded"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steinitzu%2Fcelery-singleton","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steinitzu%2Fcelery-singleton/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steinitzu%2Fcelery-singleton/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steinitzu%2Fcelery-singleton/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/steinitzu","download_url":"https://codeload.github.com/steinitzu/celery-singleton/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248607044,"owners_count":21132462,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["celery","celery-tasks","distributed-locks","python","rate-limiting","redis","singleton-task"],"created_at":"2024-10-11T19:02:59.348Z","updated_at":"2025-04-12T17:39:48.140Z","avatar_url":"https://github.com/steinitzu.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Celery-Singleton\n\nDuplicate tasks clogging up your message broker? Do time based rate limits make you feel icky? Look no further!\nThis is a baseclass for celery tasks that ensures only one instance of the task can be queued or running at any given time. Uses the task's name+arguments to determine uniqueness.\n\n\u003c!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc --\u003e\n**Table of Contents**\n\n- [Celery-Singleton](#celery-singleton)\n    - [Prerequisites](#prerequisites)\n    - [Quick start](#quick-start)\n    - [How does it work?](#how-does-it-work)\n    - [Handling deadlocks](#handling-deadlocks)\n    - [Backends](#backends)\n    - [Task configuration](#task-configuration)\n        - [unique\\_on](#uniqueon)\n        - [raise\\_on\\_duplicate](#raiseonduplicate)\n    - [App Configuration](#app-configuration)\n    - [Testing](#testing)\n    - [Contribute](#contribute)\n\n\u003c!-- markdown-toc end --\u003e\n\n\n## Prerequisites\ncelery-singleton uses the JSON representation of a task's `delay()` or `apply_async()` arguments to generate a unique lock and stores it in redis.\nBy default it uses the redis server of the celery [result backend](http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html#keeping-results). If you use a different/no result backend or want to use a different redis server for celery-singleton, refer the [configuration section](#app-configuration) for how to customize the redis. To use something other than redis, refer to the section on [backends](#backends)\n\nSo in gist:\n1. Make sure all your tasks arguments are JSON serializable\n2. Your celery app is configured with a redis result backend or you have specified another redis/compatible backend in your config\n\nIf you're already using a redis backend and a mostly default celery config, you're all set!\n\n## Quick start\n`$ pip install celery-singleton`\n\n```python\nimport time\nfrom celery_singleton import Singleton\nfrom somewhere import celery_app\n\n@celery_app.task(base=Singleton)\ndef do_stuff(*args, **kwargs):\n\ttime.sleep(4)\n\treturn 'I just woke up'\n\n# run the task as normal\nasync_result = do_stuff.delay(1, 2, 3, a='b')\nasync_result2 = do_stuff.delay(1, 2, 3, a='b')\n\nassert async_result == async_result2  # These are the same, task is only queued once\n```\n\nThat's it! Your task is a singleton and calls to `do_stuff.delay()` will either queue a new task or return an AsyncResult for the currently queued/running instance of the task.\n\n\n## How does it work?\n\nThe `Singleton` class overrides `apply_async()` of the base task implementation to only queue a task if an identical task is not already running. Two tasks are considered identical if both have the same name and same arguments.\n\nThis is achieved by using redis for distributed locking.\n\nWhen you call `delay()` or `apply_async()` on a singleton task it first attempts to aquire a lock in redis using a hash of [task_name+arguments] as a key and a new task ID as a value. `SETNX` is used for this to prevent race conditions.\nIf a lock is successfully aquired, the task is queued as normal with the `apply_async` method of the base class.\nIf another run of the task already holds a lock, we fetch its task ID instead and return an `AsyncResult` for it. This way it works seamlessly with a standard celery setup, there are no \"duplicate exceptions\" you need to handle, no timeouts. `delay()` always returns an `AsyncResult` as expected, either for the task you just spawned or for the task that aquired the lock before it.\nSo continuing on with the \"Quick start\" example:\n\n```python\na = do_stuff.delay(1, 2, 3)\nb = do_stuff.delay(1, 2, 3)\n\nassert a == b  # Both are AsyncResult for the same task\n\nc = do_stuff.delay(4, 5, 6)\n\nassert a != c  # c has different arguments so it spawns a new task\n```\n\nThe lock is released only when the task has finished running, using either the `on_success` or `on_failure` handler, after which you're free to start another identical run.\n\n```python\n# wait for a to finish\na.get()\n\n# Now we can spawn a duplicate of it\nd = do_stuff.delay(1, 2, 3)\n\nassert a != d\n```\n\n\n## Handling deadlocks\nSince the task locks are only released when the task is actually finished running (on success or on failure), you can sometimes end up in a situation where the lock remains but there's no task available to release it.\nThis can for example happen if your celery worker crashes before it can release the lock.\n\nA convenience method is included to clear all existing locks, you can run it on celery worker startup or any other celery signal like so:\n\n```python\nfrom celery.signals import worker_ready\nfrom celery_singleton import clear_locks\nfrom somewhere import celery_app\n\n@worker_ready.connect\ndef unlock_all(**kwargs):\n    clear_locks(celery_app)\n```\n\nAn alternative is to set a [lock expiry](#lock\\_expiry) time in the task or app config. This makes it so that locks are always released after a given time.\n\n## Backends\n\nRedis is the default storage backend for celery singleton. This is where task locks are stored where they can be accessed across celery workers.\nA custom redis url can be set using the `singleton_backend_url` config variable in the celery config. By default Celery Singleton attempts to use the redis url of the celery result backend and if that fails the celery broker.\n\nIf you don't want to use redis you can implement a custom storage backend.\nAn abstract base class to inherit from is included in `celery_singleton.backends.BaseBackend` and [the source code of `RedisBackend`](celery_singleton/backends/redis.py) serves as an example implementation.\nOnce you have your backend implemented, set the `singleton_backend_class` [configuration](#app-configuration) variables to point to your class.\n\n\n## Task configuration\n\n### unique\\_on\n\nThis can be used to make celery-singleton only consider certain arguments when deciding whether two tasks are identical.\n(By default, two tasks are considered identical to each other if their name and all arguments are the same).\n\nFor example, this task allows only one instance per username, other arguments don't matter:\n\n```python\n@app.task(base=Singleton, unique_on=['username', ])\ndef do_something(username, otherarg=None):\n    time.sleep(5)\n\n\ntask1 = do_something.delay(username='bob', otherarg=99)\ntask2 = do_something.delay(username='bob', otherarg=100)  # this is a duplicate of task1\nassert task1 == task2\n```\n\nSpecify an empty list to consider the task name only.\n\n### raise\\_on\\_duplicate\n\nWhen this option is enabled the task's `delay` and `apply_async` method will raise a `DuplicateTaskError` exception when attempting to spawn a duplicate task instead of returning the existing task's `AsyncResult`\nThis is useful when you want only one of a particular task at a time, but want more control over what happens on duplicate attempts.\n\n```python\nfrom celery_singleton import Singleton, DuplicateTaskError\n\n\n@app.task(base=Singleton, raise_on_duplicate=True)\ndef do_something(username):\n    time.sleep(5)\n\ntask1 = do_something.delay('bob')\ntry:\n    task2 = do_something.delay('bob')\nexcept DuplicateTaskerror as e:\n    print(\"You tried to create a duplicate of task with ID\", e.task_id)\n```\n\nThis option can also be applied globally to all `Singleton` tasks by setting `singleton_raise_on_duplicate` in the [app config](#app-configuration). The task level option always overrides the app config when supplied.\n\n### lock\\_expiry\n\nNumber of seconds until the task lock expires. This is useful when you want a max of one task queued within a given time frame rather than strictly one at a time.\nThis also adds some safety to your application as it guarantees that locks will eventually be released in case of worker crashes and network failures. For this use case it's recommended to set the lock expiry to a value slightly longer than the expected task duration.\n\nExample\n\n```python\n@app.task(base=Singleton, lock_expiry=10)\ndef runs_for_12_seconds():\n    self.time.sleep(12)\n\n\ntask1 = runs_for_12_seconds.delay()\ntime.sleep(11)\ntask2 = runs_for_12_seconds.delay()\n\nassert task1 != task2  # These are two separate task instances\n```\n\nThis option can be applied globally in the [app config](#app-configuration) with `singleton_lock_expiry`. Task option supersedes the app config.\n\n\n## App Configuration\n\nCelery singleton supports the following configuration option. These should be added to your Celery app config.\nNote: if using old style celery config with uppercase variables and a namespace, make sure the singleton config matches. E.g. `CELERY_SINGLETON_BACKEND_URL` instead of `singleton_backend_url`\n\n\n| Key                            | Default                                 | Description                                                                                                                                                          |\n|--------------------------------|-----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `singleton_backend_url`        | `celery_backend_url`                    | The URL of the storage backend. If using the default backend implementation, this should be a redis URL. It is passed as the first argument to the backend class.    |\n| `singleton_backend_class`      | `celery_singleton.backend.RedisBackend` | The full import path of a backend class as string or a reference to the class                                                                                       |\n| `singleton_backend_kwargs`     | `{}`                                    | Passed as keyword arguments to the backend class                                                                                                                     |\n| `singleton_json_encoder_class` | `None` ([`json.JSONEncoder`]) | Optional JSON encoder class for generating lock. Useful for task arguments where objects can be reliably marshalled to string (such as [`uuid.UUID`])                                                                                              |\n| `singleton_key_prefix`         | `SINGLETONLOCK_`                        | Locks are stored as `\u003ckey_prefix\u003e\u003clock\u003e`. Use to prevent collisions with other keys in your database.                                                                |\n| `singleton_raise_on_duplicate` | `False`                                 | When `True` an attempt to queue a duplicate task will raise a `DuplicateTaskerror`. The default behavior is to return the `AsyncResult` for the existing task.       |\n| `singleton_lock_expiry`        | `None` (Never expires)                  | Lock expiry time in second for singleton task locks. When lock expires identical tasks are allowed to run regardless of whether the locked task has finished or not. |\n|                                |                                         |                                                                                                                                                                      |\n\n[`json.JSONEncoder`]: https://docs.python.org/3/library/json.html#json.JSONEncoder\n[`uuid.UUID`]: https://docs.python.org/3/library/uuid.html#uuid.UUID\n\n## Testing\n\nTests are located in the `/tests` directory can be run with pytest\n\n```\npip install -r dev-requirements.txt\npython -m pytest\n```\n\nSome of the tests require a running redis server on `redis://localhost`\nTo use a redis server on a different url/host, set the env variable `CELERY_SINGLETON_TEST_REDIS_URL`\n\n\n## Contribute\nPlease open an issue if you encounter a bug, have any questions or suggestions for improvements or run into any trouble at all using this package.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsteinitzu%2Fcelery-singleton","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsteinitzu%2Fcelery-singleton","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsteinitzu%2Fcelery-singleton/lists"}