https://github.com/joealcorn/celery-dedupe
Deduplication of Celery tasks
https://github.com/joealcorn/celery-dedupe
celery pluggable python redis
Last synced: about 1 year ago
JSON representation
Deduplication of Celery tasks
- Host: GitHub
- URL: https://github.com/joealcorn/celery-dedupe
- Owner: joealcorn
- License: mit
- Created: 2015-08-19T06:12:29.000Z (almost 11 years ago)
- Default Branch: master
- Last Pushed: 2023-10-25T16:45:11.000Z (over 2 years ago)
- Last Synced: 2025-04-13T07:05:27.762Z (about 1 year ago)
- Topics: celery, pluggable, python, redis
- Language: Python
- Homepage:
- Size: 19.5 KB
- Stars: 7
- Watchers: 2
- Forks: 2
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# celery-dedupe [](https://travis-ci.org/joealcorn/celery-dedupe)
celery-dedupe is a project that aims to be a pluggable solution for deduplicating queued celery tasks.
Only redis is supported as a storage backend for the moment.
# Usage
```python
from celery.task import task
from celery_dedupe import DedupeTask
from celery_dedupe.storage.redis import RedisStorage
from redis import StrictRedis
redis = StrictRedis()
storage = RedisStorage(redis, expiry=60)
@task(base=DedupeTask, storage=storage)
def noop_task(*a, **kw):
return None
```
# Caveats and things to watch out for
#### CELERY_ALWAYS_EAGER can not return results
Any subsequent invocation of a task when `CELERY_ALWAYS_EAGER` is set to `True` is unable to return
an `EagerResult` object.
#### Unacknowledged tasks
Any unacked tasks will not be able to be requeued by usual means.
Setting an expiry on the redis storage backend will allow you to work around this if it's not vital
that multiple tasks are queued at once.
#### Manually purging your broker
If you need to manually consume tasks or completely purge your broker for whatever reason you'll
need to remove the corresponding keys from your storage backend as well so that tasks can continue
to run.
#### Tasks with ETAs, countdowns or retry delays
Any task with a delayed start time will prevent other tasks from running immediately. If a task
retries many times with an exponential backoff this can be a long period of time. The key expiry
on the redis backend can minimise the effect of this.
#### Unregistered tasks
If you somehow queue an unregistered task (most likely during development), you will not be able
to requeue even when registered without clearing the lock from your storage backend