{"id":17128218,"url":"https://github.com/joealcorn/celery-dedupe","last_synced_at":"2025-04-13T07:05:58.023Z","repository":{"id":36710049,"uuid":"41016599","full_name":"joealcorn/celery-dedupe","owner":"joealcorn","description":"Deduplication of Celery tasks","archived":false,"fork":false,"pushed_at":"2023-10-25T16:45:11.000Z","size":20,"stargazers_count":7,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-13T07:05:27.762Z","etag":null,"topics":["celery","pluggable","python","redis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joealcorn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-08-19T06:12:29.000Z","updated_at":"2025-04-08T20:58:25.000Z","dependencies_parsed_at":"2022-09-10T05:01:30.255Z","dependency_job_id":null,"html_url":"https://github.com/joealcorn/celery-dedupe","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joealcorn%2Fcelery-dedupe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joealcorn%2Fcelery-dedupe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joealcorn%2Fcelery-dedupe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joealcorn%2Fcelery-dedupe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joealcorn","download_url":"https://codeload.github.com/joealcorn/celery-dedupe/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248675426,"owners_count":21143768,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["celery","pluggable","python","redis"],"created_at":"2024-10-14T19:06:32.271Z","updated_at":"2025-04-13T07:05:57.897Z","avatar_url":"https://github.com/joealcorn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# celery-dedupe [![Build Status](https://travis-ci.org/joealcorn/celery-dedupe.svg?branch=master)](https://travis-ci.org/joealcorn/celery-dedupe)\n\ncelery-dedupe is a project that aims to be a pluggable solution for deduplicating queued celery tasks.\nOnly redis is supported as a storage backend for the moment.\n\n# Usage\n\n```python\nfrom celery.task import task\nfrom celery_dedupe import DedupeTask\nfrom celery_dedupe.storage.redis import RedisStorage\nfrom redis import StrictRedis\n\nredis = StrictRedis()\nstorage = RedisStorage(redis, expiry=60)\n\n@task(base=DedupeTask, storage=storage)\ndef noop_task(*a, **kw):\n    return None\n```\n\n\n# Caveats and things to watch out for\n\n#### CELERY_ALWAYS_EAGER can not return results\n\nAny subsequent invocation of a task when `CELERY_ALWAYS_EAGER` is set to `True` is unable to return\nan `EagerResult` object.\n\n\n#### Unacknowledged tasks\n\nAny unacked tasks will not be able to be requeued by usual means.\nSetting an expiry on the redis storage backend will allow you to work around this if it's not vital\nthat multiple tasks are queued at once.\n\n\n#### Manually purging your broker\n\nIf you need to manually consume tasks or completely purge your broker for whatever reason you'll\nneed to remove the corresponding keys from your storage backend as well so that tasks can continue\nto run.\n\n\n#### Tasks with ETAs, countdowns or retry delays\n\nAny task with a delayed start time will prevent other tasks from running immediately. If a task\nretries many times with an exponential backoff this can be a long period of time. The key expiry\non the redis backend can minimise the effect of this.\n\n\n#### Unregistered tasks\n\nIf you somehow queue an unregistered task (most likely during development), you will not be able\nto requeue even when registered without clearing the lock from your storage backend\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoealcorn%2Fcelery-dedupe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoealcorn%2Fcelery-dedupe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoealcorn%2Fcelery-dedupe/lists"}