{"id":18446462,"url":"https://github.com/ovh/celery-dyrygent","last_synced_at":"2025-04-09T21:18:58.832Z","repository":{"id":42535651,"uuid":"195953256","full_name":"ovh/celery-dyrygent","owner":"ovh","description":"Celery extension which allows to orchestrate 100/1000/10000 tasks combined into a complex workflow","archived":false,"fork":false,"pushed_at":"2023-09-13T09:22:03.000Z","size":85,"stargazers_count":100,"open_issues_count":4,"forks_count":16,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-09T21:18:54.938Z","etag":null,"topics":["automation","celery","workflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ovh.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-07-09T07:13:13.000Z","updated_at":"2025-03-31T14:10:39.000Z","dependencies_parsed_at":"2023-01-31T06:31:11.024Z","dependency_job_id":null,"html_url":"https://github.com/ovh/celery-dyrygent","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovh%2Fcelery-dyrygent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovh%2Fcelery-dyrygent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovh%2Fcelery-dyrygent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovh%2Fcelery-dyrygent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ovh","download_url":"https://codeload.github.com/ovh/celery-dyrygent/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248111973,"owners_count":21049578,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","celery","workflow"],"created_at":"2024-11-06T07:09:27.297Z","updated_at":"2025-04-09T21:18:58.773Z","avatar_url":"https://github.com/ovh.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Celery Dyrygent\n\n[![Python 3.8](https://img.shields.io/badge/python-3.8-blue.svg)](https://www.python.org/downloads/release/python-380/)\n[![Python 3.9](https://img.shields.io/badge/python-3.9-blue.svg)](https://www.python.org/downloads/release/python-390/)\n[![Python 3.10](https://img.shields.io/badge/python-3.10-blue.svg)](https://www.python.org/downloads/release/python-3100/)\n[![Celery Supported](https://img.shields.io/badge/celery-4.2%2B-green)](https://docs.celeryq.dev/en/v4.2.0/whatsnew-4.2.html)\n[![Unit Tests](https://github.com/ovh/celery-dyrygent/actions/workflows/unit-tests.yml/badge.svg)](https://github.com/ovh/celery-dyrygent/actions/workflows/unit-tests.yml)\n[![Integration Tests](https://github.com/ovh/celery-dyrygent/actions/workflows/integration-tests.yml/badge.svg)](https://github.com/ovh/celery-dyrygent/actions/workflows/integration-tests.yml)\n[![PyPI](https://img.shields.io/pypi/v/celery-dyrygent)](https://pypi.org/project/celery-dyrygent/)\n[![PyPI - Wheel](https://img.shields.io/pypi/wheel/celery-dyrygent)](https://pypi.org/project/celery-dyrygent/#files)\n\n\nThis project aims to support full DAG workflow processing.\nIt's designed as celery extension and uses celery as an execution backend.\nCelery-dyrygent is released under modified BSD license. See [license](LICENSE)\n\n## What is it?\nThe reasons behind this project so as the implementation details were described in the following blogpost\n[https://www.ovh.com/blog/doing-big-automation-with-celery/](https://blog.ovhcloud.com/doing-big-automation-with-celery/)\n\n### What is a DAG workflow?\nDAG is a shortcut for Directed Acyclic Graph.\nWhile DAG workflow would be any combination of celery primitives:\n- groups\n- chains\n- chords\n\nCelery Dyrygent is able to process any kind of DAG workflows.\n\n### Why not to use native celery stuff?\nCelery struggles a bit with complex workflows built from combining primitives.\nThe execution might be unreliable, there are a lot of corner cases where workflow might not work as desired.\nSerialization of complex workflows causes memory issues.\nSome of the encountered problems which aren't solved (celery 4.2.1):\n- https://github.com/celery/celery/issues/5000\n- https://github.com/celery/celery/issues/5286\n- https://github.com/celery/celery/issues/5327\n\n## How does it work?\nThe whole workflow machinery works simialar to ```chord_unlock``` repeating celery task which waits till some tasks are done (header) and then executes further tasks (body).\nCelery Dyrygent introduces a workflow processor task which orchestrates an execution of a whole workflow.\nOnce the workflow is started the workflow processor task is repeated till the workflow execution is done or till some TTL timestamp is reached (not to repeat indefinitely).\nThe workflow processor schedules the execution of tasks according to their relations, retries itself, then checks if the tasks are done so the new ones can be scheduled, repeat.\nThat's it, the idea is quite simple.\n\n### Advantages\n- execution part is done by Celery, so all celery machinery with its features is available (retries, countdowns, etc.)\n- each workflow is executed in the same way\n- Celery operates on simple tasks only - no nested structures which causes troubles\n- link error for whole workflow can be implemented\n- finalizing task for whole workflow can be implemented (e.g. do something always when workflow finishes)\n- workflow execution is SOLID and RELIABLE\n- it's possible to track progress through signals (might need to implement a new signal for each tick)\n\n### Drawbacks\n- At the moment workflow processor doesn't pass task results from preceding tasks to following tasks (can be implemented, not implemented at the moment).\n- Workflow processor task is doing repeating ticks (like celery chord unlock) and new tasks are scheduled only within the ticks. This may result in noticeably longer execution time of task chains (e.g. if ticks are done each 2s, next task in chain will only be each 2s)\n- Reliable result backend has to be enabled\n\n# How to use it?\n## Which celery versions are supported?\n- celery 4.2 and newer\n\n## Integration\n### Initialize workflows\nYou need to register workflow processor task in your celery app\n```python\nfrom celery_dyrygent.tasks import register_workflow_processor\n\napp = Celery() #  your celery application instance\n\nworkflow_processor = register_workflow_processor(app)\n```\n\n### Use workflow on you celery canvas\nWorkflows can consume celery canvas to properly build internal relations\n```python\nfrom celery_dyrygent.workflows import Workflow\n\ncanvas = celery.chain() | celery.chord() #  define your canvas using native celery mechanisms\n\nwf = Workflow()\nwf.add_celery_canvas(canvas)\nwf.apply_async()\n```\n\nWorkflow processor task will be scheduled holding all signatures from canvas and their relations. It will execute signatures according to their relations.\n\n### Signals support\nCelery Dyrygent provides additional signals which can be used e.g. for tracking workflow progress. Following signals are available:\n- ```after_active_tick```\n- ```on_finish```\n- ```on_state_change```\n\n#### How to use signals?\nWhen a signal is emitted all registered signal handlers are executed. In order to register signal handler you need to use ```Workflow.connect``` function. See examples below. The handler is called with two parameters: workflow instance and payload (optional).\n\n#### Using ```on_state_change``` signal\nSignal is emitted when workflow state changes. Supported states are:\n- INITIAL\n- RUNNING\n- SUCCESS\n- FAILURE\n- ERROR\n\nHandler is called with two params:\n- workflow instance\n- payload - current state of a workflow\n\n``` python\nfrom celery_dyrygent.workflows import Workflow\n\n@Workflow.connect('on_state_change')\ndef handle_state_change(workflow, payload):\n    print(\n        \"Workflow {} has new state {}\"\n        .format(workflow.id, payload)\n    )\n```\n\n#### Using ```on_finish``` signal\nSignal is emitted when workflow is finished (or can't move forward due to failed tasks)\n\nHandler params are:\n- workflow instance\n- paylod - None\n\n#### Using ```after_active_tick```\nSignal is emitted when workflow has scheduled new tasks\n\nHandler params are:\n- workflow instance\n- payload - None\n\n\n### Support for custom data\nBoth `Workflow`and `WorkflowNode` have a `custom_payload` dictionary member that can be used to store \nadditional data. For example, one can use those dictionnary to store some application specific \nmetadata.\n\n```python\n...\nwf = Workflow()\nfor task in task_list:\n    sig = create_celery_task(task)\n    sig.freeze()\n    node = wf.add_signature(sig)\n    node.custom_payload['user_id'] = task.user_id\n...\n```\n\n#### Using celery task options\nYou can define custom options for your tasks, as defined in:\nhttps://docs.celeryproject.org/en/stable/reference/celery.app.task.html#celery.app.task.Task.apply_async\n\nThese options may be different between the workflow task and user tasks.\n``` python\nwf = Workflow(options={'priority': 10})\nwf.add_celery_canvas(canvas)\nwf.apply_async(options={'priority': 8})\n```\n\n## TODO\n- Proper documentation (e.g. sphinx)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovh%2Fcelery-dyrygent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fovh%2Fcelery-dyrygent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovh%2Fcelery-dyrygent/lists"}