{"id":18349166,"url":"https://github.com/ddeutils/ddeutil-workflow","last_synced_at":"2025-10-24T07:17:26.802Z","repository":{"id":239791828,"uuid":"800584406","full_name":"ddeutils/ddeutil-workflow","owner":"ddeutils","description":"πŸƒ Lightweight Workflow Orchestration with YAML template","archived":false,"fork":false,"pushed_at":"2025-10-01T12:06:50.000Z","size":3684,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-17T17:57:05.512Z","etag":null,"topics":["fastapi","pydantic-v2","workflow-orchestration","yaml-template"],"latest_commit_sha":null,"homepage":"https://ddeutils.github.io/ddeutil-workflow/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ddeutils.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-05-14T15:58:23.000Z","updated_at":"2025-09-21T05:20:04.000Z","dependencies_parsed_at":"2025-04-02T16:31:42.052Z","dependency_job_id":"a09dd957-c896-4c51-9aed-53f503892d60","html_url":"https://github.com/ddeutils/ddeutil-workflow","commit_stats":null,"previous_names":["ddeutils/ddeutil-pipe","ddeutils/ddeutil-workflow"],"tags_count":88,"template":false,"template_full_name":null,"purl":"pkg:github/ddeutils/ddeutil-workflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ddeutils%2Fddeutil-workflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ddeutils%2Fddeutil-workflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ddeutils%2Fddeutil-workflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ddeutils%2Fddeutil-workflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ddeutils","download_url":"https://codeload.github.com/ddeutils/ddeutil-workflow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ddeutils%2Fddeutil-workflow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280756573,"owners_count":26385363,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-24T02:00:06.418Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fastapi","pydantic-v2","workflow-orchestration","yaml-template"],"created_at":"2024-11-05T21:20:54.219Z","updated_at":"2025-10-24T07:17:26.793Z","avatar_url":"https://github.com/ddeutils.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Workflow Orchestration\n\n[![test](https://github.com/ddeutils/ddeutil-workflow/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/ddeutils/ddeutil-workflow/actions/workflows/tests.yml)\n[![codecov](https://codecov.io/gh/ddeutils/ddeutil-workflow/graph/badge.svg?token=3NDPN2I0H9)](https://codecov.io/gh/ddeutils/ddeutil-workflow)\n[![pypi version](https://img.shields.io/pypi/v/ddeutil-workflow)](https://pypi.org/project/ddeutil-workflow/)\n[![python support version](https://img.shields.io/pypi/pyversions/ddeutil-workflow)](https://pypi.org/project/ddeutil-workflow/)\n[![size](https://img.shields.io/github/languages/code-size/ddeutils/ddeutil-workflow)](https://github.com/ddeutils/ddeutil-workflow)\n[![gh license](https://img.shields.io/github/license/ddeutils/ddeutil-workflow)](https://github.com/ddeutils/ddeutil-workflow/blob/main/LICENSE)\n[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\nThe **Lightweight Workflow Orchestration** with fewer dependencies the was created\nfor easy to make a simple metadata driven data workflow. It can use for data operator\nby a `.yaml` template.\n\n\u003e [!WARNING]\n\u003e This package provide only orchestration workload. That mean you should not\n\u003e use the workflow stage to process any large volume data which use a lot of compute\n\u003e resource :cold_sweat:.\n\n---\n\n**:pushpin: \u003cu\u003eRules of This Workflow\u003c/u\u003e**:\n\n1. The Minimum frequency unit of built-in scheduling is **1 Minute** πŸ•˜\n2. **Can not** re-run only failed stage and its pending downstream ↩️\n3. All parallel tasks inside workflow core engine use **Multi-Threading** pool\n (Python 3.13 unlock GIL πŸπŸ”“)\n4. Recommend to pass a **Secret Value** with environment variable in YAML template πŸ”\n5. Any datatime value convert to **UTC Timezone** 🌐\n\n---\n\n**:memo: \u003cu\u003eWorkflow Diagrams\u003c/u\u003e**:\n\nThis diagram show where is this application run on the production infrastructure.\nYou will see that this application do only running code with stress-less which mean\nyou should to set the data layer separate this core program before run this application.\n\n```mermaid\nflowchart LR\n A((fa:fa-user User))\n\n subgraph Docker Container\n direction TB\n G@{ shape: rounded, label: \"πŸ“‘Observe\u003cbr\u003eApplication\" }\n end\n\n subgraph Docker Container\n direction TB\n B@{ shape: rounded, label: \"πŸƒWorkflow\u003cbr\u003eApplication\" }\n end\n\n A \u003c--\u003e|action \u0026\u003cbr\u003eresponse| B\n B -...-\u003e |response| G\n G -...-\u003e |request| B\n\n subgraph Data Context\n D@{ shape: processes, label: \"Logs\" }\n E@{ shape: lin-cyl, label: \"Audit\u003cbr\u003eLogs\" }\n end\n\n subgraph Config Context\n F@{ shape: tag-rect, label: \"YAML\u003cbr\u003efiles\" }\n end\n\n A ---\u003e |push| H(Repo)\n H -.-\u003e |pull| F\n\n B \u003c--\u003e|disable \u0026\u003cbr\u003eread| F\n\n B \u003c--\u003e|read \u0026\u003cbr\u003ewrite| E\n\n B --\u003e|write| D\n\n D -.-\u003e|read| G\n E -.-\u003e|read| G\n```\n\n\u003e [!WARNING]\n\u003e _**Disclaimer**_: I inspire the dynamic YAML statement from the [**GitHub Action**](https://github.com/features/actions),\n\u003e and my experience of data framework configs pattern. :grimacing:\n\u003e\n\u003e Other workflow orchestration services that I interest and pick them to be\n\u003e this project inspiration:\n\u003e\n\u003e - [Google **Workflows**](https://cloud.google.com/workflows)\n\u003e - [AWS **Step Functions**](https://aws.amazon.com/step-functions/)\n\n## πŸ“¦ Installation\n\nThis project need `ddeutil` and `ddeutil-io` extension namespace packages to be\nthe base deps.\nIf you want to install this package with application add-ons, you should add\n`app` in installation;\n\n| Use-case | Install Optional | Support |\n|----------------|-------------------------|:-------:|\n| Python | `ddeutil-workflow` | βœ… |\n| FastAPI Server | `ddeutil-workflow[all]` | βœ… |\n\nCheck the version of the current workflow package:\n\n```shell\n$ pip install ddeutil-workflow\n$ workflow-cli version\n```\n\nInitial workflow project:\n\n```shell\n$ workflow-cli init\n```\n\n## πŸ“– Documentation\n\nFor comprehensive API documentation, examples, and best practices:\n\n- **[Full Documentation](https://ddeutils.github.io/ddeutil-workflow/)** - Complete user guide and API reference\n- **[Getting Started](https://ddeutils.github.io/ddeutil-workflow/getting-started/)** - Quick start guide\n- **[API Reference](https://ddeutils.github.io/ddeutil-workflow/api/workflow/)** - Detailed API documentation\n\n## 🎯 Usage\n\nThis is examples that use workflow file for running common Data Engineering\nuse-case.\n\n\u003e [!IMPORTANT]\n\u003e I recommend you to use the `call` stage for all actions that you want to do\n\u003e with workflow activity that you want to orchestrate. Because it is able to\n\u003e dynamic an input argument with the same call function that make you use less\n\u003e time to maintenance your data workflows.\n\n```yaml\nrun-py-local:\n\n # Validate model that use to parsing exists for template file\n type: Workflow\n on:\n # If workflow deploy to schedule, it will run every 5 minutes\n # with Asia/Bangkok timezone.\n - cronjob: '*/5 * * * *'\n timezone: \"Asia/Bangkok\"\n params:\n # Incoming execution parameters will validate with this type. It allows\n # to set default value or templating.\n source-extract: str\n run-date: datetime\n jobs:\n getting-api-data:\n runs-on:\n type: local\n stages:\n - name: \"Retrieve API Data\"\n id: retrieve-api\n uses: tasks/get-api-with-oauth-to-s3@requests\n with:\n # Arguments of source data that want to retrieve.\n method: post\n url: https://finances/open-data/currency-pairs/\n body:\n resource: ${{ params.source-extract }}\n\n # You can use filtering like Jinja template but this\n # package does not use it.\n filter: ${{ params.run-date | fmt(fmt='%Y%m%d') }}\n auth:\n type: bearer\n keys: ${API_ACCESS_REFRESH_TOKEN}\n\n # Arguments of target data that want to land.\n writing_mode: flatten\n aws:\n path: my-data/open-data/${{ params.source-extract }}\n\n # This Authentication code should implement with your custom call\n # function. The template allow you to use environment variable.\n access_client_id: ${AWS_ACCESS_CLIENT_ID}\n access_client_secret: ${AWS_ACCESS_CLIENT_SECRET}\n```\n\nBefore execute this workflow, you should implement caller function first.\n\n```text\nregistry-caller/\n ╰─ tasks.py\n```\n\nThis function will store as module that will import from `WORKFLOW_CORE_REGISTRY_CALLER`\nvalue (This config can override by extra parameters with `registry_caller` key).\n\n\u003e [!NOTE]\n\u003e You can use Pydantic Model as argument of your caller function. The core workflow\n\u003e engine will auto use the `model_validate` method before run your caller function.\n\n```python\nfrom ddeutil.workflow import Result, CallerSecret, tag\nfrom ddeutil.workflow.errors import StageError\nfrom pydantic import BaseModel\n\n\nclass AwsCredential(BaseModel):\n path: str\n access_client_id: str\n access_client_secret: CallerSecret\n\n\nclass RestAuth(BaseModel):\n type: str\n keys: CallerSecret\n\n\n@tag(\"requests\", alias=\"get-api-with-oauth-to-s3\")\ndef get_api_with_oauth_to_s3(\n method: str,\n url: str,\n body: dict[str, str],\n auth: RestAuth,\n writing_node: str,\n aws: AwsCredential,\n result: Result,\n) -\u003e dict[str, int]:\n \"\"\"Get the data from RestAPI via Authenticate with OAuth and then store to\n AWS S3 service.\n \"\"\"\n result.trace.info(\"[CALLER]: Start get data via RestAPI to S3.\")\n result.trace.info(f\"... {method}: {url}\")\n if method != \"post\":\n raise StageError(f\"RestAPI does not support for {method} action.\")\n # NOTE: If you want to use secret, you can use `auth.keys.get_secret_value()`.\n return {\"records\": 1000}\n```\n\nThe above workflow template is main executor pipeline that you want to do. If you\nwant to schedule this workflow, you want to dynamic its parameters change base on\nexecution time such as `run-date` should change base on that workflow running date.\n\n```python\nfrom ddeutil.workflow import Workflow, Result\n\nworkflow: Workflow = Workflow.from_conf('run-py-local')\nresult: Result = workflow.execute(\n params={\"source-extract\": \"USD-THB\", \"run-date\": \"2024-01-01\"}\n)\n```\n\n## :cookie: Configuration\n\nThe main configuration that use to dynamic changing this workflow engine for your\nobjective use environment variable only. If any configuration values do not set yet,\nit will use default value and do not raise any error to you.\n\n\u003e [!IMPORTANT]\n\u003e The config value that you will set on the environment should combine with\n\u003e prefix, component, and name which is `WORKFLOW_{component}_{name}` (Upper case).\n\n| Name | Component | Default | Description |\n|:----------------------------|:---------:|:---------------------------------------|:---------------------------------------------------------------------------------------|\n| **REGISTRY_CALLER** | CORE | `.` | List of importable string for the call stage. |\n| **REGISTRY_FILTER** | CORE | `ddeutil.workflow.reusables` | List of importable string for the filter template. |\n| **CONF_PATH** | CORE | `./conf` | The config path that keep all template `.yaml` files. |\n| **STAGE_DEFAULT_ID** | CORE | `false` | A flag that enable default stage ID that use for catch an execution output. |\n| **GENERATE_ID_SIMPLE_MODE** | CORE | `true` | A flog that enable generating ID with `md5` algorithm. |\n| **DEBUG_MODE** | LOG | `true` | A flag that enable logging with debug level mode. |\n| **TIMEZONE** | LOG | `Asia/Bangkok` | A Timezone string value that will pass to `ZoneInfo` object. |\n| **TRACE_HANDLERS** | LOG | `[{\"type\": \"console\"}]` | A Json string of list of trace handler config data that use to emit log message. |\n| **AUDIT_CONF** | LOG | `{\"type\": \"file\", \"path\": \"./audits\"}` | A Json string of audit config data that use to write audit metrix. |\n| **AUDIT_ENABLE_WRITE** | LOG | `true` | A flag that enable writing audit log after end execution in the workflow release step. |\n\n## :rocket: Deployment\n\nThis package able to run as an application service for receive manual trigger\nfrom any node via RestAPI with the FastAPI package.\n\n### API Server\n\nThis server use FastAPI package to be the base application.\n\n```shell\n(.venv) $ workflow-cli api --host 127.0.0.1 --port 80\n```\n\n\u003e [!NOTE]\n\u003e If this package already deploy, it is able to use multiprocess;\n\u003e `$ workflow-cli api --host 127.0.0.1 --port 80 --workers 4`\n\n### Docker Container\n\nBuild a Docker container from this package.\n\n```shell\n$ docker pull ghcr.io/ddeutils/ddeutil-workflow:latest\n$ docker run --rm ghcr.io/ddeutils/ddeutil-workflow:latest ddeutil-worker\n```\n\n## :speech_balloon: Contribute\n\nI do not think this project will go around the world because it has specific propose,\nand you can create by your coding without this project dependency for long term\nsolution. So, on this time, you can open [the GitHub issue on this project :raised_hands:](https://github.com/ddeutils/ddeutil-workflow/issues)\nfor fix bug or request new feature if you want it.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fddeutils%2Fddeutil-workflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fddeutils%2Fddeutil-workflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fddeutils%2Fddeutil-workflow/lists"}