{"id":19491665,"url":"https://github.com/rudderlabs/rudder-workflow-engine","last_synced_at":"2025-04-06T10:11:04.252Z","repository":{"id":131292662,"uuid":"538396035","full_name":"rudderlabs/rudder-workflow-engine","owner":"rudderlabs","description":"A generic embeddable workflow execution engine library","archived":false,"fork":false,"pushed_at":"2025-03-24T06:04:00.000Z","size":1608,"stargazers_count":37,"open_issues_count":2,"forks_count":3,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-30T09:09:10.763Z","etag":null,"topics":["automation","embedded","json","jsonata","nodejs","templates","workflow-engine"],"latest_commit_sha":null,"homepage":"https://transformers-workflow-engine.rudderstack.com/#/workflow-engine","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rudderlabs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-19T08:06:00.000Z","updated_at":"2025-02-13T06:38:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"ebfba975-2b23-4d2a-89de-c8e890e2f3e5","html_url":"https://github.com/rudderlabs/rudder-workflow-engine","commit_stats":null,"previous_names":[],"tags_count":77,"template":false,"template_full_name":"rudderlabs/rudder-repo-template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rudderlabs%2Frudder-workflow-engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rudderlabs%2Frudder-workflow-engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rudderlabs%2Frudder-workflow-engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rudderlabs%2Frudder-workflow-engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rudderlabs","download_url":"https://codeload.github.com/rudderlabs/rudder-workflow-engine/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247464220,"owners_count":20942970,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","embedded","json","jsonata","nodejs","templates","workflow-engine"],"created_at":"2024-11-10T21:17:41.801Z","updated_at":"2025-04-06T10:11:04.232Z","avatar_url":"https://github.com/rudderlabs.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![codecov](https://codecov.io/gh/rudderlabs/rudder-workflow-engine/branch/main/graph/badge.svg?token=M6N01L8OJS)](https://codecov.io/gh/rudderlabs/rudder-workflow-engine)\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://rudderstack.com/\"\u003e\n    \u003cimg src=\"https://user-images.githubusercontent.com/59817155/121357083-1c571300-c94f-11eb-8cc7-ce6df13855c9.png\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003cb\u003eThe Customer Data Platform for Developers\u003c/b\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003e\n    \u003ca href=\"https://rudderstack.com\"\u003eWebsite\u003c/a\u003e\n    ·\n    \u003ca href=\"\"\u003eDocumentation\u003c/a\u003e\n    ·\n    \u003ca href=\"https://rudderstack.com/join-rudderstack-slack-community\"\u003eCommunity Slack\u003c/a\u003e\n  \u003c/b\u003e\n\u003c/p\u003e\n\n---\n\n# rudder-workflow-engine\n\n## Overview\n\nIn the [rudder-transformer](https://github.com/rudderlabs/rudder-transformer/) service, we're transforming customer event data into the format required for different destinations. To make this process clearer and easier to maintain, we needed to divide it into logical steps. So, we developed this workflow engine to meet our needs.\n\n- Validation\n  - Event validation\n  - Destination config validation\n- Source to destination data mapping\n- Enriching data with destination API calls\n- Handling different types of events\n  - Track, Identify, Page, etc.\n  - Custom categories:\n    - Product Viewed\n    - Product Purchased\n- Multiplexing\n- Batching\n- Response building.\n\nCurrently, most steps are implemented using Javascript code, which provides the most flexibility. Still, it is getting difficult to maintain, understand, debug, test, and develop in a standardized way. To bring standardization, we are building a workflow engine that is config driven to provide improved readability, testability, reusability, and speed of development.\n\nSince we want to express the transformation of the logic using easy to read and write template based language. We support following template languages:\n\n- [JSONata](https://github.com/jsonata-js/jsonata)\n- [JsonTemplate](https://github.com/rudderlabs/rudder-json-template-engine)\n- Easily extendable to [more template languages](https://github.com/rudderlabs/rudder-workflow-engine/tree/main/src/steps/base/simple/executors/template).\n  **Workflow Example using Jsonata:**\n\n```yaml\ntemplateType: Jsonata\nsteps:\n  - name: unsupported\n    condition: $not(op in [\"+\", \"-\", \"*\", \"/\"])\n    template: |\n      $doThrow(\"unsupported operation\")\n  - name: add\n    description: Do addition\n    condition: op = \"+\"\n    template: |\n      ( a + b )\n  - name: subtract\n    description: Do subtraction\n    condition: op = \"-\"\n    template: |\n      ( a - b )\n  - name: multiply\n    description: Do multiplication\n    condition: op = \"*\"\n    template: |\n      ( a * b )\n  - name: divide\n    description: Do division\n    condition: op = \"/\"\n    template: |\n      ( \n        $assert( b != 0, \"division by zero is not allowed\");\n        a / b \n      )\n```\n\n## Getting started\n\n- `npm install @rudderstack/workflow-engine`\n\n```js\nimport { WorkflowEngineFactory } from '@rudderstack/workflow-engine';\nconst workflowEngine = WorkflowEngineFactory.createFromFilePath('workflow.yaml', options);\nworkflowEngine.execute(input);\n```\n\n## Features\n\n### Config Driven\n\nUsers should be able to express the destination transformation logic as a series of steps in a YAML file as a workflow. Steps can be written as template base languages.\n\n### Bindings\n\nSupports importing of external functions and data using bindings.\n\n#### Workflow Bindings\n\n- Bindings are similar to imports, which allow importing of externally defined functions and data to the workflow.\n- **Types**\n  - Type 1: Import a specific field from a file.\n    ```yaml\n    name: EventType\n    path: ./config\n    ```\n    - **EventType** is defined in **./config** file then it will be imported as **$EventType**\n  - Type 2: Import everything from a file as something.\n    ```yaml\n    name: MappingData\n    path: ./mapping\n    exportAll: true\n    ```\n    - Everything from **./mapping** file will be imported to the variable **MappingData**\n    - If **something1** and **something2** are defined in **./mapping** then we need to access them using **$MappingData.something1** and **$MappingData.something2**\n  - Type 3: Import everything as it is defined in the file\n    ```yaml\n    path: ./utils\n    ```\n    - Everything from **./utils** file will be imported with the same names.\n    - If **something1** and **something2** are defined in **./utils** then we need to access them using **$something1** and **$something2**\n- Full example:\n  ```yaml\n  bindings:\n    - name: EventType\n      path: ./config\n    - name: MappingData\n      path: ./mapping\n      exportAll: true\n    - path: ./utils\n  ```\n- These are user-specified bindings while defining the workflow.\n\n#### Platform bindings\n\n- The platform provides these bindings, which can be used directly without defining them in the **bindings** block.\n- We will soon release detailed documentation on these bindings.\n\n#### Execution bindings\n\n- **$outputs:** Provides access to the outputs of the previous steps executed before the current step.\n  ```yaml\n  steps:\n    - name: step1\n      template: |\n        {\n          \"a\": something\n        }\n    - name: step2\n      template: |\n      {\n        \"b\": $doSomething($outputs.step1.a)\n      }\n  ```\n  - **Step2** uses the output of **step1.**\n  - Workflow Engine automatically bindings step outputs to the **$outputs** variable.\n- **$setContext:** It is a function to store any data in $context and use it later. **$outputs** are read-only variables for users to refer to the previous step outputs, so we can’t use them to pass a modifiable result. So if we want to update the same variable in multiple steps, then **$setContext\\*\\* should be used.\n  - Example:\n    ```yaml\n    steps:\n      - name: setAForCase1\n        condition: $isCase1(message)\n        template: |\n          $setContext(\"a\", something1)\n      - name: setAForCase2\n        condition: $isCase2(message)\n        template: |\n          $setContext(\"a\", something2)\n      - name: updateA\n        template: |\n          $setContext(\"a\", $updateA($context.a))\n      - name: useA\n        template: |\n          $doSomething($context.a)\n    ```\n    - In this example, we update the variable repeatedly in several steps, so it is impossible to use **$outputs.**\n    - A practical scenario for this feature is: that we want to populate an object differently based on some conditions and later use it.\n  - **$context:** To access variables set using **$setContext** function. Please refer to the above example for clarity.\n\n### Steps\n\n- Steps are the main execution blocks of the workflow.\n- Steps must contain a **name** to track outputs.\n- Steps can contain an optional **description** field to describe the details.\n- The step can contain an optional **condition** field to execute only if the condition is satisfied.\n- The step can contain an optional **inputTemplate** field to customize the input, which will be passed while executing the step.\n- There are two different types of steps supported:\n  - SimpleStep\n  - WorkflowStep\n\n### Conditions\n\n- A step in a workflow can mention an optional condition so that it gets executed only when the condition is satisfied.\n- Condition is also a [Jsonata](https://docs.jsonata.org/overview.html) code.\n  ```yaml\n  steps:\n    - name: commonValidation\n      template: |\n        ( common validations for events )\n    - name: ValidateInputOfTrackEvent\n      condition: message.type = EventType.Track\n      template: |\n        ( some validations specific to track events)\n  ```\n\n### InputTemplate\\*\\*\n\n- By default, all steps receive the same input as the workflow input, but when we want to modify the input before executing the step, we can use this feature.\n  ```yaml\n  steps:\n    - name: step1\n      (some logic ...)\n    - name: step2\n      inputTemplate: |\n        (customize the input)\n    - name: step3\n      (some logic ...)\n  ```\n  - In the above example: step1 and step3 will be executed with the workflow’s input, but the step2 receives custom input as defined in the **inputTemplate**\n\n### ContextTemplate\n\n- By default, all steps receive the current context, but we can use this feature when we want to modify the context before executing the step. This is useful when using external workflows, workflow steps, or template paths.\n  ```yaml\n  steps:\n    - name: step1\n      (some logic to prepareContext)\n    - name: step2\n      contextTemplate: |\n        (customize the context for step2)\n      (some logic ...)\n    - name: step3\n      (some logic ...)\n  ```\n  - In the above example: step 3 will execute with the context prepared in step 1, but step 2 receives custom context as defined in the **contextTemplate.**\n\n### LoopOverInput\n\n- We can use this feature when the input is an array, and we want to execute the step logic for each element independently.\n- This is mainly used for batch processing, and we report failed and successful executions without failing the step if an error occurs while processing a particular step.\n  ```yaml\n  name: executeForEach\n  loopOverInput: true\n  template: |\n    ( do something )\n  ```\n  - If the input for the step is [e1, e2, e3], then the step will be executed for all elements independently, and imagine that it failed for e1 and succeeded for e2 and e3 then, the overall step output will be the following:\n    ```json\n    [\n      {\n        \"error\": someErrorForE1\n      },\n      {\n        \"output\": someOutputForE2\n      },\n      {\n        \"output\": someOutputForE3\n      }\n    ]\n    ```\n\n### OnComplete\n\n- When the step is completed, the next step will be executed, but if we want to exit the workflow with the output of a particular step, then we can use this.\n- This feature should be used only in a conditional step.\n- Example 1: Avoid reprocessing, so return without modifying the input message.\n  ```yaml\n    steps:\n      - name: checkIfProcessed\n        condition: message.processed = true\n        template: |\n          message\n        **onComplete: return**\n      - name: processMessage\n        template: |\n          (...)\n  ```\n  - In the above example, we don’t want to reprocess messages, so we need to return them immediately if they are already processed.\n- Example 2: Return early after processing the input message.\n\n  ```yaml\n  steps:\n    - name: step1\n      template: |\n        (doSomeProcessing)\n    - name: **step2**\n      condition: someCondition\n      template: |\n        (doSomeProcessing)\n      onComplete: return\n    - name: step3\n      template: |\n        (doSomeProcessing)\n  ```\n\n  - In this example, we want to **return early** after successfully processing the message in **step2** since this step is conditional, and if the condition is not satisfied, then **step3** will be executed.\n\n### OnError\n\n- By default, if any step fails, then the entire workflow fails but if the step uses **OnError: continue** setting, then the workflow will ignore the error and continue with execution.\n  ```yaml\n  steps:\n    - name: step1\n      template: |\n        (doSomeProcessing)\n    - name: **step2**\n      template: |\n        (doSomeProcessing)\n      onError: continue\n    - name: step3\n      template: |\n        (doSomeProcessing)\n  ```\n  - In the above example, if any error occurs in either step1 or step3, the workflow will exit immediately, but when step2 fails, the workflow ignores the error and continues to execute step3.\n\n### Custom Workflow Executor\n\nWhen you may want finer control on how workflow steps needs to executed by workflow engine then you can use this feature. You need to implement [WorkflowExecutor](./src/workflow/types.ts#L65) and use it in following ways to override the default workflow executor.\n\n#### Specify directly in YAML\n\n```yaml\nexecutor: myCustomWorkflowExecutor\nbindings:\n  - name: myCustomWorkflowExecutor\n    path: ./custom_executor\nsteps:\n  - name: step1\n    template: |\n      doSomething\n```\n\nRefer this [example](./test/scenarios/custom_executor/workflow.yaml) for more details.\n\n#### Specify using workflow options\n\n```ts\nWorkflowEngineFactory.createFromFilePath('workflow.yaml', {\n  executor: myCustomWorkflowExecutor,\n});\n```\n\n### Custom Bindings Provider\n\nWhen you want to implement custom logic to resolve the bindings then you can use this feature. You need to implement [WorkflowBindingProvider](./src/workflow/types.ts#L69).\n\n```ts\nWorkflowEngineFactory.createFromFilePath('workflow.yaml', {\n  bindingProvider: myCustomBindingsProvider,\n});\n```\n\n## Steps\n\n### Simple Step\n\n- Simple step is the basic unit of execution in the workflow.\n- A simple step can be a **function** that is defined in the **Bindings**.\n\n  ```yaml\n  bindings:\n    - name: **processTrackEvent\n      path: ./transform # actual file name is transform.js**\n  steps:\n    - name: processTrackEvent\n      functionName: processTrackEvent\n  ```\n\n  - We can omit **.js** extension while defining the bindings.\n  - **processTrackEvent** must have the following definition.\n\n  ```ts\n  (input: any, bindings: Record\u003cstring, any\u003e) =\u003e {\n    error?: any,\n    output?: any\n  }\n  ```\n\n- A simple step can be a JSONata template.\n  ```yaml\n  name: processTrackEvent\n  template: |\n    (JSONata template to process track events)**\n  ```\n  - The template also can be imported from the file path.\n    ```yaml\n    name: processTrackEvent\n    templatePath: ./trackTemplate.yaml\n    ```\n- We can use an **external workflow** in a simple step.\n  ```yaml\n  steps:\n    - name: prepareContext\n      template: $setContext(\"batchMode\", true)\n    - name: transform\n      **externalWorkflow:\n        path: ./pinterest_tag_single_workflow.yaml**\n      loopOverInput: true\n  ```\n  - We are reusing the single event workflow in the batch events transformation workflow.\n  - The **external workflow** will be executed as a black box, so we can only access the final output of the workflow but not the individual outputs of steps.\n  - The external workflow is executed with **step input** and **context** of the original workflow.\n  - The context of the parent workflow is passed to the child workflow (**externalWorkflow**) but not vice-versa. This is helpful to customize the child workflow execution based on where it is used.\n  - The **external workflow** doesn’t have access to the parent workflow **outputs.**\n\n### Workflow Step\n\n- Series of **simple** steps.\n  ```yaml\n  steps:\n    - name: category\n      template: |\n        (compute category)\n    - name: ecom\n      condition: $outputs.category = \"ecom\"\n      steps:\n        - name: validateInput\n          description: Common validation for all ECom pages\n          template: |\n            (assert everything is fine)\n        - name: page\n          template: |\n            (compute page using $outputs.category)\n        - name: processSearchPage\n          condition: $outputs.ecom.page = \"search\"\n          template: |\n            (search page template)\n        - name: processDetailPage\n          condition: $outputs.ecom.page = \"detail\"\n          template: |\n            (detail page template)\n        - name: processCartPage\n          condition: $outputs.ecom.page = \"cart\"\n          template: |\n            (cart page template)\n  ```\n  - We can access **outputs** of previous steps normally like **$outputs.category.**\n  - To access outputs of the child steps of the workflowStep, we need to use **$outputs.workflowStepName.childStepName**, for example: $outputs.ecom.page.\n    - The outputs of the child steps are not available outside the workflow step.\n    - The last successfully executed child step’s output will become the output of the workflow step, and we can only access that outside the workflow step as **$output**.**workflowStepName,** for example, $output.ecom.\n  - Currently, we don’t support nested workflow steps.\n- Workflow Step can be imported from a file.\n  ```yaml\n  steps:\n    - name: processECommerace\n      workflowStepPath: ./ecomWorkflow.yaml\n  ```\n- Supports additional **Bindings**\n  ```yaml\n  bindings:\n    - name: commonBinding\n      path: ./bindings\n  steps:\n    - name: processECommerace\n      bindings:\n        - name: stepBinding\n          path: ./workflow_step_bindings\n      steps:\n        - name: validateInput\n          description: Common validation for all ECom pages\n          template: |\n            (assert with $commonBinding)\n        - name: page\n          template: |\n            (compute page using $workflowBinding)\n        - name: processSearchPage\n          condition: $outputs.ecom.page = \"search\"\n          template: |\n            (search page template)\n  ```\n  - In the above example: **processECommerace** step is the workflow step and importing additional bindings. Both workflow’s (**commonBinding**) and step’s (**stepBinding**) bindings are available to the workflow step.\n\n### Batch Step\n\nThis helps to batch the inputs using filter and by length or size. We can also define our own batching by implementing [**BatchExecutor**](./src/steps/types.ts#L118) interface.\n\n#### Syntax\n\n```yaml\nsteps:\n  - name: batchStep\n    type: batch\n    batches:\n      - key: heroes\n        filter: .type === \"hero\"\n        length: 5\n      - key: villains\n        filter: .type === \"villain\"\n        size: 100\n```\n\nHere we are using keys (heroes or villains) to indicate batches and these will be reflected in the output for further processing.\n\n#### Custom Batch Executor\n\nRefer this [example](./test/scenarios/batch_step/using_executor.yaml).\n\n### Custom Step\n\nWhen you want to bring your own steps to workflows then you use this feature.\n\n#### Using Executor\n\nWhen your custom step doesn't require initialization specific to workflow then you can directly a provide executor instance in bindings by implementing [CustomStepExecutor](./src/steps/types.ts#130).\n\nRefer this [example](./test/scenarios/custom_step/executor.yaml).\n\n#### Using Provider\n\nWhen your custom step requires initialization with custom params then you can pass a provider in bindings by implementing [CustomStepExecutorProvider](./src/steps/types.ts#133).\n\nRefer this [example](./test/scenarios/custom_step/provider.yaml).\n\n## Testing\n\n#### Test Scenarios using Jest\n\n- `npm run jest:scenarios --  --scenarios=\u003ccomma separate scenarios\u003e`\n- Example: `npm run jest:scenarios --  --scenarios=basic_workflow,to_array`\n\n#### Manually Test Scenario\n\n- `npm run test:scenario -- -s \u003cscenario_folder\u003e  -i \u003ctest_case_index_from_data.json\u003e`\n- Example: `npm run test:scenario -- -s outputs -i 1`\n- Note: It just run the test case and produces results but won't run any validations of the results.\n\n## Contribute\n\nWe would love to see you contribute to RudderStack. Get more information on how to contribute [**here**](CONTRIBUTING.md).\n\n## License\n\nThe RudderStack `rudder-workflow-engine` is released under the [**MIT License**](https://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frudderlabs%2Frudder-workflow-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frudderlabs%2Frudder-workflow-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frudderlabs%2Frudder-workflow-engine/lists"}