{"id":17218599,"url":"https://github.com/jwulf/zeebe-map-reduce","last_synced_at":"2026-01-29T03:12:38.142Z","repository":{"id":43929806,"uuid":"184373655","full_name":"jwulf/zeebe-map-reduce","owner":"jwulf","description":"Reusable Map/Reduce workflow in Zeebe","archived":false,"fork":false,"pushed_at":"2021-08-10T23:56:22.000Z","size":886,"stargazers_count":1,"open_issues_count":4,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-12T13:17:52.125Z","etag":null,"topics":["bpmn","data-processing","map-reduce","microservices","workflow-engine","zeebe"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jwulf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-05-01T05:35:38.000Z","updated_at":"2022-07-08T07:51:27.000Z","dependencies_parsed_at":"2022-09-06T11:31:29.788Z","dependency_job_id":null,"html_url":"https://github.com/jwulf/zeebe-map-reduce","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwulf%2Fzeebe-map-reduce","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwulf%2Fzeebe-map-reduce/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwulf%2Fzeebe-map-reduce/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwulf%2Fzeebe-map-reduce/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jwulf","download_url":"https://codeload.github.com/jwulf/zeebe-map-reduce/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247451466,"owners_count":20940946,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bpmn","data-processing","map-reduce","microservices","workflow-engine","zeebe"],"created_at":"2024-10-15T03:47:31.930Z","updated_at":"2026-01-29T03:12:38.115Z","avatar_url":"https://github.com/jwulf.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Zeebe Map/Reduce\n\nReusable Map/Reduce workflow in [Zeebe](https://zeebe.io) - a workflow engine for orchestrating microservices - as a Node module.\n\nNot meant to be used as-is in-production, but:\n\n-   Gives you an idea of how to orchestrate map/reduce tasks with Zeebe. The BPMN diagram in `bpmn/map-reduce.bpmn` implements a map/reduce workflow.\n-   Gives you an idea of how to create an API / library of Zeebe functionality that can be composed/reused at the Application level. The `ZeebeMapProcess` class in `src/lib/ZeebeMapProcess` can map any workflow over any set of inputs.\n-   Demonstrates (optional) strong typing of job variables and messages using Generics with the 0.17-compatible version of the Zeebe TypeScript library.\n\n## Prerequisites\n\n-   Node\n-   ts-node\n-   Zeebe 0.20.0 (Recommended: [Zeebe Docker](https://github.com/zeebe-io/zeebe-docker-compose))\n\n```\nnpm i -g ts-node\n```\n\n## To run the example\n\n-   Start a Zeebe Broker ([Docker configurations here](https://github.com/jwulf/zeebe-operate-docker))\n-   Install dependencies:\n\n```bash\nnpm i\n```\n\n-   Start a single mapFunction worker:\n\n```bash\nts-node typescript/src/example/simple/MapFunctionWorker.ts\n```\n\n-   In another window, start the map workflow:\n\n```bash\nts-node typescript/src/example/simple/MapFunction.ts\n```\n\n-   You can start additional mapFunction workers in other terminals to observe the effect of scaling up workers on the overall speed of the process.\n\n![](img/running-example.png)\n\n## Data-processing pipeline using Map/Reduce\n\nThis Zeebe workflow (`bpmn/map-reduce.bpmn`) implements a generic map/reduce pattern:\n\n![](img/map-reduce-flow.png)\n\nFrom an application-programming perspective, the `Split Inputs/Map` task takes an input with this shape:\n\n```javascript\n{\n    elements: T[],\n    mapFunctionId: string\n}\n```\n\nIt has the same signature as the [lodash `map` method](https://dustinpfister.github.io/2018/02/02/lodash_map/):\n\n```javascript\n_.map(elements, mapFunction)\n```\n\nIt takes the elements to map over, and a \"function\" to map over them.\n\nIn this case, the \"function\" that it will map over the elements is another Zeebe workflow. We pass in the id for this workflow.\n\nUsing a separate workflow for this achieves three outcomes:\n\n-   It allows the workers that service the mapFunction workflow to be scaled.\n-   It allows us to separate the wiring mechanics of map/reduce from the specific transformation in the workflow diagrams.\n-   It allows us to encapsulate and reuse this generic map/reduce functionality at the application-programming level.\n\nThe `mapFunction\u003cT\u003e` workflow needs to consume an input that has this shape:\n\n```javascript\n{\n    element: T,\n    correlationKey: string\n}\n```\n\nThis implements the lambda function provided to the `map` method in many languages, for example:\n\n```typescript\nelements.map(element =\u003e mapFunction(element))\n```\n\nIt is inspired by [Kotlin's approach](https://medium.com/@elye.project/kotlin-for-loop-vs-foreach-7eb594960333), where the input parameter signature defaults to `it`:\n\n```kotlin\n(0..10).forEach { println(it) }\n```\n\nIn the case of Kotlin, you get a closure with a parameter `it`. It's an opinionated shortcut.\n\nIn this case, rather than `it`, the input parameter name is set to `element`.\n\nYour mapFunction workflow will be called for each `element` in the `elements` array of inputs. It needs to consume the `element` variable for the workflow, and the `correlationKey` is used to publish the result back to the main map workflow, where it will be collected.\n\nThis repo contains an example `mapFunction` implementation in `bpmn/do-processing.bpmn`.\n\n![](img/do-processing-flow.png)\n\nThere is a `mapFunction\u003cstring\u003e` worker implementation for this workflow in `example/simple/MapFunctionWorker.ts`. This example implementation takes in a string, reverses it, and converts it to uppercase. It also delays for 2 - 15 seconds to simulate a longer running process.\n\nThe `mapFunction` workflow sends the result of its operation back to the main workflow by publishing a message using the `correlationKey` that was passed in, with the variable `element` set to the output of its transformation.\n\nThe main workflow is now in a loop, receiving messages from one or more workers that are servicing the tasks in the `mapFunction` workflow instances.\n\nThe variables in a message are merged into the variables of the main workflow, and the `Reduce Results` task is executed. In practice, this means that the `Reduce Results` task is executed once for each completed `mapFunction` workflow, with the result of the workflow in the `element` variable.\n\nThe main workflow has an `accumulator` variable, which is of type `T[]`. As each message comes in from the mapFunction workers, the result of their work - which appears in the `element` variable - is pushed into `accumulator`.\n\nThe reducer then does a check to see if the `accumulator` variable has the same number of members as the `elements` variable (this is the input set). If it does, the reducer sets the variable `done` to true.\n\nAfter the reducer updates the accumulator, the workflow either goes back to the message catch to process another mapFunction result, or if `done` is true, moves to the `output` task.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjwulf%2Fzeebe-map-reduce","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjwulf%2Fzeebe-map-reduce","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjwulf%2Fzeebe-map-reduce/lists"}