{"id":36703501,"url":"https://github.com/emaarco/distributed-horcruxes","last_synced_at":"2026-01-12T11:40:29.518Z","repository":{"id":268513130,"uuid":"894394922","full_name":"emaarco/distributed-horcruxes","owner":"emaarco","description":"Practical solutions for tackling distributed transaction challenges with Zeebe and Spring Boot 🔋","archived":false,"fork":false,"pushed_at":"2026-01-02T11:22:49.000Z","size":486,"stargazers_count":6,"open_issues_count":3,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-08T12:42:18.440Z","etag":null,"topics":["distributed-transactions","kotlin","spring-boot","zeebe"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/emaarco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-11-26T09:30:36.000Z","updated_at":"2026-01-02T11:22:52.000Z","dependencies_parsed_at":"2025-01-22T16:21:02.790Z","dependency_job_id":"b3402ae8-346c-4846-84cc-01bca8a7d910","html_url":"https://github.com/emaarco/distributed-horcruxes","commit_stats":null,"previous_names":["emaarco/distributed-horcruxes"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/emaarco/distributed-horcruxes","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emaarco%2Fdistributed-horcruxes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emaarco%2Fdistributed-horcruxes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emaarco%2Fdistributed-horcruxes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emaarco%2Fdistributed-horcruxes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/emaarco","download_url":"https://codeload.github.com/emaarco/distributed-horcruxes/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emaarco%2Fdistributed-horcruxes/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28338972,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T10:58:46.209Z","status":"ssl_error","status_checked_at":"2026-01-12T10:58:42.742Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-transactions","kotlin","spring-boot","zeebe"],"created_at":"2026-01-12T11:40:28.650Z","updated_at":"2026-01-12T11:40:29.508Z","avatar_url":"https://github.com/emaarco.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🧙‍♂️ Distributed Horcruxes\n\n\u003e 🚧 This repository is subject of constant change. It deals with a problem that has not yet been examined and solved in\n\u003e depth in the context of remote process engines (like Zeebe).\n\n## **Introduction** 🗂\n\nDistributed transactions are a common challenge in distributed systems.\nWhen multiple services or systems need to coordinate to ensure consistent data,\nfailures—like communication issues or incomplete updates—can cause problems.\n\nMuch like horcruxes, distributed transactions are tricky to manage,\ndangerous if left unchecked, and can create chaos in your system if not handled properly.\n\nThis is particularly relevant when working with external systems,\nsuch as process engines like Zeebe or Camunda 7, where state is managed independently.\nIf proper strategies aren’t in place, workflows or interactions may lead to inconsistencies\nbetween the process engine and your application’s database.\n\nThis repository aims to provide practical examples and proven patterns\nto handle these distributed transaction problems effectively in a Spring Boot and Zeebe environment.\n\n## **The Distributed Transaction Problem** 🕵️\n\nDistributed transactions are a generic problem in software architecture that occurs when a single operation spans\nmultiple systems, such as:\n\n- A **database** for storing application data.\n- A **process engine** like Zeebe for orchestrating workflows.\n- Other systems like **APIs** or **message brokers**.\n\nProblems arise when we can’t guarantee that all systems succeed or fail together. This can lead to:\n\n- **Inconsistent states**: Some systems finish their operations, while others fail.\n- **Duplicate actions**: Systems retry failed tasks, causing duplicate operations.\n- **Data conflicts**: Tasks execute out of order or with incomplete data.\n\n## **The Challenge with Zeebe** ⚙️\n\nZeebe, as a distributed system itself, is designed for high availability and scalability.\nThis means it manages its own state independently from your application's database,\ncreating a boundary between two systems that must remain synchronized.\n\nWhen coordinating these two independent systems, various challenges emerge.\nOne of the most common issues is a **timing problem**: Zeebe assumes that tasks can start as soon as they are triggered,\nbut your database transaction might not be committed yet—or might fail during the commit.\n\nConsider this simple scenario:\n\n1. You save data to the database and immediately notify Zeebe to start a process.\n2. If the database transaction fails, Zeebe has already started tasks based on incomplete or incorrect data.\n\nWithout proper strategies, this can cause:\n\n- Tasks to fail unexpectedly.\n- The database and process engine states to drift apart.\n- Retrying the same operation to produce duplicates.\n\n\u003e 💡 **Want to understand these challenges in more detail?**\u003cbr/\u003e\n\u003e Check out the [detailed breakdown of distributed transaction challenges](CHALLENGES.md),\n\u003e which includes examples, diagrams, and a reference architecture to explain these problems in depth.\n\n## **How Do We Solve This?** 🛠\n\nDistributed transactions are a common problem in software architecture,\nand luckily, there are some **well-known and tested solutions** to address them.\nThese solutions are applicable across many distributed systems and can also be adapted to work with Zeebe.\n\nHere are some of the most effective patterns, we want to explore:\n\n### **1. After-Transaction Hook** ✅\n\nTrigger Zeebe only after the database transaction is successfully committed.\nThis avoids notifying Zeebe with incomplete or uncommitted data.\n\n### **2. Outbox Pattern** 📦\n\nSave Zeebe messages in an \"outbox\" table as part of the same database transaction.\nA scheduler or worker then reliably sends these messages to Zeebe after the transaction is complete.\n\n### **3. Idempotency Pattern** 🔁\n\nTrack completed operations to prevent duplicate processing when Zeebe retries job workers.\nUses a database table to record which operations have been completed.\n\n### **4. Saga Pattern** ⏪\n\nHandle distributed transaction rollbacks using BPMN compensation events.\nWhen a later step fails, compensation handlers automatically undo previously completed operations.\n\nThese patterns are widely used in distributed systems\nand provide different trade-offs between simplicity, reliability, and performance.\n\n## **Examples in This Repository** 📚\n\nThis repository contains examples that demonstrate both the problem and proven solutions:\n\n- ⚠️ [**Base Scenario**](./examples/base-scenario/README.md):\n  The naive implementation that demonstrates what goes wrong without proper transaction handling.\n- ✅ [**After-Transaction Hook**](./examples/after-transaction/README.md):\n  Ensuring Zeebe interactions occur only after the transaction commits.\n- 📦 [**Outbox Pattern**](./examples/outbox-pattern/README.md):\n  Using a database outbox and scheduler for reliable message delivery.\n- 🔁 [**Idempotency Pattern**](./examples/idempotency-pattern/README.md):\n  Preventing duplicate processing from Zeebe's at-least-once delivery semantics.\n- ⏪ [**Saga Pattern**](./examples/saga-pattern/README.md):\n  Handling distributed transaction rollbacks using BPMN compensation events.\n\n## **Setup** ⚙️\n\nGetting started with the examples is simple! Follow these steps:\n\n**1: Start the Infrastructure**:  \nNavigate to the `stack` folder and bring up the infrastructure (Zeebe, Operate, etc.) using Docker Compose:\n\n   ```bash\n   cd stack\n   docker-compose up\n   ```\n\n**2: Run the Example**:  \nGo to the folder of the example you want to try and start the application by running the `ExampleApplication.kt` main\nclass.\nEach example is a standard Spring Boot application,\nso you can run it using your preferred IDE or command line.\n\n**3: Interact with the Process**:  \nEach example uses the same newsletter subscription process.\nTo interact with the process, you need to send requests to the REST API provided by the example services.\n\n- To make this easier, there are predefined API call files located in the [bruno](bruno) directory.\n- If you use [Bruno](https://www.usebruno.com/), simply open the folder and execute these files.\n- Alternatively, you can use any other tool like curl or Postman to send the requests manually.\n\n**4: Monitor the Processes**:  \nOnce the infrastructure is running, you can monitor and debug workflows using **Operate**\nat [http://localhost:9091/operate](http://localhost:9091/operate).\nThe credentials are `demo/demo`.\n\n## **Contribute to This Project** 🤝\n\nDistributed transactions are tricky, but together we can solve them!\nIf you have ideas, improvements, or new examples,\nfeel free to contribute by opening a pull request or issue.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femaarco%2Fdistributed-horcruxes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Femaarco%2Fdistributed-horcruxes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femaarco%2Fdistributed-horcruxes/lists"}