{"id":24625133,"url":"https://github.com/ricall/kafkasaga","last_synced_at":"2026-04-07T07:45:31.660Z","repository":{"id":208728018,"uuid":"347325251","full_name":"ricall/KafkaSaga","owner":"ricall","description":"Simple microservice that uses Kafka + Saga Pattern to provide complicated message processing logic using simple actions.","archived":false,"fork":false,"pushed_at":"2021-03-26T12:42:54.000Z","size":195,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-03T16:31:23.954Z","etag":null,"topics":["docker","docker-compose","draw-io","gradle","java","kafka","lombok","postgres","saga-pattern","spring","spring-boot","yaml-configuration"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ricall.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-03-13T09:19:48.000Z","updated_at":"2023-03-14T17:10:00.000Z","dependencies_parsed_at":"2023-11-23T01:39:10.897Z","dependency_job_id":"fd250132-e114-4a2b-9d6e-035e8fbcb495","html_url":"https://github.com/ricall/KafkaSaga","commit_stats":null,"previous_names":["ricall/kafkasaga"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ricall/KafkaSaga","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ricall%2FKafkaSaga","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ricall%2FKafkaSaga/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ricall%2FKafkaSaga/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ricall%2FKafkaSaga/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ricall","download_url":"https://codeload.github.com/ricall/KafkaSaga/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ricall%2FKafkaSaga/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31504897,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-07T03:10:19.677Z","status":"ssl_error","status_checked_at":"2026-04-07T03:10:13.982Z","response_time":105,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","draw-io","gradle","java","kafka","lombok","postgres","saga-pattern","spring","spring-boot","yaml-configuration"],"created_at":"2025-01-25T04:13:05.956Z","updated_at":"2026-04-07T07:45:31.634Z","avatar_url":"https://github.com/ricall.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# KafkaSaga Application\nApplication spike that uses Kafka + Orchestration style Saga processing to break\ncomplex transactions into simple steps using a simple State Machine\n\n## Getting Started\n\n### Trying it out\nYou can spin up the entire application using:\n\n```bash\nmake start\n```\n\nThis will bring up the following:\n* Zookeeper service\n* Kafka service\n* Postgres DB service\n* 4 KafkaSaga instances\n\n#### Sending a http request to the microservice\n```http request\nPOST http://localhost:8181/v1/message\nContent-Type: application/json\nAccept: application/json\n\n{\n\t\"source\": \"source\",\n\t\"destination\": \"kafkasaga1\",\n\t\"message\": \"Simple message sent to the first kafka saga microservice\"\n}\n```\nThe other three servers are running on ports 8182, 8183 and 8184\n\n#### How the kafka saga microservice works\n![Services](src/images/KafkaSaga%20Overview.png)\n\nThe Kafka Saga instances use Kafka to transport Messages and manage MessageEvents, any Messages received\nare persisted into the Postgres Database.\n\n#### Kafka topics and message flow\n![Topics](src/images/KafkaTopics.png)\n\nThe message flow is:\n1. KafkaSaga instance receives HTTP request containing a new Message\n2. The message is written to the `dev.message` kafka topic\n3. A (potentially new) KafkaSaga instance receives the message from the `dev.message` kafka topic\n4. The message is written to the DB\n5. A MessageEvent is published to `dev.message.event.internal` kafka topic with a status of PERSISTED\n6. The PERSISTED event is sent to all Kafka instances and is ignored\n7. The PERSISTED event is sent to a (potentially new) KafkaSaga `MessageEventHandler` which delgates the\nwork to the `StandardiseMessageAction` class\n8. (n) The message is standardised written to the database\n9 The `StandardiseMessageAction` class returns a new MessageEvent with a status of STANDARDISED\n10. (n+1) The STANDARDISED event is sent to all KafkaSaga instances and the one that handled the http request\nloads the standardised message from the Database\n11. (n+2) The KafkaSaga instance returns the http response to the caller\n12. The STANDARDISED event is sent to the KafkaSaga `MessageEventHandler` which delegates the \nwork to the `PropagateMessageAction` class\n13. The message is propagated to other external systems\n14. The `PropagateMessageAction` class returns a new MessageEvent with a status of PROPAGATED\n15. The PROPAGATED event is sent to all KafkaSaga instances and is ignored\n16. The PROPAGATED event is sent to the KafkaSaga `MessageEventHandler` which delegates the\nwork to the `ProcessingCompleteAction` class\n17. Any cleanup work is performed after processing finishes\n18. The `ProcessingCompleteAction` class returns a new MessageEvent with a status of PROCESSED\n19. `MessageEventHandler` writes a tombstone record into `dev.message.event.internal`\n\n#### Message processing\nThere are three main parts to the microservice:\n* `MesageController` REST interface - Provides the controller that receives messages and puts them on the `dev.message` topic\n* `MessageHandler` Reads the message off the `dev.message` topic, write it to the DB and triggers a PERSISTED\n  event on `dev.message.event.internal` topic\n* `MessageEventHandler` Listens to message events and implements a simple state machine to process the messages\n\n```text\nsaga1_1      | 2021-03-13 12:57:30.556  INFO 6 --- [a-application-1] i.g.r.k.s.k.c.MessageController          : Message Sent: 744259f2-0759-49b6-985f-77948e94096e\nsaga3_1      | 2021-03-13 12:57:30.556  INFO 6 --- [ntainer#2-1-C-1] i.g.r.k.s.k.listener.MessageHandler      : Received message Message(id=null, source=source, destination=destination1, message=sample message) [744259f2-0759-49b6-985f-77948e94096e]\nsaga3_1      | 2021-03-13 12:57:30.563  INFO 6 --- [ntainer#2-1-C-1] i.g.r.k.s.k.listener.MessageHandler      :   Wrote message: [744259f2-0759-49b6-985f-77948e94096e]\nsaga4_1      | 2021-03-13 12:57:30.566  INFO 7 --- [ntainer#1-1-C-1] i.g.r.k.s.k.s.a.StandardiseMessageAction :   PERSISTED -\u003e Standardising message: [744259f2-0759-49b6-985f-77948e94096e]\nsaga4_1      | 2021-03-13 12:57:33.322  INFO 7 --- [ntainer#1-1-C-1] i.g.r.k.s.k.s.a.PropagateMessageAction   :   STANDARDISED -\u003e Propagating message: [744259f2-0759-49b6-985f-77948e94096e]\nsaga4_1      | 2021-03-13 12:57:36.279  INFO 7 --- [ntainer#1-1-C-1] i.g.r.k.s.k.s.a.ProcessingCompleteAction :   PROPAGATED -\u003e Processing finished: [744259f2-0759-49b6-985f-77948e94096e]\n```\nHere we can see from the log that the REST interface received a message on `INSTANCE 1`.\nThe Message was received on `INSTANCE 3` and was written to the database and a `PERSISTED` event was triggered\nThe Saga processor listened to the events on the `dev.message.event.internal` processing each event and triggering state changes\non `INSTANCE 4`\n\n*The instance that processes the message is determined by the way the broker has allocated partitions and the\nway the correlation id is mapped to partitions.*\n\nNOTE: The MessageController is listening to a reactive stream of `MessageEvent` objects from the `dev.message.event.internal`\ntopic - when it reaches `STANDARDISED` state the MessageController returns the `StandardisedMessage` associated with the event.\n\n#### Saga State Machine\nThe saga is represented as a simple state machine where message events are sent to Actions that process\nthem and return a new message event with a potentially different state.\n\nThis approach allows failures in Kafka, the Database or KafkaSaga instances to be recoverable. For example\nif a KafkaSaga instance crashes halfway through processing a MessageEvent when the Kafka broker detects it\nhas failed the topic partitions will be reallocated between the remaining KafaSaga instances and the MessageEvent\nwill be redelivered and reprocessed.\n\n![SagaStates](src/images/SagaStates.png)\n\n\n### Local Development\nBefore the application can be run it needs a local kafka and postgres database running - this can be done\nusing docker:\n\n```bash\nmake start-dev\n```\n\nThis will start all the services (kafka, zookeeper and postgres)\n- Kafka should be available on localhost:9092\n- Zookeeper should be available on localhost:2181\n- Postgres should be available on localhost:5432\n\n### Accessing the database\nIn a separate terminal you can access the database to make sure it is working:\n```bash\nmake db-terminal\n```\nYou can open a SQL terminal using:\n```bash\npsql --host=database --username=admin --dbname=messages\n```\nThe password when prompted is `password`\n```sql\nSELECT * FROM Message;\n```\nCan be run after the application has started.\n\n## Running the application\n```bash\nmake run\n```\n\n## Running Servers in parallel\n\n### Build the application\n```bash\n./gradlew build\n```\n\n### Start Application Instances (different terminal for each command)\n```bash\nexport INSTANCE_ID=one \u0026\u0026 java -jar build/libs/KafkaSaga-0.0.1-SNAPSHOT.jar --server.port=8181\nexport INSTANCE_ID=two \u0026\u0026 java -jar build/libs/KafkaSaga-0.0.1-SNAPSHOT.jar --server.port=8182\nexport INSTANCE_ID=three \u0026\u0026 java -jar build/libs/KafkaSaga-0.0.1-SNAPSHOT.jar --server.port=8183\nexport INSTANCE_ID=four \u0026\u0026 java -jar build/libs/KafkaSaga-0.0.1-SNAPSHOT.jar --server.port=8184\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fricall%2Fkafkasaga","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fricall%2Fkafkasaga","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fricall%2Fkafkasaga/lists"}