{"id":22484812,"url":"https://github.com/gamussa/streams-mini-workshop","last_synced_at":"2026-05-19T07:34:59.506Z","repository":{"id":138710794,"uuid":"309874004","full_name":"gAmUssA/streams-mini-workshop","owner":"gAmUssA","description":"A mini version of a workshop I did during #Livestreams","archived":false,"fork":false,"pushed_at":"2025-10-23T17:53:02.000Z","size":449,"stargazers_count":3,"open_issues_count":2,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-23T19:37:11.971Z","etag":null,"topics":["confluent-cloud","kafka","kafka-streams","ksqldb","spring-boot"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gAmUssA.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-04T03:35:58.000Z","updated_at":"2021-11-28T18:58:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"3a98cad5-37cf-4523-8ac4-05a69b0dc972","html_url":"https://github.com/gAmUssA/streams-mini-workshop","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/gAmUssA/streams-mini-workshop","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gAmUssA%2Fstreams-mini-workshop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gAmUssA%2Fstreams-mini-workshop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gAmUssA%2Fstreams-mini-workshop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gAmUssA%2Fstreams-mini-workshop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gAmUssA","download_url":"https://codeload.github.com/gAmUssA/streams-mini-workshop/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gAmUssA%2Fstreams-mini-workshop/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33206320,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-19T07:16:55.748Z","status":"ssl_error","status_checked_at":"2026-05-19T07:16:54.366Z","response_time":58,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["confluent-cloud","kafka","kafka-streams","ksqldb","spring-boot"],"created_at":"2024-12-06T17:11:36.980Z","updated_at":"2026-05-19T07:34:59.482Z","avatar_url":"https://github.com/gAmUssA.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"= 👨🏼‍💻 Workshop — Developing Event-driven Microservices with Spring Boot 🍃, Confluent Cloud ☁️, and Java ☕️.\nViktor Gamov \u003cviktor@confluent.io\u003e, © 2020 Confluent, Inc.\n2020-11-04\n:linkattrs:\n:ast: \u0026ast;\n:toc: auto\n:toc-placement: auto\n:toc-position: auto\n:toc-title: Table of contents\n:toclevels: 3\n:idprefix:\n:idseparator: -\n:sectanchors:\n:icons: font\n:source-highlighter: highlight.js\n:highlightjs-theme: idea\n:experimental:\n:imagesdir: ./images\n\nDeveloping Event-driven Microservices with Spring Boot 🍃, Confluent Cloud ☁️, and Java ☕️.\n\ntoc::[]\n\n== Workshop prerequisites and setup\n\n=== Prerequisites\n\nTIP: You should have your login and password information handy provided by workshop organizers.\n\n* https://jdk.dev[Java 11 (or later)]\n* Your favorite IDE or text editor\n** Personally, I recommend https://www.jetbrains.com/idea/[IntelliJ IDEA].\n* Git is __Optional__\n\n== 🏁 Implementing a Stream Processor with Kafka Streams\n\nWe will implement a Kafka Streams topology to process atomic transactions to any request submitted to the `transaction-request` topic.\n\nWithin the *workshop* project folder, you will find a `transactions-processor` subfolder representing a Kafka Streams application.\nSpring Boot and the `spring-kafka` project handled the boilerplate code required to connect to Kafka.\nThis workshop will focus on writing a `Kafka Streams` topology with the function processing for our use case.\n\n=== Atomic transaction processing with Kafka Streams\n\nOur business requirement states that we must check whether the funds are sufficient for every request received before updating the balance of the account being processed.\nWe should never have two transactions being processed at the same time for the same account.\nThis would create a race condition for which we have no guarantee we can enforce the balance check before withdrawing funds.\n\n_The Data Generator_ writes transaction requests to the Kafka topic with a key equal to the transaction's account number.\nTherefore, we can be sure all messages of an account will be processed by a single thread for our Transaction Service no matter how many instances are concurrently running.\n\nNOTE: Kafka Streams won't commit any message offset until it completes our business logic of managing a transaction request.\n\nimage::transaction-service.png[Transaction Service]\n\n==== Implement the Transaction Transformer\n\nBecause of our stream processor's transaction nature, we require a specific component from Kafka Streams named a `Transformer.`\nThis utility allows us to process events one by one while interacting with a `State Store`–another\ncomponent of Kafka Streams that help us to persist our account balance in a local instance of an embedded database - RocksDB.\n\nOpen the `io.confluent.developer.ccloud.demo.kstream.TransactionTransformer`\nJava class and implement the `transform` function to return a `TransactionResult` based on the validity of the transaction request.\nThe `TransactionResult` contains a `success` flag set to `true` if the funds were successfully updated.\n\nThe `transform` method also updates the `store` State Store.\nThe class already has utility functions to help you execute our business logic.\n\n[source,java]\n.TransactionTransformer.transform()\n----\n  @Override\n  public TransactionResult transform(Transaction transaction) {\n\n    if (transaction.getType().equals(Transaction.Type.DEPOSIT)) {\n      return new TransactionResult(transaction,\n                                   depositFunds(transaction),\n                                   true,\n                                   null);\n    }\n\n    if (hasEnoughFunds(transaction)) {\n      return new TransactionResult(transaction, withdrawFunds(transaction), true, null);\n    }\n\n    log.info(\"Not enough funds for account {}.\", transaction.getAccount());\n\n    return new TransactionResult(transaction,\n                                 getFunds(transaction.getAccount()),\n                                 false,\n                                 TransactionResult.ErrorType.INSUFFICIENT_FUNDS);\n  }\n----\n\n.Validate implementation by running TopologyTestDriver test\n****\nRun `io.confluent.developer.ccloud.demo.kstream.TransactionTransformerTest` to validate that implementation is correct.\n****\n\n==== Implement the Streaming Topology\n\nIn Kafka Streams, a `Topology` is the definition of your data flow.\nIt's a  manifest for all operations and transformations to be applied to your data.\n\nTo start a stream processor, Kafka Streams only requires you to build a`Topology` and hand it over.\nKafka Streams will take care of managing the underlying consumers and producers.\n\nThe `io.confluent.developer.ccloud.demo.kstream.KStreamConfig` Java class already contains all the boilerplate code required by Kafka Streams to start our processor.\nIn this exercise, we will leverage a `StreamsBuilder` to define and instantiate a `Topology` that will handle our transaction processing.\n\nOpen the `io.confluent.developer.ccloud.demo.kstream.KStreamConfig.defineStreams` method and get ready to write your first Kafka Streams Topology.\n\n==== Create a KStream from the source topic.\n\nUse the `stream` method of `streamsBuilder` to turn a topic into a `KStream.`\n\n[source,java]\n----\nKStream\u003cString, Transaction\u003e transactionStream = streamsBuilder.stream(\"transaction-request\");\n----\n\n==== Leverage the Transformer to process our requests\n\nTo inform Kafka Streams that we want to update the `funds` State Store for all incoming requests atomically, we can leverage the `transformValues` operator to plugin our `TransactionTransformer.`\nThis operator requires us to specify the `funds` State Store that the `Transformer` will use.\nThis also instructs Kafka Streams to keep track of events from our `transaction-request` since they will result in a change of state for our store.\n\n[source,java]\n----\nKStream\u003cString, TransactionResult\u003e resultStream = transactionStream.transformValues(() -\u003e new TransactionTransformer(storeName), storeName);\n----\n\n==== Redirect the transaction result to the appropriate topic.\n\nWith a new derived stream containing `TransactionResult,` we can now use the information contained in the payload to feed a success or failure topic.\n\nWe will achieve this by deriving two streams from our `resultStream.`\nEach stream will be built by applying a `filter` and `filterNot` operator with a predicate on the `success` flag from our `TransactionResult` payload.\nWith the two derived streams, we can explicitly call the `to` operator to instruct Kafka\nStreams to write the mutated events to their respective topics.\n\n[source,java]\n----\nresultStream\n  .filter(this::success)\n  .to(\"transaction-successs\");\n\nresultStream\n  .filterNot(this::success)\n  .to(\"transaction-failed\");\n----\n\n==== The implemented `defineStreams` method\n\nUse this reference implementation to validate you have the right stream definition.\n\n[source,java]\n----\nprotected void defineStreams(StreamsBuilder streamsBuilder) {\n\n    KStream\u003cString, Transaction\u003e transactionStream = streamsBuilder.stream(transactionRequestConfiguration.getName());\n\n    final String storeName = fundsStoreConfig.getName();\n    KStream\u003cString, TransactionResult\u003e resultStream = transactionStream.transformValues(() -\u003e new TransactionTransformer(storeName), storeName);\n\n    resultStream\n        .filter(this::success)\n        .to(transactionSuccessConfiguration.getName());\n\n    resultStream\n        .filterNot(this::success)\n        .to(transactionFailedConfiguration.getName());\n  }\n----\n\n.Validate implementation by running TopologyTestDriver test\n****\nRun `io.confluent.developer.ccloud.demo.kstream.KStreamConfigTest` to validate that implementation is correct.\n****\n\n=== Running the Kafka Streams application\n\nNOTE: If you are running the application from your IDE, launch the main method from `io.confluent.developer.ccloud.demo.kstream.KStreamDemoApplication`.\n\nIf you want to run with the CLI, you must build the application before launching it.\n\n.To build the application, run the following command:\n----\n./gradlew :transactions-processor:build\n----\n\n.To run the application run the following command\n----\njava -jar transactions-processor/build/libs/kstreams-demo-0.0.1-SNAPSHOT.jar\n----\n\n=== Generate some transactions using the Data Generator endpoint\n\nEnsure your Data Generator application is still running from the previous section.\n\nThe utility script `scripts/generate-transaction.sh` will let you generate transactions.\nGenerate a few transactions using the following commands:\n\n[source,shell script]\n----\nscripts/generate-transaction.sh 1 DEPOSIT 100 CAD\nscripts/generate-transaction.sh 1 DEPOSIT 200 CAD\nscripts/generate-transaction.sh 1 DEPOSIT 300 CAD\nscripts/generate-transaction.sh 1 WITHDRAW 300 CAD\nscripts/generate-transaction.sh 1 WITHDRAW 10000 CAD\n\nscripts/generate-transaction.sh 2 DEPOSIT 100 CAD\nscripts/generate-transaction.sh 2 DEPOSIT 50 CAD\nscripts/generate-transaction.sh 2 DEPOSIT 300 CAD\nscripts/generate-transaction.sh 2 WITHDRAW 300 CAD\n----\n\nThe script will pass in the following arguments:\n\n* The account number.\n* The amount.\n* The type of operation (`DEPOSIT` or `WITHDRAW`).\n* The currency.\n\n=== Monitor the successful transaction results\n\n. Access Confluent Cloud user interface from https://confluent.cloud.\n. From the main screen, navigate to the environment that looks like `demo-env-\u003csome-number\u003e.`\n. Inside of the environment, you should see a cluster that looks like `demo-kafka-cluster-\u003csome-number\u003e.`\nOn the left side, click on `Topics.`\n. Click on the `transaction-success` topic and access the `messages` tab.\n. Click on the `offset` textbox and type `0` and press enter to load all messages from partition 0 starting from offset 0.\n\nYou should see `transaction-success` events in the user interface. If you\ndon't see any messages, try your luck with partition 1 starting from offset 0.\n\n=== Monitor the failed transaction results from Control Center\n\n. Click on the `topic` tab from the cluster navigation menu.\n. Select the `transaction-failed` topic and access the `messages` tab.\n. Click on the `offset` textbox and type `0` and press enter to load all messages from partition 0 starting from offset 0.\n\nYou should see `transaction-failed` events in the user interface. \nIf you don't see any messages, try your lock with partition 1 starting from offset 0.\n\n// TODO\n// image::transaction-failed.png[transaction-failed]\n\nIn the next section, we will explore how writing Stream Processor can be simplified with `ksqlDB.`\n\n== 🎁 DEMO: Enrich transaction results with ksqlDB\n\nI will show how to write a second Stream Processor to generate a detailed transaction statement enriched with account details.\n\nWe will leverage ksqlDB to declare a stream processor that will enrich our transaction data in real-time with our referential data coming from the `account` topic. \nThe objective of this section is to show how you can use an SQL-like query language to generate streams processors like Kafka Streams, without having to compile and run any custom software.\n\nimage::transaction-statement-overview.png[Transaction Statements]\n\n=== Create the account table\n\nksqlDB is built on top of Kafka Streams. \nAs such, the `KStream` and `KTable` are both key constructs for defining stream processors.\n\nThe first step requires us to instruct ksqlDB that we wish to turn the `account` topic into a `Table`.\nThis table will allow us to join each `transaction-success` event with the latest `account` event of the underlying  topic.\nRun the following command in your ksqlDB CLI terminal:\n\n[source,sql]\n----\nCREATE TABLE ACCOUNT (\n  numkey string PRIMARY KEY,\n  number INT,\n  cityAddress STRING,\n  countryAddress STRING,\n  creationDate BIGINT,\n  firstName STRING,\n  lastName STRING,\n  numberAddress STRING,\n  streetAddress STRING,\n  updateDate BIGINT\n) WITH (\n  KAFKA_TOPIC = 'account',\n  VALUE_FORMAT='JSON'\n);\n----\n\n// CCLOUD_KSQL_ID=`ccloud ksql app list -ojson | jq -r '.[0].id'`\n// CCLOUD_KAFKA_ID=`ccloud kafka cluster list -ojson | jq -r '.[0].id'`\n// ccloud ksql app configure-acls ${CCLOUD_KSQL_ID} \"*\" --cluster ${CCLOUD_KAFKA_ID}\n\n=== Create the transaction-success stream\n\nBefore we create the `Transaction Statement` stream processor, we must also inform ksqlDB that we wish to turn the `transaction-success` into a `Stream.`\nRun the following command in your ksqlDB CLI terminal:\n\n[source,sql]\n----\nCREATE STREAM TRANSACTION_SUCCESS (\n  numkey string KEY,\n  transaction STRUCT\u003cguid STRING, account STRING, amount DOUBLE, type STRING, currency STRING, country STRING\u003e,\n  funds STRUCT\u003caccount STRING, balance DOUBLE\u003e,\n  success boolean,\n  errorType STRING\n) WITH (\n  kafka_topic='transaction-success',\n  value_format='json'\n);\n----\n\n=== Create the transaction statement stream\n\nNow that we have all the ingredients of our `Transaction Statement` stream processor, we can now create a new stream derived from our `transaction-success` events paired with the latest data from the `account` topic. \nWe will instruct ksqlDB to create a new stream as a query. \nBy default, ksqlDB will publish any output to a new `TRANSACTION_STATEMENT` topic. \nThe select query provides the details about with events to subscribe and which table to join each notification. \nThe output of this new stream processor will be a mix of the transaction details coupled with all the matching account details. \nThe key from `transaction-success` and `account` will be used as matching criteria for the `LEFT JOIN` command. \n`EMIT CHANGES` informs ksqlDB that the query is long-running and should be kept alive–as if it were a Kafka Streams application to be 100% available to process all events. \nRun  the following command in your ksqlDB CLI prompt:\n\n[source,sql]\n----\nCREATE STREAM TRANSACTION_STATEMENT AS\n  SELECT *\n  FROM TRANSACTION_SUCCESS\n  LEFT JOIN ACCOUNT ON TRANSACTION_SUCCESS.numkey = ACCOUNT.numkey\n  EMIT CHANGES;\n----\n\n== ✅ It's a wrap!\n\nCongratulations! \nNow you know how to build event-driven microservices using Spring Boot, Kafka Streams, and ksqlDB.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgamussa%2Fstreams-mini-workshop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgamussa%2Fstreams-mini-workshop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgamussa%2Fstreams-mini-workshop/lists"}