{"id":15403614,"url":"https://github.com/rogervinas/spring-cloud-stream-kafka-streams-first-steps","last_synced_at":"2026-02-27T22:38:31.811Z","repository":{"id":44958722,"uuid":"407799875","full_name":"rogervinas/spring-cloud-stream-kafka-streams-first-steps","owner":"rogervinas","description":"🍀 Spring Cloud Stream \u0026 Kafka Streams Binder - first steps","archived":false,"fork":false,"pushed_at":"2026-02-09T19:38:27.000Z","size":420,"stargazers_count":2,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-02-09T22:58:04.565Z","etag":null,"topics":["kafka","kafka-streams","kotlin","spring-cloud-stream"],"latest_commit_sha":null,"homepage":"https://dev.to/rogervinas/spring-cloud-stream-kafka-stream-binder-first-steps-1pch","language":"Kotlin","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rogervinas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-09-18T08:16:28.000Z","updated_at":"2026-02-07T19:25:43.000Z","dependencies_parsed_at":"2024-02-14T09:47:39.014Z","dependency_job_id":"5a95b77e-efc3-427f-b363-bd9c8391592e","html_url":"https://github.com/rogervinas/spring-cloud-stream-kafka-streams-first-steps","commit_stats":{"total_commits":68,"total_committers":4,"mean_commits":17.0,"dds":0.5294117647058824,"last_synced_commit":"7cf83884b138444a5a9abc1da77261640fffc3a6"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/rogervinas/spring-cloud-stream-kafka-streams-first-steps","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogervinas%2Fspring-cloud-stream-kafka-streams-first-steps","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogervinas%2Fspring-cloud-stream-kafka-streams-first-steps/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogervinas%2Fspring-cloud-stream-kafka-streams-first-steps/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogervinas%2Fspring-cloud-stream-kafka-streams-first-steps/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rogervinas","download_url":"https://codeload.github.com/rogervinas/spring-cloud-stream-kafka-streams-first-steps/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogervinas%2Fspring-cloud-stream-kafka-streams-first-steps/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29917844,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-27T19:37:42.220Z","status":"ssl_error","status_checked_at":"2026-02-27T19:37:41.463Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-streams","kotlin","spring-cloud-stream"],"created_at":"2024-10-01T16:09:27.115Z","updated_at":"2026-02-27T22:38:31.804Z","avatar_url":"https://github.com/rogervinas.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CI](https://github.com/rogervinas/spring-cloud-stream-kafka-streams-first-steps/actions/workflows/ci.yml/badge.svg)](https://github.com/rogervinas/spring-cloud-stream-kafka-streams-first-steps/actions/workflows/ci.yml)\n![Java](https://img.shields.io/badge/Java-21-blue?labelColor=black)\n![Kotlin](https://img.shields.io/badge/Kotlin-2.x-blue?labelColor=black)\n![SpringBoot](https://img.shields.io/badge/SpringBoot-4.x-blue?labelColor=black)\n\n# Spring Cloud Stream \u0026 Kafka Streams Binder first steps\n\n[Spring Cloud Stream](https://spring.io/projects/spring-cloud-stream) is the solution provided by **Spring** to build applications connected to shared messaging systems.\n\nIt offers an abstraction (the **binding**) that works the same whatever underneath implementation we use (the **binder**):\n* **Apache Kafka**\n* **Rabbit MQ**\n* **Kafka Streams**\n* **Amazon Kinesis**\n* ...\n\nYou can also check out [Spring Cloud Stream Kafka step by step](https://github.com/rogervinas/spring-cloud-stream-step-by-step) where I got working a simple example using **Kafka binder**.\n\nLet's try this time a simple example using **Kafka Streams binder**! 🤩\n\n* [First steps](#first-steps)\n* [Goal](#goal)\n* [Integration Test](#integration-test)\n* [Kafka Streams binder configuration](#kafka-streams-binder-configuration)\n* [TotalScoreProcessor first implementation](#totalscoreprocessor-first-implementation)\n* [TotalScoreProcessor test using kafka-streams-test-utils](#totalscoreprocessor-test-using-kafka-streams-test-utils)\n* [Final implementation](#final-implementation)\n* [Test this demo](#test-this-demo)\n* [Run this demo](#run-this-demo)\n* See also\n  * :octocat: [Spring Cloud Stream Kafka step by step](https://github.com/rogervinas/spring-cloud-stream-kafka-step-by-step)\n  * :octocat: [Spring Cloud Stream \u0026 Kafka Confluent Avro Schema Registry](https://github.com/rogervinas/spring-cloud-stream-kafka-confluent-avro-schema-registry)\n  * :octocat: [Spring Cloud Stream Multibinder](https://github.com/rogervinas/spring-cloud-stream-multibinder)\n  * :octocat: [Spring Cloud Stream \u0026 Kafka Streams Binder + Processor API](https://github.com/rogervinas/spring-cloud-stream-kafka-streams-processor)\n\nYou can browse older versions of this repo:\n* [Spring Boot 2.x](https://github.com/rogervinas/spring-cloud-stream-kafka-streams-first-steps/tree/spring-boot-2.x)\n\n## First steps\n\nA bit of documentation to start with:\n\n* [Spring Cloud Kafka Streams Binder](https://cloud.spring.io/spring-cloud-stream-binder-kafka/spring-cloud-stream-binder-kafka.html#_kafka_streams_binder)\n* [Kafka Streams Documentation](https://kafka.apache.org/documentation/streams/)\n  * [Kafka Streams Developer Guide](https://kafka.apache.org/documentation/streams/developer-guide/)\n  * [Kafka Streams DSL](https://kafka.apache.org/documentation/streams/developer-guide/dsl-api.html)\n  * [Testing Kafka Streams](https://kafka.apache.org/documentation/streams/developer-guide/testing.html)\n\n## Goal\n\nWe want to implement this flow:\n\n![Diagram1](doc/diagram1.png)\n\n1. We receive messages with key = username and value = { score: number } from topic **pub.scores**\n2. We have to calculate the **total score** received by username on fixed windows of 10 seconds and send it to topic **pub.totals**\n\n## Integration Test\n\nFirst we create a project using this [spring initializr configuration](https://start.spring.io/#!type=gradle-project\u0026language=kotlin\u0026packaging=jar\u0026groupId=com.rogervinas\u0026artifactId=springcloudstreamkafkastreamsbinder\u0026name=springcloudstreamkafkastreamsbinder\u0026description=Spring%20Cloud%20Streams%20%26%20Kafka%20Streams%20Binder\u0026packageName=com.rogervinas.springcloudstreamkafkastreamsbinder\u0026dependencies=cloud-stream) and we add **Kafka Streams** binder dependency **spring-cloud-stream-binder-kafka-streams**.\n\nUsing [testcontainers](https://www.testcontainers.org/) and [docker-compose](docker-compose.yml) with a **Kafka** container, we write the following **integration test**:\n\n```kotlin\n@Test\nfun `should publish total scores`() {\n  kafkaProducerHelper.send(TOPIC_SCORES, USERNAME_1, \"{\\\"score\\\": 10}\")\n  kafkaProducerHelper.send(TOPIC_SCORES, USERNAME_2, \"{\\\"score\\\": 20}\")\n  kafkaProducerHelper.send(TOPIC_SCORES, USERNAME_1, \"{\\\"score\\\": 30}\")\n  kafkaProducerHelper.send(TOPIC_SCORES, USERNAME_2, \"{\\\"score\\\": 40}\")\n  kafkaProducerHelper.send(TOPIC_SCORES, USERNAME_1, \"{\\\"score\\\": 50}\")\n  kafkaProducerHelper.send(TOPIC_SCORES, USERNAME_2, \"{\\\"score\\\": 60}\")\n\n  Thread.sleep(totalScoreWindow.plusSeconds(1).toMillis())\n\n  // Send at least one more message so the previous window is closed\n  kafkaProducerHelper.send(TOPIC_SCORES, USERNAME_1, \"{\\\"score\\\": 1}\")\n  kafkaProducerHelper.send(TOPIC_SCORES, USERNAME_2, \"{\\\"score\\\": 1}\")\n\n  val records = kafkaConsumerHelper.consumeAtLeast(2, Duration.ofMinutes(1))\n\n  assertThat(records).hasSize(2)\n  assertThat(records.associate { record -\u003e record.key() to record.value() }).satisfies { valuesByKey -\u003e\n    JSONAssert.assertEquals(\"{\\\"totalScore\\\": 90}\", valuesByKey[USERNAME_1], true)\n    JSONAssert.assertEquals(\"{\\\"totalScore\\\": 120}\", valuesByKey[USERNAME_2], true)\n  }\n}\n```\n\nThis test will obviously fail, but it should work once we have finished our implementation.\n\nNote that we have to send another message after the window has expired to force **Kafka Streams** close the window.\nIt is the only way for **Kafka Streams** to be sure that there are no more messages left for that window.\nIn other words, what we are really implementing is:\n\n![Diagram2](doc/diagram2.png)\n\n## Kafka Streams binder configuration\n\nNext we configure the **Kafka Streams binder**:\n\n```yaml\nspring:\n  application:\n    name: \"spring-cloud-stream-kafka-streams-first-steps\"\n  cloud:\n    function:\n      definition: totalScoreProcessor\n    stream:\n      bindings:\n        totalScoreProcessor-in-0:\n          destination: \"pub.scores\"\n        totalScoreProcessor-out-0:\n          destination: \"pub.totals\"\n      kafka:\n        streams:\n          binder:\n            applicationId: \"${spring.application.name}\"\n            brokers: \"localhost:9092\"\n            configuration:\n              default:\n                key.serde: org.apache.kafka.common.serialization.Serdes$StringSerde\n                value.serde: org.apache.kafka.common.serialization.Serdes$StringSerde\n```\n\nWith this configuration:\n* **Spring Cloud Stream** will create a **Kafka Streams binder** connected to **localhost:9092**\n* We need to create a **@Bean** named **totalScoreProcessor** that should implement `Function\u003cKStream, KStream\u003e` interface\n  * This **@Bean** will connect a **KStream** subscribed to **pub.scores** topic to another **KStream** publishing to **pub.totals** topic\n\nYou can find all the available configuration properties documented in [Kafka Streams Properties](https://cloud.spring.io/spring-cloud-stream-binder-kafka/spring-cloud-stream-binder-kafka.html#_kafka_streams_properties).\n\n## TotalScoreProcessor first implementation\n\nWe can start with a simple implementation for a **TotalScoreProcessor** that for every **ScoreEvent** received will generate a **TotalScoreEvent** with the same value:\n\n```kotlin\ndata class ScoreEvent(val score: Int)\ndata class TotalScoreEvent(val totalScore: Int)\n\nclass MyTotalScoreProcessor(private val window: Duration) : Function\u003cKStream\u003cString, ScoreEvent\u003e, KStream\u003cString, TotalScoreEvent\u003e\u003e {\n  override fun apply(input: KStream\u003cString, ScoreEvent\u003e): KStream\u003cString, TotalScoreEvent\u003e {\n      return input.map { key, scoreEvent -\u003e KeyValue(key, TotalScoreEvent(scoreEvent.score)) }\n  }\n}\n\n@Configuration\nclass MyApplicationConfiguration {\n  @Bean\n  fun totalScoreProcessor(): Function\u003cKStream\u003cString, ScoreEvent\u003e, KStream\u003cString, TotalScoreEvent\u003e\u003e \n    = MyTotalScoreProcessor()\n}\n```\n\n💡 We are using **Spring Cloud Stream**'s default serialization/deserialization of **Kotlin** data classes to Json. In order for this to work we need to add **tools.jackson.module:jackson-module-kotlin** dependency.\n\nThis implementation is not fulfilling our goal yet, just execute [MyApplicationIntegrationTest](src/test/kotlin/com/rogervinas/kafkastreams/MyApplicationIntegrationTest.kt) and see it still failing! 😓\n\n## TotalScoreProcessor test using kafka-streams-test-utils\n\nUsing the [Test Pyramid](https://martinfowler.com/articles/practical-test-pyramid.html#TheTestPyramid) principle we should use **integration tests** to test the simple test cases and test the more complicated ones using **unit tests** (if not **unit tests** at least less \"integrated\" tests).\n\nTo create these less \"integrated\" tests we can use [kafka-streams-test-utils](https://kafka.apache.org/documentation/streams/developer-guide/testing.html).\n\nThey will be faster and more reliable (not needing **Kafka**) and with some cool features like \"advance time\" to simulate messages published at different instants in time.\n\nHere it is one way to create a **TopologyTestDriver** from [kafka-streams-test-utils](https://kafka.apache.org/documentation/streams/developer-guide/testing.html) to test our **TotalScoreProcessor**:\n\n```kotlin\n@BeforeEach\nfun beforeEach() {\n  val stringSerde = Serdes.StringSerde()\n  val scoreEventSerializer = JacksonJsonSerde(ScoreEvent::class.java).serializer()\n  val totalScoreEventDeserializer = JacksonJsonSerde(TotalScoreEvent::class.java).deserializer()\n  val streamsBuilder = StreamsBuilder()\n\n  // This way we test MyTotalScoreProcessor\n  MyTotalScoreProcessor()\n    .apply(streamsBuilder.stream(TOPIC_IN))\n    .to(TOPIC_OUT)\n\n  val config = Properties().apply {\n    setProperty(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, stringSerde.javaClass.name)\n    setProperty(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, JacksonJsonSerde::class.java.name)\n    setProperty(StreamsConfig.APPLICATION_ID_CONFIG, \"test\")\n    setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, \"test-server\")\n    setProperty(JacksonJsonDeserializer.TRUSTED_PACKAGES, \"*\")\n  }\n  val topology = streamsBuilder.build()\n  topologyTestDriver = TopologyTestDriver(topology, config)\n  topicIn = topologyTestDriver.createInputTopic(TOPIC_IN, stringSerde.serializer(), scoreEventSerializer)\n  topicOut = topologyTestDriver.createOutputTopic(TOPIC_OUT, stringSerde.deserializer(), totalScoreEventDeserializer)\n}\n```\n\nAnd then we can write tests like this one:\n\n```kotlin\n@Test\nfun `should publish total score of one username when window expires`() {\n  topicIn.pipeInput(USERNAME_1, ScoreEvent(25))\n  topicIn.pipeInput(USERNAME_1, ScoreEvent(37))\n  topicIn.pipeInput(USERNAME_1, ScoreEvent(13))\n\n  topicIn.advanceTime(TOTAL_SCORE_WINDOW.plusMillis(100))\n\n  // Send at least one more message so the previous window is closed\n  topicIn.pipeInput(USERNAME_1, ScoreEvent(1))\n\n  assertThat(topicOut.readKeyValuesToList()).singleElement().satisfies { topicOutMessage -\u003e\n    assertThat(topicOutMessage.key).isEqualTo(USERNAME_1)\n    assertThat(topicOutMessage.value).isEqualTo(TotalScoreEvent(75))\n  }\n}\n```\n\n## Final implementation\n\nAfter a few iterations your **TotalScoreProcessor** implementation should look similar to this:\n\n```kotlin\noverride fun apply(input: KStream\u003cString, ScoreEvent\u003e): KStream\u003cString, TotalScoreEvent\u003e {\n  return input\n    .groupByKey()\n    .windowedBy(TimeWindows.ofSizeAndGrace(totalScoreWindow, Duration.ZERO))\n    .aggregate(\n      { TotalScoreEvent(0) },\n      { _, scoreEvent, totalScoreEvent -\u003e TotalScoreEvent(scoreEvent.score + totalScoreEvent.totalScore) },\n      Materialized.`as`\u003cString?, TotalScoreEvent?, WindowStore\u003cBytes, ByteArray\u003e?\u003e(\"total-score\")\n        .withKeySerde(Serdes.StringSerde())\n        .withValueSerde(JacksonJsonSerde(TotalScoreEvent::class.java))\n    )\n    .suppress(Suppressed.untilWindowCloses(unbounded()))\n    .toStream()\n    .map { key, value -\u003e KeyValue(key.key(), value) }\n}\n```\n\nNote that we use the [suppression operator](https://kafka.apache.org/27/documentation/streams/developer-guide/dsl-api.html#window-final-results) to emit nothing for a window until it closes, and then emit the final result. If we were not using it we would have an output message for each input message.\n\nNow you can play around with [Kafka Streams DSL](https://kafka.apache.org/documentation/streams/developer-guide/dsl-api) and do more complicated stuff!\n\nHappy coding! 💙\n\n## Test this demo\n\n```shell\n./gradlew test\n```\n\n## Run this demo\n\n```shell\ndocker-compose up -d\n./gradlew bootRun\ndocker-compose down\n```\n\nThen you can use [kcat](https://github.com/edenhill/kcat) to produce/consume to/from **Kafka**:\n```shell\n# consume\nkcat -b localhost:9092 -C -t pub.scores -f '%k %s\\n'\nkcat -b localhost:9092 -C -t pub.totals -f '%k %s\\n'\n\n# produce\necho 'john:{\"score\":100}' | kcat -b localhost:9092 -P -t pub.scores -K:\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frogervinas%2Fspring-cloud-stream-kafka-streams-first-steps","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frogervinas%2Fspring-cloud-stream-kafka-streams-first-steps","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frogervinas%2Fspring-cloud-stream-kafka-streams-first-steps/lists"}