{"id":15403612,"url":"https://github.com/rogervinas/spring-cloud-stream-multibinder","last_synced_at":"2025-08-16T11:35:07.886Z","repository":{"id":44958721,"uuid":"413160278","full_name":"rogervinas/spring-cloud-stream-multibinder","owner":"rogervinas","description":"🍀 Spring Cloud Stream Multibinder - Kafka \u0026 Kafka Streams","archived":false,"fork":false,"pushed_at":"2025-06-25T10:15:11.000Z","size":254,"stargazers_count":5,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-25T11:25:42.353Z","etag":null,"topics":["kafka","kafka-streams","kotlin","spring-cloud-stream"],"latest_commit_sha":null,"homepage":"https://dev.to/rogervinas/spring-cloud-stream-multibinder-2c4j","language":"Kotlin","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rogervinas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-10-03T18:16:53.000Z","updated_at":"2025-06-25T10:15:13.000Z","dependencies_parsed_at":"2024-03-01T09:28:32.453Z","dependency_job_id":"d9953fd9-2c3d-4b59-bad1-f6291d46f90b","html_url":"https://github.com/rogervinas/spring-cloud-stream-multibinder","commit_stats":{"total_commits":56,"total_committers":3,"mean_commits":"18.666666666666668","dds":0.4821428571428571,"last_synced_commit":"7873f25b734cbb2050213aa6f6c11d5a34ff7c45"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/rogervinas/spring-cloud-stream-multibinder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogervinas%2Fspring-cloud-stream-multibinder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogervinas%2Fspring-cloud-stream-multibinder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogervinas%2Fspring-cloud-stream-multibinder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogervinas%2Fspring-cloud-stream-multibinder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rogervinas","download_url":"https://codeload.github.com/rogervinas/spring-cloud-stream-multibinder/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogervinas%2Fspring-cloud-stream-multibinder/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270706569,"owners_count":24631720,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-16T02:00:11.002Z","response_time":91,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-streams","kotlin","spring-cloud-stream"],"created_at":"2024-10-01T16:09:26.215Z","updated_at":"2025-08-16T11:35:07.877Z","avatar_url":"https://github.com/rogervinas.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CI](https://github.com/rogervinas/spring-cloud-stream-multibinder/actions/workflows/ci.yml/badge.svg)](https://github.com/rogervinas/spring-cloud-stream-multibinder/actions/workflows/ci.yml)\n![Java](https://img.shields.io/badge/Java-21-blue?labelColor=black)\n![Kotlin](https://img.shields.io/badge/Kotlin-2.x-blue?labelColor=black)\n![SpringBoot](https://img.shields.io/badge/SpringBoot-3.x-blue?labelColor=black)\n![SpringCloud](https://img.shields.io/badge/SpringCloud-2024.x-blue?labelColor=black)\n\n# Spring Cloud Stream Multibinder\n\n[Spring Cloud Stream](https://spring.io/projects/spring-cloud-stream) is the solution provided by **Spring** to build applications connected to shared messaging systems.\n\nIt offers an abstraction (the **binding**) that works the same whatever underneath implementation we use (the **binder**):\n* **Apache Kafka**\n* **Rabbit MQ**\n* **Kafka Streams**\n* **Amazon Kinesis**\n* ...\n\nBut what if we need more than one binder in the same application? 🤔\n\nNot a problem! You can specify multiple binder configurations as documented in [Connecting to Multiple Systems](https://cloud.spring.io/spring-cloud-stream/spring-cloud-stream.html#multiple-systems)\n\nLet's put the theory into practice 🛠️ ...\n\n* [Goal](#goal)\n* [Create the project](#create-the-project)\n* [Integration Test](#integration-test)\n* [Spring Cloud Stream binders configuration](#spring-cloud-stream-binders-configuration)\n* [TextProducer](#textproducer)\n* [TextController](#textcontroller)\n* [TextProducer implementation](#textproducer-implementation)\n* [TextLengthProcessor](#textlengthprocessor)\n* [LengthConsumer](#lengthconsumer)\n* [LengthConsumer implementation](#lengthconsumer-implementation)\n* [Wiring it all together](#wiring-it-all-together)\n* [Test this demo](#test-this-demo)\n* [Run this demo](#run-this-demo)\n* See also\n  * :octocat: [Spring Cloud Stream Kafka step by step](https://github.com/rogervinas/spring-cloud-stream-kafka-step-by-step)\n  * :octocat: [Spring Cloud Stream \u0026 Kafka Confluent Avro Schema Registry](https://github.com/rogervinas/spring-cloud-stream-kafka-confluent-avro-schema-registry)\n  * :octocat: [Spring Cloud Stream \u0026 Kafka Streams Binder first steps](https://github.com/rogervinas/spring-cloud-stream-kafka-streams-first-steps)\n  * :octocat: [Spring Cloud Stream \u0026 Kafka Streams Binder + Processor API](https://github.com/rogervinas/spring-cloud-stream-kafka-streams-processor)\n\nYou can browse older versions of this repo:\n* [Spring Boot 2.x](https://github.com/rogervinas/spring-cloud-stream-multibinder/tree/spring-boot-2.x)\n\n## Goal\n\nWe want to implement this flow:\n\n![Diagram1](doc/diagram1.png)\n\n* User will POST string payloads to **/text** endpoint\n* A **KafkaProducer** will send these payloads to topic **pub.texts** as `{ \"text\" : string }`\n* A **KafkaStreams** transformation will consume from topic **pub.texts** and produce events to topic **pub.lengths** as `{ \"length\" : number }`\n* A **KafkaConsumer** will consume events from topic **pub.lengths** and log them to the console\n\nSo we will use two **Spring Cloud Stream** binders:\n* **Kafka**\n* **Kafka Streams**\n\n## Create the project\n\nWe use this [spring initializr configuration](https://start.spring.io/#!type=gradle-project\u0026language=kotlin\u0026packaging=jar\u0026groupId=com.rogervinas\u0026artifactId=springcloudstreammultibinder\u0026name=springcloudstreammultibinder\u0026description=Spring%20Cloud%20Streams%20%26%20Kafka%20Streams%20Binder\u0026packageName=com.rogervinas.springcloudstreammultibinder\u0026dependencies=cloud-stream) and we add:\n* **Kafka** binder lib **spring-cloud-stream-binder-kafka**\n* **Kafka Streams** binder lib **spring-cloud-stream-binder-kafka-streams**\n\n## Integration Test\n\nWe start writing the following **integration test**, using:\n* [Testcontainers](https://www.testcontainers.org/) and [docker-compose](docker-compose.yml) with a **Kafka** container\n* JUnit Jupiter [OutputCaptureExtension](https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/test/system/OutputCaptureExtension.html)\n\n```kotlin\n@Test\nfun `should process text lengths`(capturedOutput: CapturedOutput) {\n  postText(\"Do\")\n  postText(\"Or do not\")\n  postText(\"There is no try\")\n\n  await().atMost(ONE_MINUTE).untilAsserted {\n    assertThat(capturedOutput.out).contains(\"Consumed length [2]\")\n    assertThat(capturedOutput.out).contains(\"Consumed length [9]\")\n    assertThat(capturedOutput.out).contains(\"Consumed length [15]\")\n  }\n}\n```\n\nThis test will obviously fail, but it should work once we have finished our implementation.\n\n## Spring Cloud Stream binders configuration\n\nNext we configure the two binders:\n\n```yaml\nspring:\n  application:\n    name: \"spring-cloud-stream-multibinder\"\n  cloud:\n    function:\n      definition: textProducer;textLengthProcessor;lengthConsumer\n    stream:\n      bindings:\n        textProducer-out-0:\n          destination: \"${kafka.topic.texts}\"\n          binder: kafka1\n        textLengthProcessor-in-0:\n          destination: \"${kafka.topic.texts}\"\n          binder: kstream1\n        textLengthProcessor-out-0:\n          destination: \"${kafka.topic.lengths}\"\n          binder: kstream1\n        lengthConsumer-in-0:\n          destination: \"${kafka.topic.lengths}\"\n          group: \"${spring.application.name}\"\n          binder: kafka1\n      binders:\n        kafka1:\n          type: kafka\n          environment:\n            spring.cloud.stream.kafka.binder:\n              brokers: \"${kafka.brokers}\"\n        kstream1:\n          type: kstream\n          environment:\n            spring.cloud.stream.kafka.streams.binder:\n              applicationId: \"${spring.application.name}-KApp\"\n              brokers: \"${kafka.brokers}\"\n\nkafka:\n  topic:\n    texts: \"pub.texts\"\n    lengths: \"pub.lengths\"\n  brokers: \"localhost:9092\"\n```\n\n* **Spring Cloud Stream** will create:\n  * A **Kafka Streams binder** connected to **localhost:9092**\n  * A **Kafka binder** connected to **localhost:9092**\n* Following the **Spring Cloud Stream functional programming model conventions** we should create:\n  * A bean named **textProducer** that should implement:\n    * In **Java**: `Supplier\u003cFlux\u003cTextEvent\u003e\u003e` interface\n    * In **Kotlin**: `() -\u003e Flux\u003cTextEvent\u003e` lambda\n  * A bean named **textLengthProcessor** that should implement:\n    * In **Java**: `Function\u003cKStream\u003cString, TextEvent\u003e, KStream\u003cString, LengthEvent\u003e\u003e` interface\n    * In **Kotlin**: the same, there is no support for lambdas yet 😅\n  * A bean named **lengthConsumer** that should implement:\n    * In **Java**: `Consumer\u003cLengthEvent\u003e` interface\n    * In **Kotlin**: `(LengthEvent) -\u003e Unit` lambda\n\n💡 We use different values for the Kafka Streams **applicationId** and the Kafka Consumers **group** to avoid undesired behaviors.\n\n💡 We are using **Spring Cloud Stream**'s default serialization/deserialization of Kotlin data classes to Json. In order for this to work we need to add **com.fasterxml.jackson.module:jackson-module-kotlin** dependency.\n\n💡 You can find all the available configuration properties documented in:\n* [Kafka binder properties](https://github.com/spring-cloud/spring-cloud-stream-binder-kafka#kafka-binder-properties)\n* [Kafka Streams binder properties](https://cloud.spring.io/spring-cloud-stream-binder-kafka/spring-cloud-stream-binder-kafka.html#_kafka_streams_properties)\n\n## TextProducer\n\nWe create **TextProducer** interface to be implemented later:\n\n```kotlin\ndata class TextEvent(val text: String)\n\ninterface TextProducer {\n  fun produce(event: TextEvent)\n}\n```\n\n## TextController\n\nOnce we have the test ...\n\n```kotlin\n@WebFluxTest(controllers = [TextController::class])\nclass TextControllerTest {\n\n  @Autowired\n  lateinit var webClient: WebTestClient\n\n  @MockitoBean\n  lateinit var textProducer: TextProducer\n\n  @Test\n  fun `should produce text events`() {\n    val text = \"Some awesome text\"\n    webClient.post()\n      .uri(\"/text\")\n      .bodyValue(text)\n      .exchange()\n      .expectStatus().isOk\n\n    verify(textProducer).produce(TextEvent(text))\n  }\n}\n```\n\n... the implementation is easy:\n\n```kotlin\n@RestController\nclass TextController(private val textProducer: TextProducer) {\n\n  @PostMapping(\"/text\", consumes = [TEXT_PLAIN_VALUE])\n  fun text(@RequestBody text: String) {\n    textProducer.produce(TextEvent(text))\n  }\n}\n```\n\n## TextProducer implementation\n\nWe implement **TextProducer** as expected by **Spring Cloud Stream** conventions like this:\n\n```kotlin\nclass TextFluxProducer : () -\u003e Flux\u003cTextEvent\u003e, TextProducer {\n\n  private val sink = Sinks.many().unicast().onBackpressureBuffer\u003cTextEvent\u003e()\n\n  override fun produce(event: TextEvent) {\n    sink.emitNext(event, FAIL_FAST)\n  }\n\n  override fun invoke() = sink.asFlux()\n}\n```\n\nAnd we can easily test the implementation as follows:\n\n```kotlin\n@Test\nfun `should produce text events`() {\n  val producer = TextFluxProducer()\n\n  val events = mutableListOf\u003cTextEvent\u003e()\n  producer().subscribe(events::add)\n\n  producer.produce(TextEvent(\"Well\"))\n  producer.produce(TextEvent(\"nobody is\"))\n  producer.produce(TextEvent(\"perfect!\"))\n\n  assertThat(events).containsExactly(\n    TextEvent(\"Well\"),\n    TextEvent(\"nobody is\"),\n    TextEvent(\"perfect!\")\n  )\n}\n```\n\n## TextLengthProcessor\n\nWe implement the transformation using Kafka Stream's mapValues method:\n\n```kotlin\nclass TextLengthProcessor : Function\u003cKStream\u003cString, TextEvent\u003e, KStream\u003cString, LengthEvent\u003e\u003e {\n\n  override fun apply(input: KStream\u003cString, TextEvent\u003e): KStream\u003cString, LengthEvent\u003e {\n    return input\n      .mapValues { event -\u003e LengthEvent(event.text.length) }\n  }\n}\n```\n\nAnd we can test it using **kafka-streams-test-utils**:\n\n```kotlin\nprivate const val TOPIC_IN = \"topic.in\"\nprivate const val TOPIC_OUT = \"topic.out\"\n\nprivate const val KEY1 = \"key1\"\nprivate const val KEY2 = \"key2\"\nprivate const val KEY3 = \"key3\"\n\ninternal class TextLengthProcessorTest {\n\n  private lateinit var topologyTestDriver: TopologyTestDriver\n  private lateinit var topicIn: TestInputTopic\u003cString, TextEvent\u003e\n  private lateinit var topicOut: TestOutputTopic\u003cString, LengthEvent\u003e\n\n  @BeforeEach\n  fun beforeEach() {\n    val stringSerde = Serdes.StringSerde()\n    val streamsBuilder = StreamsBuilder()\n\n    TextLengthProcessor()\n      .apply(streamsBuilder.stream(TOPIC_IN))\n      .to(TOPIC_OUT)\n\n    val config = Properties().apply {\n      setProperty(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, stringSerde.javaClass.name)\n      setProperty(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, JsonSerde::class.java.name)\n      setProperty(StreamsConfig.APPLICATION_ID_CONFIG, \"test\")\n      setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, \"test-server\")\n      setProperty(JsonDeserializer.TRUSTED_PACKAGES, \"*\")\n    }\n    val topology = streamsBuilder.build()\n    topologyTestDriver = TopologyTestDriver(topology, config)\n    topicIn = topologyTestDriver.createInputTopic(TOPIC_IN, stringSerde.serializer(), JsonSerde(TextEvent::class.java).serializer())\n    topicOut = topologyTestDriver.createOutputTopic(TOPIC_OUT, stringSerde.deserializer(), JsonSerde(LengthEvent::class.java).deserializer())\n  }\n\n  @AfterEach\n  fun afterEach() {\n    topologyTestDriver.close()\n  }\n\n  @Test\n  fun `should produce length events from text events`() {\n    topicIn.pipeInput(KEY1, TextEvent(\"Hello!\"))\n    topicIn.pipeInput(KEY2, TextEvent(\"How are you?\"))\n    topicIn.pipeInput(KEY3, TextEvent(\"Bye!\"))\n\n    assertThat(topicOut.readKeyValuesToList()).containsExactly(\n      KeyValue(KEY1, LengthEvent(6)),\n      KeyValue(KEY2, LengthEvent(12)),\n      KeyValue(KEY3, LengthEvent(4))\n    )\n  }\n}\n```\n\n## LengthConsumer\n\nWe implement **LengthStreamConsumer** as expected by **Spring Cloud Stream** conventions like this:\n\n```kotlin\ndata class LengthEvent(val length: Int)\n\nclass LengthStreamConsumer(private val processor: LengthProcessor) : (LengthEvent) -\u003e Unit {\n\n  override fun invoke(event: LengthEvent) {\n    processor.process(event)\n  }\n}\n```\n\nWe decouple the final implementation using the interface **LengthProcessor**:\n\n```kotlin\ninterface LengthProcessor {\n  fun process(event: LengthEvent)\n}\n```\n\nAnd we can test everything with this code:\n\n```kotlin\n@Test\nfun `should consume length events`() {\n  val lengthProcessor = mock(LengthProcessor::class.java)\n  val lengthStreamConsumer = LengthStreamConsumer(lengthProcessor)\n\n  lengthStreamConsumer(LengthEvent(10))\n  lengthStreamConsumer(LengthEvent(20))\n  lengthStreamConsumer(LengthEvent(30))\n\n  verify(lengthProcessor).process(LengthEvent(10))\n  verify(lengthProcessor).process(LengthEvent(20))\n  verify(lengthProcessor).process(LengthEvent(30))\n}\n```\n\n## LengthConsumer implementation\n\nFor this demo the implementation just logs the event:\n\n```kotlin\nclass LengthConsoleProcessor : LengthProcessor {\n\n  private val logger = LoggerFactory.getLogger(LengthConsoleProcessor::class.java)\n\n  override fun process(event: LengthEvent) {\n    logger.info(\"Consumed length [${event.length}]\")\n  }\n}\n```\n\nAnd we can also test it using JUnit Jupiter [OutputCaptureExtension](https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/test/system/OutputCaptureExtension.html):\n\n```kotlin\n@ExtendWith(OutputCaptureExtension::class)\ninternal class LengthConsoleProcessorTest {\n\n  @Test\n  fun `should log consumed length event to console`(capturedOutput: CapturedOutput) {\n    val lengthConsoleProcessor = LengthConsoleProcessor()\n\n    lengthConsoleProcessor.process(LengthEvent(53))\n\n    assertThat(capturedOutput.out).contains(\"Consumed length [53]\")\n  }\n}\n```\n\n## Wiring it all together\n\nWe only need to create all the required instances naming them accordingly to the binder configuration:\n\n```kotlin\n@Configuration\nclass MyApplicationConfiguration {\n\n  @Bean\n  fun textFluxProducer() = TextFluxProducer()\n\n  @Bean\n  fun textProducer(textProducer: TextFluxProducer): () -\u003e Flux\u003cTextEvent\u003e = textProducer\n\n  @Bean\n  fun textLengthProcessor(): Function\u003cKStream\u003cString, TextEvent\u003e, KStream\u003cString, LengthEvent\u003e\u003e =\n    TextLengthProcessor()\n\n  @Bean\n  fun lengthConsumer(lengthProcessor: LengthProcessor): (LengthEvent) -\u003e Unit =\n    LengthStreamConsumer(lengthProcessor)\n\n  @Bean\n  fun lengthConsoleProcessor() = LengthConsoleProcessor()\n}\n```\n\nPlease note that:\n* The three Spring Cloud functions defined in [application.yml](src/main/resources/application.yml) will be bound **by name** to the beans **textProducer**, **textLengthProcessor** and **lengthConsumer**.\n  * For the Kafka binder ones, **textProducer** and **lengthConsumer**, we have to **define them explicitly as Kotlin lambdas** (required by **KotlinLambdaToFunctionAutoConfiguration**).\n    * If we were using **Java** we should use `java.util.function` types: `Supplier` and `Consumer`.\n  * For the Kafka Stream binder one, **textLengthProcessor**, we have to **define it explicitly as a `java.util.function.Function`**, there is no support for **Kotlin** lambdas yet (check **KafkaStreamsFunctionBeanPostProcessor**).\n* Beans **textFluxProducer** and **textProducer** return the same instance ...\n  * We need **textFluxProducer** to inject it whenever a **TextProducer** interface is needed (the **TextController** for example).\n  * We need **textProducer** to bind it to the **textProducer** Spring Cloud function required by the Kafka binder.\n\nAnd that is it, now [MyApplicationIntegrationTest](src/test/kotlin/com/rogervinas/multibinder/MyApplicationIntegrationTest.kt) should work! 🤞\n\nHappy coding! 💙\n\n## Test this demo\n\n```shell\n./gradlew test\n```\n\n## Run this demo\n\n```shell\ndocker-compose up -d\n./gradlew bootRun\ndocker-compose down\n```\n\nUse [curl](https://curl.se/) or [httpie](https://httpie.io/) or any other tool you like to POST a /text request:\n```shell\ncurl -v -X POST http://localhost:8080/text \\\n -H \"Content-Type: text/plain\" \\\n -d \"Toto, I have a feeling we are not in Kansas anymore\"\n```\n\nUse [kcat](https://github.com/edenhill/kcat) to produce/consume to/from Kafka:\n```shell\n# consume\nkcat -b localhost:9092 -C -t pub.texts -f '%k %s\\n'\nkcat -b localhost:9092 -C -t pub.lengths -f '%k %s\\n'\n\n# produce\necho 'key1:{\"text\":\"I feel the need - the need for speed!\"}' \\\n | kcat -b localhost:9092 -P -t pub.texts -K:\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frogervinas%2Fspring-cloud-stream-multibinder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frogervinas%2Fspring-cloud-stream-multibinder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frogervinas%2Fspring-cloud-stream-multibinder/lists"}