{"id":17248761,"url":"https://github.com/nomisrev/kotlin-kafka","last_synced_at":"2025-04-05T23:11:46.053Z","repository":{"id":37576835,"uuid":"428349598","full_name":"nomisRev/kotlin-kafka","owner":"nomisRev","description":"Kafka bindings for Kotlin `suspend`, and Kafka streaming operators for KotlinX Flow.","archived":false,"fork":false,"pushed_at":"2025-03-18T10:43:58.000Z","size":2570,"stargazers_count":108,"open_issues_count":19,"forks_count":10,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-29T22:09:52.813Z","etag":null,"topics":["hacktoberfest","kafka","kafka-client","kafka-consumer","kafka-producer","kotlin","kotlin-coroutines","kotlinx-coroutines","reactive","streaming"],"latest_commit_sha":null,"homepage":"https://nomisRev.github.io/kotlin-kafka/","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nomisRev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-15T16:58:00.000Z","updated_at":"2025-03-24T10:30:20.000Z","dependencies_parsed_at":"2023-02-10T21:15:38.469Z","dependency_job_id":"7a7cbda3-6684-484a-a913-19fc58f56a4d","html_url":"https://github.com/nomisRev/kotlin-kafka","commit_stats":{"total_commits":211,"total_committers":3,"mean_commits":70.33333333333333,"dds":0.4928909952606635,"last_synced_commit":"278c82073dd095aafae133f1251b8c3a63573dc7"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nomisRev%2Fkotlin-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nomisRev%2Fkotlin-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nomisRev%2Fkotlin-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nomisRev%2Fkotlin-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nomisRev","download_url":"https://codeload.github.com/nomisRev/kotlin-kafka/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247411239,"owners_count":20934653,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest","kafka","kafka-client","kafka-consumer","kafka-producer","kotlin","kotlin-coroutines","kotlinx-coroutines","reactive","streaming"],"created_at":"2024-10-15T06:42:05.114Z","updated_at":"2025-04-05T23:11:46.018Z","avatar_url":"https://github.com/nomisRev.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"Module kotlin-kafka\n\n[![Maven Central](https://img.shields.io/maven-central/v/io.github.nomisrev/kotlin-kafka?color=4caf50\u0026label=latest%20release)](https://maven-badges.herokuapp.com/maven-central/io.github.nomisrev/kotlin-kafka)\n\n\u003c!--- TOC --\u003e\n\n* [Rationale](#rationale)\n* [Goals](#goals)\n* [Adding Dependency](#adding-dependency)\n* [Example](#example)\n\n\u003c!--- END --\u003e\n\n## Rationale\n\nAt the time of starting this repository I didn't find any bindings between Kafka SDK and Kotlin suspension, or KotlinX Coroutines Flow.\nThese operators should be implemented low-level, so they can guarantee correct cancellation support, and high optimised runtimes.\n\nSome important aspects of Kafka are tricky to implement with the \"low-level\" Kafka API,\nespecially properly streaming records from Kafka and correctly committing them.\nAdditional complexity is involved in this process, more details [here](https://tuleism.github.io/blog/2021/parallel-backpressured-kafka-consumer/).\n\nTo solve these problems a couple of projects in the JVM already exist:\n - [Alpakka Kafka](https://github.com/akka/alpakka-kafka)\n - [reactor-kafka](https://github.com/reactor/reactor-kafka)\n\nThere was no implementation for KotlinX Coroutines Flow,\nyou can however quite easily use reactor-kafka with [KotlinX Coroutines Reactor bindings](https://github.com/Kotlin/kotlinx.coroutines/blob/master/reactive/kotlinx-coroutines-reactor/README.md).\n\nThis project implements the same strategies as [reactor-kafka] directly on top of KotlinX Coroutines to benefit from **all** their benefits,\nand to open the door to potentially becoming a Kotlin MPP library in the future.\n\n## Goals\n\n- Lean Core library built on top of Kotlin Std \u0026 KotlinX Coroutines\n- Extensions to easily operate over the Kafka SDK with KotlinX Coroutines and `suspend`.\n- Flow based operators, so you can easily compose KotlinX Flow based Kafka programs\n- Strong guarantees about committing record offsets, and performance optimisations in regard to re-balancing/partitioning.\n- example for testing Kafka with Test Containers in Kotlin.\n\n## Adding Dependency\n\nSimply add the following dependency as `implementation` in the `build.gradle` dependencies` block.\n\n```groovy\ndependencies {\n  implementation(\"io.github.nomisrev:kotlin-kafka:0.3.0\")\n}\n```\n\n## Example\n\n\u003c!--- INCLUDE\nimport arrow.continuations.SuspendApp\nimport io.github.nomisRev.kafka.Admin\nimport io.github.nomisRev.kafka.AdminSettings\nimport io.github.nomisRev.kafka.createTopic\nimport io.github.nomisRev.kafka.imap\nimport io.github.nomisRev.kafka.map\nimport io.github.nomisRev.kafka.publisher.Acks\nimport io.github.nomisRev.kafka.publisher.KafkaPublisher\nimport io.github.nomisRev.kafka.publisher.PublisherSettings\nimport io.github.nomisRev.kafka.receiver.AutoOffsetReset\nimport io.github.nomisRev.kafka.receiver.KafkaReceiver\nimport io.github.nomisRev.kafka.receiver.ReceiverSettings\nimport kotlinx.coroutines.Dispatchers\nimport kotlinx.coroutines.flow.map\nimport kotlinx.coroutines.flow.take\nimport kotlinx.coroutines.launch\nimport org.apache.kafka.clients.admin.NewTopic\nimport org.apache.kafka.clients.producer.ProducerRecord\nimport org.apache.kafka.common.serialization.IntegerDeserializer\nimport org.apache.kafka.common.serialization.IntegerSerializer\nimport org.apache.kafka.common.serialization.StringDeserializer\nimport org.apache.kafka.common.serialization.StringSerializer\nimport java.util.UUID\n--\u003e\n\n```kotlin\n@JvmInline\nvalue class Key(val index: Int)\n\n@JvmInline\nvalue class Message(val content: String)\n\nfun main(): Unit = SuspendApp {\n  val topicName = \"test-topic\"\n  val msgCount = 10\n  val kafka = Kafka.container\n\n  Admin(AdminSettings(kafka.bootstrapServers)).use { client -\u003e\n    client.createTopic(NewTopic(topicName, 1, 1))\n  }\n\n  launch(Dispatchers.IO) { // Send 20 messages, and then close the producer\n    val settings: PublisherSettings\u003cKey, Message\u003e = PublisherSettings(\n      kafka.bootstrapServers,\n      IntegerSerializer().imap { key: Key -\u003e key.index },\n      StringSerializer().imap { msg: Message -\u003e msg.content },\n      Acks.All\n    )\n    KafkaPublisher(settings).use { publisher -\u003e\n      publisher.publishScope {\n        (1..msgCount).forEach { index -\u003e\n          offer(ProducerRecord(topicName, Key(index), Message(\"msg: $index\")))\n        }\n      }\n    }\n  }\n\n  launch(Dispatchers.IO) { // Consume 20 messages as a stream, and then close the consumer\n    val settings: ReceiverSettings\u003cKey, Message\u003e = ReceiverSettings(\n      kafka.bootstrapServers,\n      IntegerDeserializer().map(::Key),\n      StringDeserializer().map(::Message),\n      groupId = UUID.randomUUID().toString(),\n      autoOffsetReset = AutoOffsetReset.Earliest\n    )\n    KafkaReceiver(settings)\n      .receive(topicName)\n      .take(msgCount)\n      .map { \"${it.key()} -\u003e ${it.value()}\" }\n      .collect(::println)\n  }\n}\n```\n\n\u003e You can get the full code [here](guide/example/example-readme-01.kt).\n\n```text\nKey(index=1) -\u003e Message(content=msg: 1)\nKey(index=2) -\u003e Message(content=msg: 2)\nKey(index=3) -\u003e Message(content=msg: 3)\nKey(index=4) -\u003e Message(content=msg: 4)\nKey(index=5) -\u003e Message(content=msg: 5)\nKey(index=6) -\u003e Message(content=msg: 6)\nKey(index=7) -\u003e Message(content=msg: 7)\nKey(index=8) -\u003e Message(content=msg: 8)\nKey(index=9) -\u003e Message(content=msg: 9)\nKey(index=10) -\u003e Message(content=msg: 10)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnomisrev%2Fkotlin-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnomisrev%2Fkotlin-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnomisrev%2Fkotlin-kafka/lists"}