{"id":20059560,"url":"https://github.com/monix/monix-kafka","last_synced_at":"2025-03-17T16:12:02.952Z","repository":{"id":11241043,"uuid":"68894798","full_name":"monix/monix-kafka","owner":"monix","description":"Monix integration with Kafka","archived":false,"fork":false,"pushed_at":"2024-08-12T23:47:36.000Z","size":583,"stargazers_count":123,"open_issues_count":41,"forks_count":38,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-02T13:11:21.457Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/monix.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-09-22T07:13:58.000Z","updated_at":"2024-05-17T16:45:12.000Z","dependencies_parsed_at":"2023-02-17T23:01:03.167Z","dependency_job_id":null,"html_url":"https://github.com/monix/monix-kafka","commit_stats":null,"previous_names":["monixio/monix-kafka"],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monix%2Fmonix-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monix%2Fmonix-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monix%2Fmonix-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monix%2Fmonix-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/monix","download_url":"https://codeload.github.com/monix/monix-kafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244066190,"owners_count":20392406,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T13:08:21.953Z","updated_at":"2025-03-17T16:12:02.930Z","avatar_url":"https://github.com/monix.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Monix-Kafka\n\n[![Build Status](https://travis-ci.org/monix/monix-kafka.svg?branch=master)](https://travis-ci.org/monix/monix-kafka)\n[![Maven Central](https://img.shields.io/maven-central/v/io.monix/monix-kafka-1x_2.12.svg)](https://search.maven.org/search?q=g:io.monix%20AND%20a:monix-kafka-1x_2.12)\n[![Scala Steward badge](https://img.shields.io/badge/Scala_Steward-helping-brightgreen.svg?style=flat\u0026logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAA4AAAAQCAMAAAARSr4IAAAAVFBMVEUAAACHjojlOy5NWlrKzcYRKjGFjIbp293YycuLa3pYY2LSqql4f3pCUFTgSjNodYRmcXUsPD/NTTbjRS+2jomhgnzNc223cGvZS0HaSD0XLjbaSjElhIr+AAAAAXRSTlMAQObYZgAAAHlJREFUCNdNyosOwyAIhWHAQS1Vt7a77/3fcxxdmv0xwmckutAR1nkm4ggbyEcg/wWmlGLDAA3oL50xi6fk5ffZ3E2E3QfZDCcCN2YtbEWZt+Drc6u6rlqv7Uk0LdKqqr5rk2UCRXOk0vmQKGfc94nOJyQjouF9H/wCc9gECEYfONoAAAAASUVORK5CYII=)](https://scala-steward.org)\n\n[![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/monix/monix?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\nMonix integration with Kafka\n\nWork in progress!\n\n## Table of Contents\n1. [Getting Started with Kafka 1.0.x or above](#getting-started-with-kafka-10x-or-above)\n2. [Getting Started with Kafka 0.11.x](#getting-started-with-kafka-011x)\n3. [Getting Started with Kafka 0.10.x](#getting-started-with-kafka-010x)\n4. [Getting Started with Kafka 0.9.x](#getting-started-with-kafka-09x)\n5. [Getting Started with Kafka 0.8.x (no longer supported)](#getting-started-with-kafka-08x)\n6. [Usage](#usage)\n7. [How can I contribute to Monix-Kafka?](#how-can-i-contribute-to-monix-kafka?)\n8. [Maintainers](#maintainers)\n9. [License](#license)\n\n## Getting Started with Kafka 1.0.x or above\n\nIn SBT:\n\n```scala\nlibraryDependencies += \"io.monix\" %% \"monix-kafka-1x\" % \"1.0.0-RC6\"\n```\n\nFor `kafka` versions higher than `1.0.x` also add a dependency override:\n\n```scala\ndependencyOverrides += \"org.apache.kafka\" % \"kafka-clients\" % \"2.1.0\"\n```\n\nOr in case you're interested in running the tests of this project, it\nnow supports embedded kafka for integration testing. You can simply run:\n\n```bash\nsbt kafka1x/test\n```\n\n## Getting Started with Kafka 0.11.x\n\nIn SBT:\n\n```scala\nlibraryDependencies += \"io.monix\" %% \"monix-kafka-11\" % \"1.0.0-RC6\"\n```\n\nOr in case you're interested in running the tests of this project, it\nnow supports embedded kafka for integration testing. You can simply run:\n\n```bash\nsbt kafka11/test\n```\n\n## Getting Started with Kafka 0.10.x\n\nIn SBT:\n\n```scala\nlibraryDependencies += \"io.monix\" %% \"monix-kafka-10\" % \"1.0.0-RC6\"\n```\n\nOr in case you're interested in running the tests of this project, it\nnow supports embedded kafka for integration testing. You can simply run:\n\n```bash\nsbt kafka10/test\n```\n\n## Getting Started with Kafka 0.9.x\n\nPlease note that `EmbeddedKafka` is not supported for Kafka `0.9.x`\n\nIn SBT:\n\n```scala\nlibraryDependencies += \"io.monix\" %% \"monix-kafka-9\" % \"1.0.0-RC6\"\n```\n\nOr in case you're interested in running the tests of this project,\nfirst download the Kafka server, version `0.9.x` from their\n[download page](https://kafka.apache.org/downloads.html) (note that\n`0.10.x` or higher do not work with `0.9`), then as the\n[quick start](https://kafka.apache.org/090/documentation.html#quickstart)\nsection says, open a terminal window and first start Zookeeper:\n\n```bash\nbin/zookeeper-server-start.sh config/zookeeper.properties\n```\n\nThen start Kafka:\n\n```bash\nbin/kafka-server-start.sh config/server.properties\n```\n\nCreate the topic we need for our tests:\n\n```bash\nbin/kafka-topics.sh --create --zookeeper localhost:2181 \\\n  --replication-factor 1 --partitions 1 \\\n  --topic monix-kafka-tests\n```\n\nAnd run the tests:\n\n```bash\nsbt kafka9/test\n```\n\n## Getting Started with Kafka 0.8.x\n\nPlease note that support for Kafka `0.8.x` is dropped and the last available version with this dependency is `0.14`.\n\nIn SBT:\n\n```scala\nlibraryDependencies += \"io.monix\" %% \"monix-kafka-8\" % \"0.14\"\n```\n\nOr in case you're interested in running the tests of this project,\nfirst download the Kafka server, version `0.8.x` from their\n[download page](https://kafka.apache.org/downloads.html) (note that\n`0.9.x` or higher do not work with `0.8`), then as the\n[quick start](https://kafka.apache.org/082/documentation.html#quickstart)\nsection says, open a terminal window and first start Zookeeper:\n\n```bash\nbin/zookeeper-server-start.sh config/zookeeper.properties\n```\n\nThen start Kafka:\n\n```bash\nbin/kafka-server-start.sh config/server.properties\n```\n\nCreate the topics we need for our tests:\n\n```bash\nbin/kafka-topics.sh --create --zookeeper localhost:2181 \\\n  --replication-factor 1 --partitions 1 \\\n  --topic monix-kafka-tests\nbin/kafka-topics.sh --create --zookeeper localhost:2181 \\\n  --replication-factor 1 --partitions 1 \\\n  --topic monix-kafka-manual-commit-tests\n```\n\nAnd run the tests:\n\n```bash\nsbt kafka8/test\n```\n\n## Usage\n\n### Producer\n\n```scala\nimport monix.kafka._\nimport monix.execution.Scheduler\n\nimplicit val scheduler: Scheduler = monix.execution.Scheduler.global\n\n// Init\nval producerCfg = KafkaProducerConfig.default.copy(\n  bootstrapServers = List(\"127.0.0.1:9092\")\n)\n\nval producer = KafkaProducer[String,String](producerCfg, scheduler)\n\n// For sending one message\nval recordMetadataF = producer.send(\"my-topic\", \"my-message\").runToFuture\n\n// For closing the producer connection\nval closeF = producer.close().runToFuture\n```\n\nCalling `producer.send` returns a [Task](https://monix.io/docs/3x/eval/task.html) of `Option[RecordMetadata]` which can then be run and transformed into a `Future`.\n\nIf the `Task` completes with `None` it means that `producer.send` method was called after the producer was closed and that the message wasn't successfully acknowledged by the Kafka broker. In case of the failure of the underlying Kafka client the producer will bubble up the exception and fail the `Task`.  All successfully delivered messages will complete with `Some[RecordMetadata]`.\n\nFor pushing an entire `Observable` to Apache Kafka:\n\n```scala\nimport monix.kafka._\nimport monix.execution.Scheduler\nimport monix.reactive.Observable\nimport org.apache.kafka.clients.producer.ProducerRecord\n\nimplicit val scheduler: Scheduler = monix.execution.Scheduler.global\n\n// Initializing the producer\nval producerCfg = KafkaProducerConfig.default.copy(\n  bootstrapServers = List(\"127.0.0.1:9092\")\n)\n\nval producer = KafkaProducerSink[String,String](producerCfg, scheduler)\n\n// Lets pretend we have this observable of records\nval observable: Observable[ProducerRecord[String,String]] = ???\n\nobservable\n  // on overflow, start dropping incoming events\n  .whileBusyDrop\n  // buffers into batches if the consumer is busy, up to a max size\n  .bufferIntrospective(1024)\n  // consume everything by pushing into Apache Kafka\n  .consumeWith(producer)\n  // ready, set, go!\n  .runToFuture\n```\n\n### Consumer\n\nThere are several ways for consuming from Apache Kafka (Version 0.11.x and above):\n\nConsumer which commits offsets itself:\n```scala\nimport monix.kafka._\n\nval consumerCfg = KafkaConsumerConfig.default.copy(\n  bootstrapServers = List(\"127.0.0.1:9092\"),\n  groupId = \"kafka-tests\"\n  // you can use this settings for At Most Once semantics:\n  // observableCommitOrder = ObservableCommitOrder.BeforeAck\n)\n\nval observable =\n  KafkaConsumerObservable[String,String](consumerCfg, List(\"my-topic\"))\n    .take(10000)\n    .map(_.value())\n```\n\nConsumer which allows you to commit offsets manually:\n```scala\nimport monix.kafka._\n\nval consumerCfg = KafkaConsumerConfig.default.copy(\n  bootstrapServers = List(\"127.0.0.1:9092\"),\n  groupId = \"kafka-tests\"\n)\n\nval observable =\n  KafkaConsumerObservable.manualCommit[String,String](consumerCfg, List(\"my-topic\"))\n    .map(message =\u003e message.record.value() -\u003e message.committableOffset)\n    .mapEval { case (value, offset) =\u003e performBusinessLogic(value).map(_ =\u003e offset) }\n    .bufferTimedAndCounted(1.second, 1000)\n    .mapEval(offsets =\u003e CommittableOffsetBatch(offsets).commitSync())\n```\n\nEnjoy!\n\n### Internal poll heartbeat\n\nStarting from Kafka _0.10.1.0_, there is `max.poll.interval.ms` setting that defines the maximum delay between\ninvocations of poll(), if it is not called in that interval, then the consumer is considered failed and  the group will rebalance in order \n    to reassign the partitions to another member.\n\nThis was an [issue](https://github.com/monix/monix-kafka/issues/101) in `monix-kafka`, \nsince poll is not called until all previous consumed ones were processed, so that slow downstream subscribers\n were in risk of being kicked off the consumer group indefinitely.\n\nIt is resolved in `1.0.0-RC8` by introducing an internal poll heartbeat interval\nthat runs in the background keeping the consumer alive. \n\n## How can I contribute to Monix-Kafka?\n\nWe welcome contributions to all projects in the Monix organization and would love\nfor you to help build Monix-Kafka. See our [contributor guide](./CONTRIBUTING.md) for\nmore information about how you can get involed.\n\n## Maintainers\n\nThe current maintainers (people who can merge pull requests) are:\n\n- Alexandru Nedelcu ([alexandru](https://github.com/alexandru))\n- Piotr Gawryś ([Avasil](https://github.com/Avasil))\n- Leandro Bolivar ([leandrob13](https://github.com/leandrob13))\n\n## License\n\nAll code in this repository is licensed under the Apache License,\nVersion 2.0.  See [LICENSE.txt](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonix%2Fmonix-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmonix%2Fmonix-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonix%2Fmonix-kafka/lists"}