{"id":21948112,"url":"https://github.com/evolution-gaming/skafka","last_synced_at":"2026-01-17T23:47:41.890Z","repository":{"id":28963752,"uuid":"119857127","full_name":"evolution-gaming/skafka","owner":"evolution-gaming","description":"Scala wrapper for kafka consumer and producer","archived":false,"fork":false,"pushed_at":"2025-04-01T14:04:40.000Z","size":1136,"stargazers_count":44,"open_issues_count":53,"forks_count":20,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-04-02T12:08:29.615Z","etag":null,"topics":["cats","cats-effect","kafka","scala"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/evolution-gaming.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-01T15:49:17.000Z","updated_at":"2025-03-27T16:01:34.000Z","dependencies_parsed_at":"2024-03-20T11:41:24.558Z","dependency_job_id":"7051c17c-a231-496b-b7ad-07e77b5ec1ed","html_url":"https://github.com/evolution-gaming/skafka","commit_stats":null,"previous_names":[],"tags_count":144,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evolution-gaming%2Fskafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evolution-gaming%2Fskafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evolution-gaming%2Fskafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evolution-gaming%2Fskafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/evolution-gaming","download_url":"https://codeload.github.com/evolution-gaming/skafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248054194,"owners_count":21039952,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cats","cats-effect","kafka","scala"],"created_at":"2024-11-29T05:12:02.039Z","updated_at":"2026-01-17T23:47:41.844Z","avatar_url":"https://github.com/evolution-gaming.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Skafka\n[![Build Status](https://github.com/evolution-gaming/skafka/workflows/CI/badge.svg)](https://github.com/evolution-gaming/skafka/actions?query=workflow%3ACI)\n[![Coverage Status](https://coveralls.io/repos/github/evolution-gaming/skafka/badge.svg?branch=master)](https://coveralls.io/github/evolution-gaming/skafka?branch=master)\n[![Codacy Badge](https://app.codacy.com/project/badge/Grade/2373830b1e624ed39d27a644dca63d17)](https://app.codacy.com/gh/evolution-gaming/skafka/dashboard?utm_source=gh\u0026utm_medium=referral\u0026utm_content=\u0026utm_campaign=Badge_grade)\n[![Version](https://img.shields.io/badge/version-click-blue)](https://evolution.jfrog.io/artifactory/api/search/latestVersion?g=com.evolutiongaming\u0026a=skafka_2.13\u0026repos=public)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellowgreen.svg)](https://opensource.org/licenses/MIT)\n\nScala wrapper for [kafka-clients v3.4.0](https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients/3.4.0)\n\n## Motivation\n\nKafka provides an official Java client out of the box, which could be used from\nScala code without any additional modifications.\n\nThe main disadvantage of using an official client directly is that it implies\na very specific threading model to the application. I.e. the consumer is not\nthread safe and also expects a rebalance listener to do the operations in the\nsame thread.\n\nThis makes wrapping a client with [Cats Effect](https://typelevel.org/cats-effect/)\nclasses a bit more complicated than just calling `IO { consumer.poll() }` unless\nthis is the only call, which is expected to be used.\n\nSkafka does exactly that: a very thin wrapper over official Kafka client to\nprovide a ready-made Cats Effect API and handle some corner cases concerning\n[ConsumerRebalanceListener](https://kafka.apache.org/34/javadoc/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.html) calls.\n\nComparing to more full-featured libraries such as\n[FS2 Kafka](https://fd4s.github.io/fs2-kafka), it might be a little bit more\nreliable, because there is little code/logic to hide the accidenital bugs in.\n\nTo summarize:\n1. If it suits your goals (i.e. you only ever need to do `consumer.poll()`\nwithout acting on rebalance etc.) then using an official Kafka client directly,\noptionally, wrapping all the calls with `cats.effect.IO`, is a totally fine idea.\n2. If more complicated integration to Cats Effect is required, i.e.\n_ConsumerRebalanceListener_ is going to be used then consider using _Skafka_.\n3. If streaming with [FS2](https://fs2.io) is required or any other features\nthe library provides then _FS2 Kafka_ could be a good choice. Note, that it is\nless trivial then _Skafka_ and may contain more bugs on top of the official\nKafka client.\n\n## Key features\n\n1. It provides null-less Scala apis for [Producer](skafka/src/main/scala/com/evolutiongaming/skafka/producer/Producer.scala) \u0026 [Consumer](skafka/src/main/scala/com/evolutiongaming/skafka/consumer/Consumer.scala)\n\n2. Makes it easy to use your effect monad with help of [cats-effect](https://typelevel.org/cats-effect/)\n\n3. Blocking calls are being executed on provided `ExecutionContext`.\n\n4. Simple `case class` based configuration\n\n5. Support of [typesafe config](https://github.com/lightbend/config)    \n\n\n## Producer usage example\n\n```scala\nval producer = Producer.of[IO](config, ecBlocking)\nval metadata: IO[RecordMetadata] = producer.use { producer =\u003e\n  val record = ProducerRecord(topic = \"topic\", key = \"key\", value = \"value\") \n  producer.send(record).flatten \n}\n```\n\n## Consumer usage example\n\n```scala\nval consumer = Consumer.of[IO, String, String](config, ecBlocking)\nval records: IO[ConsumerRecords[String, String]] = consumer.use { consumer =\u003e \n  for {\n    _       \u003c- consumer.subscribe(Nel(\"topic\"), None)\n    records \u003c- consumer.poll(100.millis)\n  } yield records \n}\n```\n\n## Java client metrics example\n\nThe example below demonstrates creation of `Consumer`, but same can be done for `Producer` as well.\n\n\u003e :warning: using `ConsumerMetricsOf.withJavaClientMetrics`  (or its alternative `metrics.exposeJavaClientMetrics`) \n\u003e registers new Prometheus collector under the hood. Please use unique prefixes for each collector \n\u003e to avoid duplicated metrics in Prometheus (i.e. runtime exception on registration). \n\u003e Prefix can be set as parameter in: `ConsumerMetricsOf.withJavaClientMetrics(prometheus, Some(\"the_prefix\"))`\n\n```scala\nimport ConsumerMetricsOf.*\n\nval config: ConsumerConfig = ???\nval prometheus: CollectorRegistry = ???\nval metrics: ConsumerMetrics[IO] = ???\n\nfor {\n  metrics   \u003c- metrics.exposeJavaClientMetrics(prometheus)\n  consumerOf = ConsumerOf.apply1(metrics1.some) \n  consumer  \u003c- consumerOf(config)\n} yield ???\n```\n\n## Setup\n\n```scala\naddSbtPlugin(\"com.evolution\" % \"sbt-artifactory-plugin\" % \"0.0.2\")\n\nlibraryDependencies += \"com.evolutiongaming\" %% \"skafka\" % \"15.0.0\"\n``` \n\n## Notes\n\nWhile _Skafka_ provides an ability to use `ConsumerRebalanceListener`\nfunctionality, not all of the method calls are supported.\n\nSee the following PRs for more details:\nhttps://github.com/evolution-gaming/skafka/pull/150\nhttps://github.com/evolution-gaming/skafka/pull/122\n\nTo our latest knowledge neither `FS2 Kafka` supports all of the\nmethods / functionality.\n\n\n## Release process\nThe release process is based on Git tags and makes use of [sbt-dynver](https://github.com/sbt/sbt-dynver) to automatically obtain the version from the latest Git tag. The flow is defined in `.github/workflows/release.yml`.  \nA typical release process is as follows:\n1. Create and push a new Git tag. The version should be in the format `vX.Y.Z` (example: `v4.1.0`). Example: `git tag v4.1.0 \u0026\u0026 git push origin v4.1.0`\n2. Create a new release in GitHub. Go to the `Releases` page, click `Draft a new release`, select `Choose a tag`, pick the tag you just created\n3. Press `Generate release notes`. Release title will be automatically filled with the tag name. Change the description if needed\n4. Press `Publish release`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevolution-gaming%2Fskafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevolution-gaming%2Fskafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevolution-gaming%2Fskafka/lists"}