{"id":18576852,"url":"https://github.com/niqdev/zio-kafka-streams","last_synced_at":"2025-04-10T09:30:39.614Z","repository":{"id":56360189,"uuid":"291442065","full_name":"niqdev/zio-kafka-streams","owner":"niqdev","description":"Kafka Streams for Scala with ZIO","archived":false,"fork":false,"pushed_at":"2021-07-21T21:46:47.000Z","size":151,"stargazers_count":13,"open_issues_count":2,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-24T19:08:27.377Z","etag":null,"topics":["avro4s","datagen","kafka","kafka-streams","kafka-streams-scala","schema-registry","zio","zio-kafka"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/niqdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-30T09:45:26.000Z","updated_at":"2024-03-27T09:17:08.000Z","dependencies_parsed_at":"2022-08-15T17:20:16.854Z","dependency_job_id":null,"html_url":"https://github.com/niqdev/zio-kafka-streams","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niqdev%2Fzio-kafka-streams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niqdev%2Fzio-kafka-streams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niqdev%2Fzio-kafka-streams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niqdev%2Fzio-kafka-streams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/niqdev","download_url":"https://codeload.github.com/niqdev/zio-kafka-streams/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248191615,"owners_count":21062537,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro4s","datagen","kafka","kafka-streams","kafka-streams-scala","schema-registry","zio","zio-kafka"],"created_at":"2024-11-06T23:26:52.773Z","updated_at":"2025-04-10T09:30:39.039Z","avatar_url":"https://github.com/niqdev.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# zio-kafka-streams\n\n[![Build Status][build-image]][build-url]\n\n[build-image]: https://travis-ci.org/niqdev/zio-kafka-streams.svg?branch=master\n[build-url]: https://travis-ci.org/niqdev/zio-kafka-streams\n\nWrite and test [Kafka Streams](https://docs.confluent.io/current/streams/developer-guide/index.html) applications using [ZIO](https://zio.dev) ~~and expose the internal state store directly via [GraphQL](https://ghostdogpr.github.io/caliban)~~\n\nAdd the following lines to your `build.sbt`\n\n```sbt\nlibraryDependencies ++= Seq(\n  \"???\" %% \"zio-kafka-streams\" % \"???\", // core DONE, incomplete api and tests\n  \"???\" %% \"zio-kafka-streams-api\" % \"???\", // TODO graphQL (?)\n  \"???\" %% \"zio-kafka-streams-testkit\" % \"???\" % Test, // core DONE, incomplete api and tests\n  \"???\" %% \"kafka-streams-serde\" % \"???\", // DONE, missing tests\n  \"???\" %% \"kafka-datagen\" % \"???\" // TODO incomplete\n)\n```\n\n:construction::construction::construction::construction::construction::construction::construction::construction::construction::construction:\n:warning: **Work in Progress** :warning:\n:construction::construction::construction::construction::construction::construction::construction::construction::construction::construction:\n\n* [Examples](#examples)\n* [TestKit](#testkit)\n* [Serdes](#serdes)\n* [DataGen](#datagen)\n* [Development](#development)\n* [TODO](#todo)\n\n## Examples\n\nExamples of how to write Kafka Streams applications using `ZStreamsBuilder`, `ZKStream` and `ZKTable`\n\n### ToUpperCase\n\nProbably the simplest Kafka Streams application you could think of\n```scala\nobject ToUpperCaseTopology {\n  // build the topology\n  lazy val topology: RIO[KafkaStreamsConfig with CustomConfig, Topology] =\n    for {\n      sourceTopic \u003c- CustomConfig.sourceTopic\n      sinkTopic   \u003c- CustomConfig.sinkTopic\n      topology    \u003c- ZStreamsBuilder { builder =\u003e\n        for {\n          // compose the topology using ZKStream and ZKTable\n          sourceStream \u003c- builder.stream[String, String](sourceTopic)\n          sinkStream   \u003c- sourceStream.mapValue(_.toUpperCase)\n          _            \u003c- sinkStream.to(sinkTopic)\n        } yield ()\n      }\n    } yield topology\n  // define the topology's layer\n  val layer: RLayer[ZEnv, KafkaStreamsTopology with KafkaStreamsConfig] =\n    ToUpperCaseConfig.layer \u003e+\u003e KafkaStreamsTopology.make(topology)\n}\n// setup runtime\nobject ToUpperCaseApp extends KafkaStreamsApp(ToUpperCaseTopology.layer)\n```\n\nHow to run the example\n```bash\n# start kafka\nmake local-up\n\n# create source topic\nmake topic-create name=example.source.v1\n\n# start application\nLOG_LEVEL=\"INFO\" sbt \"examples/runMain com.github.niqdev.ToUpperCaseApp\"\n\n# access kafka\ndocker exec -it local-kafka bash\n\n# publish messages\nkafka-console-producer --bootstrap-server kafka:9092 --topic example.source.v1\n\n# consume messages\nkafka-console-consumer --bootstrap-server kafka:9092 --topic example.sink.v1\n```\n\nComplete example of [ToUpperCaseApp](https://github.com/niqdev/zio-kafka-streams/blob/master/examples/src/main/scala/com/github/niqdev/ToUpperCaseApp.scala)\n\n### GitHubApp\n\nJoining Avro streams integrated with Schema Registry has never been so easy ;-)\n```scala\nobject GitHubTopology {\n  lazy val topology: RIO[KafkaStreamsConfig with CustomConfig with Logging, Topology] =\n    for {\n      config \u003c- KafkaStreamsConfig.config\n      _      \u003c- log.info(s\"Running ${config.applicationId}\")\n      _      \u003c- CustomConfig.prettyPrint.flatMap(values =\u003e log.info(values))\n      topics \u003c- CustomConfig.topics\n      topology \u003c- ZStreamsBuilder { builder =\u003e\n        for {\n          userStream         \u003c- builder.streamAvro[UserKey, UserValue](topics.userSource)\n          repositoryStream   \u003c- builder.streamAvro[RepositoryKey, RepositoryValue](topics.repositorySource)\n          ghUserStream       \u003c- userStream.mapKey(GitHubEventKey.fromUser)\n          ghRepositoryStream \u003c- repositoryStream.mapKey(GitHubEventKey.fromRepository)\n          ghUserTable        \u003c- ghUserStream.toTableAvro\n          ghRepositoryTable  \u003c- ghRepositoryStream.toTableAvro\n          gitHubTable        \u003c- ghUserTable.joinAvro(ghRepositoryTable)(GitHubEventValue.joinUserRepository)\n          gitHubStream       \u003c- gitHubTable.toStream\n          _                  \u003c- gitHubStream.toAvro(topics.gitHubSink)\n        } yield ()\n      }\n    } yield topology\n\n  val layer: RLayer[ZEnv, KafkaStreamsTopology with KafkaStreamsConfig] =\n    Logging.console() ++ GitHubConfig.envLayer \u003e+\u003e KafkaStreamsTopology.make(topology)\n}\nobject GitHubApp extends KafkaStreamsApp(GitHubTopology.layer)\n```\n\nHow to run the example\n```bash\n# start kafka\nmake local-up\n\n# create source topics\nmake topic-create-all\n\n# generate avsc and register schema\nmake schema-register-all\n\n# start application\nmake local-run\n```\n\nHow to publish messages locally\n```bash\n# format example data\nmake format-data-all\n\n# access schema-registry\ndocker exec -it local-schema-registry bash\n\n# export producer config (see below)\nSCHEMA_KEY_ID=XXX\nSCHEMA_VALUE_ID=YYY\nTOPIC_NAME=ZZZ\n\n# start avro producer\nkafka-avro-console-producer \\\n  --bootstrap-server kafka:29092 \\\n  --property schema.registry.url=\"http://schema-registry:8081\" \\\n  --property parse.key=true \\\n  --property key.schema=\"$(curl -s http://schema-registry:8081/schemas/ids/$SCHEMA_KEY_ID | jq -r .schema)\" \\\n  --property value.schema=\"$(curl -s http://schema-registry:8081/schemas/ids/$SCHEMA_VALUE_ID | jq -r .schema)\" \\\n  --property key.separator=::: \\\n  --topic $TOPIC_NAME\n```\n\nHow to consume messages locally\n```bash\n# access schema-registry\ndocker exec -it local-schema-registry bash\n\n# export consumer config (see below)\nTOPIC_NAME=XYZ\n\n# start avro consumer\nkafka-avro-console-consumer \\\n  --bootstrap-server kafka:29092 \\\n  --property schema.registry.url=\"http://schema-registry:8081\" \\\n  --property schema.id.separator=: \\\n  --property print.key=true \\\n  --property print.schema.ids=true \\\n  --property key.separator=, \\\n  --topic $TOPIC_NAME \\\n  --from-beginning \\\n  --max-messages 10\n```\n\nConfigurations\n```bash\n# produce to \"user\" topic\nSCHEMA_KEY_ID=1\nSCHEMA_VALUE_ID=2\nTOPIC_NAME=example.user.v1\n\n# produce to \"repository\" topic\nSCHEMA_KEY_ID=3\nSCHEMA_VALUE_ID=4\nTOPIC_NAME=example.repository.v1\n\n# consume from topics\nTOPIC_NAME=example.user.v1\nTOPIC_NAME=example.repository.v1\nTOPIC_NAME=example.github.v1\n```\n\nComplete example of [GitHubApp](https://github.com/niqdev/zio-kafka-streams/blob/master/examples/src/main/scala/com/github/niqdev/GitHubApp.scala)\n\n## TestKit\n\nHow to test [ToUpperCaseTopology](https://github.com/niqdev/zio-kafka-streams/blob/master/examples/src/main/scala/com/github/niqdev/ToUpperCaseApp.scala) topology with `ZTestTopology`, `ZTestInput` and `ZTestOutput`\n```scala\n// LOG_LEVEL=WARN sbt \"test:testOnly *ToUpperCaseSpec\"\n\ntestM(\"topology\") {\n  for {\n    sourceTopic \u003c- CustomConfig.sourceTopic\n    sinkTopic   \u003c- CustomConfig.sinkTopic\n    outputValue \u003c- ZTestTopology.driver.use { driver =\u003e\n      for {\n        input  \u003c- driver.createInput[String, String](sourceTopic)\n        output \u003c- driver.createOutput[String, String](sinkTopic)\n        _      \u003c- input.produceValue(\"myValue\")\n        value  \u003c- output.consumeValue\n      } yield value\n    }\n  } yield assert(outputValue)(equalTo(\"MYVALUE\"))\n}.provideSomeLayerShared(testLayer)\n```\n\nMore examples in the [tests](https://github.com/niqdev/zio-kafka-streams/tree/master/modules/tests/src/test/scala) module\n\n## Serdes\n\n`kafka-streams-serde` is an independent module without ZIO dependencies useful to build [Serdes](https://docs.confluent.io/current/streams/developer-guide/datatypes.html) with your favourite effect system\n\nExample of how to autoderive Avro serde for keys and values integrated with Confluent [Schema Registry](https://docs.confluent.io/current/schema-registry/index.html) leveraging [avro4s](https://github.com/sksamuel/avro4s)\n```scala\nimport kafka.streams.serde._\n\nfinal case class DummyValue(string: String)\nobject DummyValue {\n  final implicit val dummyValueAvroCodec: AvroCodec[DummyValue] =\n    AvroCodec.genericValue[DummyValue]\n}\n```\n\nFor more examples with [refined](https://github.com/fthomas/refined), [newtype](https://github.com/estatico/scala-newtype), [enumeratum](https://github.com/lloydmeta/enumeratum) and custom types see the [schema](https://github.com/niqdev/zio-kafka-streams/tree/master/examples/src/main/scala/com/github/niqdev/schema) package\n\nExample of how to build a syntax with [Cats Effect](https://typelevel.org/cats-effect) using `Record` and `AvroRecord`\n```scala\nobject syntax {\n  final implicit def streamsBuilderSyntax[F[_]](builder: StreamsBuilder): StreamsBuilderOps[F] =\n    new StreamsBuilderOps(builder)\n}\n\nfinal class StreamsBuilderOps[F[_]](private val builder: StreamsBuilder) extends AnyVal {\n  def streamF[K, V](\n    topic: String,\n    schemaRegistry: String\n  )(implicit F: Sync[F], C: AvroRecordConsumed[K, V]): F[KStream[K, V]] =\n    F.delay(builder.stream(topic)(C.consumed(schemaRegistry)))\n}\n```\n\nComplete example of [KafkaStreamsCatsApp](https://github.com/niqdev/zio-kafka-streams/blob/master/examples/src/main/scala/com/github/niqdev/KafkaStreamsCatsApp.scala)\n\n## DataGen\n\n\u003e TODO\n\n```bash\nLOG_LEVEL=INFO sbt \"examples/runMain com.github.niqdev.SimpleKafkaGenApp\"\n```\n\n## Development\n\n```bash\n# start containers in background\n# zookeeper|kafka|kafka-rest|kafka-ui|schema-registry|schema-registry-ui|kowl\nmake local-up\n\n# stop all containers\nmake local-down\n\n# cli\nmake topic-list\nmake topic-describe name=\u003cTOPIC_NAME\u003e\nmake topic-create name=\u003cTOPIC_NAME\u003e\nmake topic-delete name=\u003cTOPIC_NAME\u003e\nmake topic-offset name=\u003cTOPIC_NAME\u003e\nmake group-list\nmake group-offset name=\u003cGROUP_NAME\u003e\n\n# [mac|linux] kafka ui\n[open|xdg-open] http://localhost:8000\n# [mac|linux] schema-registry ui\n[open|xdg-open] http://localhost:8001\n# [mac|linux] kowl (another ui)\n[open|xdg-open] http://localhost:8002\n```\n\nHow to build an event manually\n\n```bash\n# example\nEVENT_NAME=\u003cmy-event-name\u003e\n\n# stringify json event\ncat local/data/$EVENT_NAME.json | jq -c | jq -R\n\n# format message \"\u003cjson_key\u003e:::\u003cjson_value\u003e\"\necho \"$(cat local/data/$EVENT_NAME-key.json | jq -c):::$(cat local/data/$EVENT_NAME-value.json | jq -c)\" \u003e local/data/$EVENT_NAME-event.txt\n\n# produce sample message\nmake produce-avro \\\n  schema-key-id=1 \\\n  schema-value-id=2 \\\n  topic-name=\u003cTOPIC_NAME\u003e \\\n  event-name=$EVENT_NAME\n```\n\nHow to interact with `schema-registry` apis\n\n```bash\nTOPIC_NAME=\u003c???\u003e\n\n# verify registered schema\ncurl -s -X GET localhost:8081/subjects/$TOPIC_NAME-value/versions\ncurl -s -X GET localhost:8081/subjects/$TOPIC_NAME-value/versions/1 | jq\n# extract avsc\ncurl -s -X GET localhost:8081/subjects/$TOPIC_NAME-value/versions/1 | jq -r \".schema\" | jq\n```\n\n## TODO\n\n* [ ] zio-prelude e.g. [newtype](https://github.com/zio/zio-prelude/blob/d34b5c0e74557edd8709f7c45b40297bf4280d77/src/main/scala/zio/prelude/NewtypeModule.scala) or refined ?\n* [ ] kafkacat docker\n    - `docker run --rm --name kafkacat edenhill/kafkacat:1.6.0`\n* [ ] json serde with circe/zio-json + xml serde (?)\n* [ ] interop-cats\n* [ ] api with Caliban (pagination + subscriptions)\n    - [Kafka Streams Interactive Queries](https://docs.confluent.io/current/streams/developer-guide/interactive-queries.html)\n* [ ] metrics with Prometheus\n* [ ] helm chart StatefulSet\n* [ ] convert GenerateSchema into sbt plugin\n* [ ] replace `kafka-streams-scala` with plain Java or zio-kafka?\n* [ ] topology description with [eisner](https://github.com/laserdisc-io/eisner)\n* [ ] Scala 3 compatibility\n\n\u003c!--\n\n# TODO resources\nhttps://docs.confluent.io/current/streams/developer-guide/test-streams.html\nhttps://www.confluent.io/blog/testing-kafka-streams\nhttps://www.confluent.io/blog/test-kafka-streams-with-topologytestdriver\nhttps://www.confluent.io/blog/stream-processing-part-2-testing-your-streaming-application\nhttps://medium.com/bakdata/fluent-kafka-streams-tests-e641785171ec\nhttps://github.com/embeddedkafka/embedded-kafka-schema-registry\nhttps://github.com/confluentinc/kafka-streams-examples/blob/5.5.1-post/src/test/java/io/confluent/examples/streams/GenericAvroIntegrationTest.java\n\n--\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fniqdev%2Fzio-kafka-streams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fniqdev%2Fzio-kafka-streams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fniqdev%2Fzio-kafka-streams/lists"}