{"id":13475244,"url":"https://github.com/niqdev/kafka-scala-examples","last_synced_at":"2025-10-31T22:30:26.042Z","repository":{"id":39985258,"uuid":"144261950","full_name":"niqdev/kafka-scala-examples","owner":"niqdev","description":"Examples of Avro, Kafka, Schema Registry, Kafka Streams, Interactive Queries, KSQL, Kafka Connect in Scala","archived":false,"fork":false,"pushed_at":"2024-08-19T17:15:27.000Z","size":11808,"stargazers_count":63,"open_issues_count":23,"forks_count":15,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-02-07T21:16:04.847Z","etag":null,"topics":["avro","interactive-queries","kafka","kafka-connect","kafka-scala","kafka-streams","ksql","schema-registry"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/niqdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-10T08:51:15.000Z","updated_at":"2024-09-10T16:16:03.000Z","dependencies_parsed_at":"2024-01-13T18:24:52.687Z","dependency_job_id":"f7f6364f-26d0-4023-8e87-0b032192150a","html_url":"https://github.com/niqdev/kafka-scala-examples","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niqdev%2Fkafka-scala-examples","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niqdev%2Fkafka-scala-examples/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niqdev%2Fkafka-scala-examples/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/niqdev%2Fkafka-scala-examples/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/niqdev","download_url":"https://codeload.github.com/niqdev/kafka-scala-examples/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239236679,"owners_count":19604912,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","interactive-queries","kafka","kafka-connect","kafka-scala","kafka-streams","ksql","schema-registry"],"created_at":"2024-07-31T16:01:18.675Z","updated_at":"2025-10-31T22:30:26.004Z","avatar_url":"https://github.com/niqdev.png","language":"Scala","funding_links":[],"categories":["Scala"],"sub_categories":[],"readme":"# kafka-scala-examples\n\n[![Build Status][travis-image]][travis-url]\n[![Scala Steward badge][scala-steward-image]][scala-steward-url]\n\n[travis-image]: https://travis-ci.org/niqdev/kafka-scala-examples.svg?branch=master\n[travis-url]: https://travis-ci.org/niqdev/kafka-scala-examples\n[scala-steward-image]: https://img.shields.io/badge/Scala_Steward-helping-blue.svg?style=flat\u0026logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAA4AAAAQCAMAAAARSr4IAAAAVFBMVEUAAACHjojlOy5NWlrKzcYRKjGFjIbp293YycuLa3pYY2LSqql4f3pCUFTgSjNodYRmcXUsPD/NTTbjRS+2jomhgnzNc223cGvZS0HaSD0XLjbaSjElhIr+AAAAAXRSTlMAQObYZgAAAHlJREFUCNdNyosOwyAIhWHAQS1Vt7a77/3fcxxdmv0xwmckutAR1nkm4ggbyEcg/wWmlGLDAA3oL50xi6fk5ffZ3E2E3QfZDCcCN2YtbEWZt+Drc6u6rlqv7Uk0LdKqqr5rk2UCRXOk0vmQKGfc94nOJyQjouF9H/wCc9gECEYfONoAAAAASUVORK5CYII=\n[scala-steward-url]: https://scala-steward.org\n\nExamples in Scala of\n* [Avro](#avro)\n* [Kafka](#kafka)\n* [Schema Registry](#schema-registry)\n* [Kafka Streams](#kafka-streams)\n    - with [cats](https://github.com/niqdev/kafka-scala-examples#demo-3)\n    - with [ZIO](https://github.com/niqdev/kafka-scala-examples#demo-4), see also [zio-kafka-streams](https://github.com/niqdev/zio-kafka-streams)\n* Interactive Queries *TODO*\n    - with REST/http4s\n    - with GraphQl/Caliban\n* [KSQL](#ksql)\n* [Kafka Connect](#kafka-connect)\n* [Extra](#extra)\n\nLocal environment\n\n```bash\n# start locally\n# - zookeeper\n# - kafka\n# - kafka-rest\n# - kafka-ui\n# - schema-registry\n# - schema-registry-ui\n# - ksql-server\n# - ksql-cli\n# - kafka-connect\n# - kafka-connect-ui\ndocker-compose up\n\n# (mac|linux) view kafka ui\n[open|xdg-open] http://localhost:8000\n\n# (mac|linux) view schema-registry ui\n[open|xdg-open] http://localhost:8001\n\n# (mac|linux) view kafka-connect ui\n[open|xdg-open] http://localhost:8002\n\n# cleanup\ndocker-compose down -v\n```\n\nIf containers are crashing, make sure you have enough resources\n\n```bash\n# verify memory and cpu usage\ndocker ps -q | xargs docker stats --no-stream\n\n# verify status\ndocker inspect \u003cCONTAINER_NAME\u003e | jq '.[].State'\n```\n\n## avro\n\n**Description**\n\n[Avro](https://avro.apache.org/docs/current/gettingstartedjava.html) serialization and deserialization examples of\n* `SpecificRecord` code generation with [sbt-avro](https://github.com/sbt/sbt-avro) [[source](avro/src/main/scala/com/kafka/demo/original/AvroCodeGeneration.scala)|[test](avro/src/test/scala/com/kafka/demo/original/AvroCodeGenerationSpec.scala)]\n* `GenericRecord` [[source](avro/src/main/scala/com/kafka/demo/original/AvroGenericRecord.scala)|[test](avro/src/test/scala/com/kafka/demo/original/AvroGenericRecordSpec.scala)]\n* [avro4s](https://github.com/sksamuel/avro4s) [[source](avro/src/main/scala/com/kafka/demo/avro4s/Avro4sExample.scala)|[test](avro/src/test/scala/com/kafka/demo/avro4s/Avro4sExampleSpec.scala)]\n* Java/Scala libraries compatibility [[test](avro/src/test/scala/com/kafka/demo/LibraryCompatibilitySpec.scala)]\n\n**Demo**\n\n```bash\n# console\nsbt avro/console\n\n# generate avro classes\n# avro/target/scala-2.12/classes/com/kafka/demo/User.class\nsbt clean avro/compile\n\n# test\nsbt clean avro/test\n```\n\n### Resources\n\n* [Data Serialization and Evolution](https://docs.confluent.io/current/avro.html)\n* [Three Reasons Why Apache Avro Data Serialization is a Good Choice](https://blog.cloudera.com/blog/2011/05/three-reasons-why-apache-avro-data-serialization-is-a-good-choice-for-openrtb)\n* [Schema evolution in Avro, Protocol Buffers and Thrift](http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html)\n* [Avro Introduction for Big Data and Data Streaming Architectures](http://cloudurable.com/blog/avro/index.html)\n* [Stream Avro Records into Kafka using Avro4s and Akka Streams Kafka](https://abhsrivastava.github.io/2017/10/02/Stream-Avro-Records-into-Kafka)\n* [Kafka, Spark and Avro](https://aseigneurin.github.io/2016/03/02/kafka-spark-avro-kafka-101.html)\n\n## kafka\n\n**Description**\n\n[Kafka](https://kafka.apache.org/documentation) apis example of\n* `KafkaProducer` [[doc](https://kafka.apache.org/20/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html)|[source](kafka/src/main/scala/com/kafka/demo/original/Producer.scala)]\nand `KafkaConsumer` [[doc](https://kafka.apache.org/20/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html)|[source](kafka/src/main/scala/com/kafka/demo/original/Consumer.scala)]\nclients\n* [CakeSolutions](https://github.com/cakesolutions/scala-kafka-client)\n`KafkaProducer` [[source](kafka/src/main/scala/com/kafka/demo/cakesolutions/Producer.scala)|[test](kafka/src/test/scala/com/kafka/demo/cakesolutions/KafkaSpec.scala)]\nand `KafkaConsumer` [[source](kafka/src/main/scala/com/kafka/demo/cakesolutions/Consumer.scala)|[test](kafka/src/test/scala/com/kafka/demo/cakesolutions/KafkaSpec.scala)]\nclients\n\n**Demo**\n\n```bash\n# access kafka\ndocker exec -it local-kafka bash\n\n# create topic\n# convention \u003cMESSAGE_TYPE\u003e.\u003cDATASET_NAME\u003e.\u003cDATA_NAME\u003e\n# example [example.no-schema.original|example.no-schema.cakesolutions]\nkafka-topics --zookeeper zookeeper:2181 \\\n  --create --if-not-exists --replication-factor 1 --partitions 1 --topic \u003cTOPIC_NAME\u003e\n\n# delete topic\nkafka-topics --zookeeper zookeeper:2181 \\\n  --delete --topic \u003cTOPIC_NAME\u003e\n\n# view topic\nkafka-topics --zookeeper zookeeper:2181 --list \nkafka-topics --zookeeper zookeeper:2181 --describe --topic \u003cTOPIC_NAME\u003e\n\n# view topic offset\nkafka-run-class kafka.tools.GetOffsetShell \\\n  --broker-list kafka:9092 \\\n  --time -1 \\\n  --topic \u003cTOPIC_NAME\u003e\n\n# list consumer groups\nkafka-consumer-groups --bootstrap-server kafka:9092 --list\n\n# view consumer group offset\nkafka-consumer-groups \\\n  --bootstrap-server kafka:9092 \\\n  --group \u003cGROUP_NAME\u003e \\\n  --describe\n\n# reset consumer group offset\nkafka-consumer-groups \\\n  --bootstrap-server kafka:9092 \\\n  --group \u003cGROUP_NAME\u003e \\\n  --topic \u003cTOPIC_NAME\u003e \\\n  --reset-offsets \\\n  --to-earliest \\\n  --execute\n\n# console producer\nkafka-console-producer --broker-list kafka:9092 --topic \u003cTOPIC_NAME\u003e\nkafkacat -P -b 0 -t \u003cTOPIC_NAME\u003e\n\n# console consumer\nkafka-console-consumer --bootstrap-server kafka:9092 --topic \u003cTOPIC_NAME\u003e --from-beginning\nkafkacat -C -b 0 -t \u003cTOPIC_NAME\u003e\n\n# producer example\nsbt \"kafka/runMain com.kafka.demo.original.Producer\"\nsbt \"kafka/runMain com.kafka.demo.cakesolutions.Producer\"\n\n# consumer example\nsbt \"kafka/runMain com.kafka.demo.original.Consumer\"\nsbt \"kafka/runMain com.kafka.demo.cakesolutions.Consumer\"\n\n# test\nsbt clean kafka/test\nsbt \"test:testOnly *KafkaSpec\"\n```\n\n### Resources\n\n* DevOps [Kafka](https://niqdev.github.io/devops/kafka)\n* [Kafka topic naming conventions](https://medium.com/@criccomini/how-to-paint-a-bike-shed-kafka-topic-naming-conventions-1b7259790073)\n* [Should you put several event types in the same Kafka topic?](http://martin.kleppmann.com/2018/01/18/event-types-in-kafka-topic.html)\n* [How to choose the number of topics/partitions in a Kafka cluster?](https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster)\n* [Kafka Partitioning](https://simplydistributed.wordpress.com/2016/12/13/kafka-partitioning)\n* [The Log: What every software engineer should know about real-time data's unifying abstraction](https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying)\n* [Apache Kafka: 8 things to check before going live](https://ariskk.com/kafka-8-things)\n* [Apache Kafka vs Apache Pulsar](https://digitalis.io/blog/kafka/apache-kafka-vs-apache-pulsar)\n\n## schema-registry\n\n**Description**\n\n* Confluent's Schema Registry [API](https://docs.confluent.io/current/schema-registry/docs/api.html)\nand [examples](https://docs.confluent.io/current/schema-registry/docs/using.html#common-sr-usage-examples)\n* Console [examples](https://docs.confluent.io/current/schema-registry/docs/serializer-formatter.html#formatter)\n\n```bash\n# register schema\n# convention \u003cTOPIC_NAME\u003e-key or \u003cTOPIC_NAME\u003e-value\nhttp -v POST :8081/subjects/example.with-schema.simple-value/versions \\\n  Accept:application/vnd.schemaregistry.v1+json \\\n  schema='{\"type\":\"string\"}'\n\n# import schema from file\nhttp -v POST :8081/subjects/example.with-schema.user-value/versions \\\n  Accept:application/vnd.schemaregistry.v1+json \\\n  schema=@avro/src/main/avro/user.avsc\n\n# export schema to file\nhttp :8081/subjects/example.with-schema.user-value/versions/latest \\\n  | jq -r '.schema|fromjson' \\\n  | tee avro/src/main/avro/user-latest.avsc\n\n# list subjects\nhttp -v :8081/subjects\n\n# list subject's versions\nhttp -v :8081/subjects/example.with-schema.simple-value/versions\n\n# fetch by version\nhttp -v :8081/subjects/example.with-schema.simple-value/versions/1\n\n# fetch by id\nhttp -v :8081/schemas/ids/1\n\n# test compatibility\nhttp -v POST :8081/compatibility/subjects/example.with-schema.simple-value/versions/latest \\\n  Accept:application/vnd.schemaregistry.v1+json \\\n  schema='{\"type\":\"string\"}'\n\n# delete version\nhttp -v DELETE :8081/subjects/example.with-schema.simple-value/versions/1\n\n# delete latest version\nhttp -v DELETE :8081/subjects/example.with-schema.simple-value/versions/latest\n\n# delete subject\nhttp -v DELETE :8081/subjects/example.with-schema.simple-value\n\n# stringify\njq tostring avro/src/main/avro/user.avsc\n```\n\n**Demo**\n\n* [`BaseKafkaSchemaRegistrySpec`](schema-registry/src/test/scala/com/kafka/demo/BaseKafkaSchemaRegistrySpec.scala) to test Kafka with SchemaRegistry\n\n* `SpecificRecord` with [sbt-avrohugger](https://github.com/julianpeeters/sbt-avrohugger)\n[[Producer](schema-registry/src/main/scala/com/kafka/demo/specific/Producer.scala)|[Consumer](schema-registry/src/main/scala/com/kafka/demo/specific/Consumer.scala)|[test](schema-registry/src/test/scala/com/kafka/demo/KafkaSchemaRegistrySpecificSpec.scala)]\n\n```bash\n# generate SpecificRecord classes under \"schema-registry/target/scala-2.12/src_managed/main/compiled_avro\"\nsbt clean schema-registry/avroScalaGenerateSpecific\n\n# (optional) create schema\nhttp -v POST :8081/subjects/example.with-schema.payment-key/versions \\\n  Accept:application/vnd.schemaregistry.v1+json \\\n  schema='{\"type\":\"string\"}'\nhttp -v POST :8081/subjects/example.with-schema.payment-value/versions \\\n  Accept:application/vnd.schemaregistry.v1+json \\\n  schema=@schema-registry/src/main/avro/Payment.avsc\n\n# access kafka\ndocker exec -it local-kafka bash\n\n# (optional) create topic\nkafka-topics --zookeeper zookeeper:2181 \\\n  --create --if-not-exists --replication-factor 1 --partitions 1 --topic example.with-schema.payment\n\n# console producer (binary)\nkafka-console-producer --broker-list kafka:9092 --topic example.with-schema.payment\n\n# console consumer (binary)\nkafka-console-consumer --bootstrap-server kafka:9092 --topic example.with-schema.payment\n\n# access schema-registry\ndocker exec -it local-schema-registry bash\n\n# avro console producer\n# example \"MyKey\",{\"id\":\"MyId\",\"amount\":10}\nkafka-avro-console-producer --broker-list kafka:29092 \\\n  --topic example.with-schema.payment \\\n  --property schema.registry.url=http://schema-registry:8081 \\\n  --property parse.key=true \\\n  --property key.separator=, \\\n  --property key.schema='{\"type\":\"string\"}' \\\n  --property value.schema='{\"namespace\":\"io.confluent.examples.clients.basicavro\",\"type\":\"record\",\"name\":\"Payment\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"}]}'\n\n# avro console consumer\nkafka-avro-console-consumer --bootstrap-server kafka:29092 \\\n  --topic example.with-schema.payment \\\n  --property schema.registry.url=http://schema-registry:8081 \\\n  --property schema.id.separator=: \\\n  --property print.key=true \\\n  --property print.schema.ids=true \\\n  --property key.separator=, \\\n  --from-beginning\n\n# producer example\nsbt \"schema-registry/runMain com.kafka.demo.specific.Producer\"\n\n# consumer example\nsbt \"schema-registry/runMain com.kafka.demo.specific.Consumer\"\n\n# tests\nsbt \"schema-registry/test:testOnly *KafkaSchemaRegistrySpecificSpec\"\n```\n\n* `GenericRecord` with [CakeSolutions](https://github.com/cakesolutions/scala-kafka-client)\n[[Producer](schema-registry/src/main/scala/com/kafka/demo/generic/Producer.scala)|[Consumer](schema-registry/src/main/scala/com/kafka/demo/generic/Consumer.scala)] and schema evolution [test](schema-registry/src/test/scala/com/kafka/demo/KafkaSchemaRegistryGenericSpec.scala)\n\n```bash\n# producer example\nsbt \"schema-registry/runMain com.kafka.demo.generic.Producer\"\n\n# consumer example\nsbt \"schema-registry/runMain com.kafka.demo.generic.Consumer\"\n\n# tests\nsbt \"schema-registry/test:testOnly *KafkaSchemaRegistryGenericSpec\"\n```\n\n### Resources\n\n* [Serializing data efficiently with Apache Avro and dealing with a Schema Registry](https://www.sderosiaux.com/articles/2017/03/02/serializing-data-efficiently-with-apache-avro-and-dealing-with-a-schema-registry)\n* [Kafka, Avro Serialization and the Schema Registry](http://cloudurable.com/blog/kafka-avro-schema-registry/index.html)\n* [Kafka, Streams and Avro serialization](https://msayag.github.io/Kafka)\n* [Avro and the Schema Registry](https://aseigneurin.github.io/2018/08/02/kafka-tutorial-4-avro-and-schema-registry.html)\n* [Producing and Consuming Avro Messages over Kafka in Scala](https://medium.com/@lukasgrasse/producing-and-consuming-avro-messages-over-kafka-in-scala-edc26d69298c)\n\n**Alternatives**\n\n* [schema-repo](https://github.com/schema-repo/schema-repo)\n* Hortonworks [Registry](https://registry-project.readthedocs.io/en/latest)\n\n\u003e TODO\n\n* generic + schema evolution\n* ovotech\n* multi-schema\n* formulation\n\n## kafka-streams\n\n**Description**\n\n[Kafka Streams](https://kafka.apache.org/documentation/streams) apis examples\n\n**Demo-1**\n\n* `ToUpperCaseApp` [[source](streams/src/main/scala/com/kafka/demo/streams/ToUpperCaseApp.scala)|[test](streams/src/test/scala/com/kafka/demo/streams/ToUpperCaseSpec.scala)]\n\n```bash\n# access kafka\ndocker exec -it local-kafka bash\n\n# create topic\n# example [example.to-upper-case-app.input|example.to-upper-case-app.output]\nkafka-topics --zookeeper zookeeper:2181 \\\n  --create --if-not-exists --replication-factor 1 --partitions 1 --topic \u003cTOPIC_NAME\u003e\n\n# ToUpperCaseApp example (input topic required)\nsbt \"streams/runMain com.kafka.demo.streams.ToUpperCaseApp\"\n\n# produce\nkafka-console-producer --broker-list kafka:9092 \\\n  --topic example.to-upper-case-app.input\n\n# consume\nkafka-console-consumer --bootstrap-server kafka:9092 \\\n  --topic example.to-upper-case-app.output\n\n# test\nsbt clean streams/test\n```\n\n**Demo-2**\n\nTested with [embedded-kafka](https://github.com/embeddedkafka/embedded-kafka) and [embedded-kafka-schema-registry](https://github.com/embeddedkafka/embedded-kafka-schema-registry)\n\n* `JsonToAvroApp` [[source](https://github.com/niqdev/kafka-scala-examples/blob/master/streams-json-avro/src/main/scala/com/kafka/demo/JsonToAvroApp.scala)|[test](https://github.com/niqdev/kafka-scala-examples/blob/master/streams-json-avro/src/test/scala/com/kafka/demo/JsonToAvroSpec.scala)]\n\n```bash\n# access kafka\ndocker exec -it local-kafka bash\n\n# create topic\n# example [json.streams-json-to-avro-app.input|avro.streams-json-to-avro-app.output]\nkafka-topics --zookeeper zookeeper:2181 \\\n  --create --if-not-exists --replication-factor 1 --partitions 1 --topic \u003cTOPIC_NAME\u003e\n\n# produce (default StringSerializer)\nkafka-console-producer \\\n  --broker-list kafka:9092 \\\n  --property \"parse.key=true\" \\\n  --property \"key.separator=:\" \\\n  --property \"key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer\" \\\n  --property \"value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer\" \\\n  --topic \u003cTOPIC_NAME\u003e\n\n# consume (default StringDeserializer)\nkafka-console-consumer \\\n  --bootstrap-server kafka:9092 \\\n  --from-beginning \\\n  --property \"print.key=true\" \\\n  --property \"key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer\" \\\n  --property \"value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer\" \\\n  --topic \u003cTOPIC_NAME\u003e\n\n# access schema-registry\ndocker exec -it local-schema-registry bash\n\n# consume avro\nkafka-avro-console-consumer --bootstrap-server kafka:29092 \\\n  --property schema.registry.url=http://schema-registry:8081 \\\n  --property schema.id.separator=: \\\n  --property print.key=true \\\n  --property print.schema.ids=true \\\n  --property key.separator=, \\\n  --from-beginning \\\n  --topic \u003cTOPIC_NAME\u003e\n\n# JsonToAvroApp example (input topic required)\nsbt \"streams-json-avro/runMain com.kafka.demo.JsonToAvroApp\"\n\n# test\nsbt clean streams-json-avro/test\n```\n\nExample\n```\n# json\nmykey:{\"valueInt\":42,\"valueString\":\"foo\"}\n\n# log\n[json.streams-json-to-avro-app.input]: mykey, JsonModel(42,foo)\n[avro.streams-json-to-avro-app.output]: KeyAvroModel(mykey), ValueAvroModel(42,FOO)\n```\n\n#### Demo-3\n\n\u003e TODO\n\n* `CatsKafkaStreamsApp` [[source](https://github.com/niqdev/kafka-scala-examples/blob/master/cats-kafka-streams/src/main/scala/com/kafka/demo/CatsKafkaStreamsApp.scala)]\n\n```bash\n# run app\nsbt -jvm-debug 5005 \"cats-kafka-streams/runMain com.kafka.demo.CatsKafkaStreamsApp\"\n```\n\n#### Demo-4\n\n\u003e TODO\n\n* `ZioKafkaStreamsApp` [[source](https://github.com/niqdev/kafka-scala-examples/blob/master/zio-kafka-streams/src/main/scala/com/kafka/demo/ZioKafkaStreamsApp.scala)]\n\n```bash\n# run app\nsbt -jvm-debug 5005 \"zio-kafka-streams/runMain com.kafka.demo.ZioKafkaStreamsApp\"\n```\n\n### Resources\n\n* [Introducing Kafka Streams: Stream Processing Made Simple](https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple)\n* [Unifying Stream Processing and Interactive Queries in Apache Kafka](https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka)\n* [Of Streams and Tables in Kafka and Stream Processing](https://www.michael-noll.com/blog/2018/04/05/of-stream-and-tables-in-kafka-and-stream-processing-part1)\n* [How to use Apache Kafka to transform a batch pipeline into a real-time one](https://medium.com/@stephane.maarek/how-to-use-apache-kafka-to-transform-a-batch-pipeline-into-a-real-time-one-831b48a6ad85)\n* [Functional Programming with Kafka Streams and Scala](https://itnext.io/a-cqrs-approach-with-kafka-streams-and-scala-49bfa78e4295)\n* [Enabling Exactly-Once in Kafka Streams](https://www.confluent.io/blog/enabling-exactly-kafka-streams)\n\n## ksql\n\n**Description**\n\n* [KSQL](https://docs.confluent.io/current/ksql/docs/index.html)\n* [ksqlDB](https://docs.ksqldb.io/en/latest)\n* [Udemy Course](https://www.udemy.com/kafka-ksql)\n\nSetup Kafka\n```bash\n# access kafka\ndocker exec -it local-kafka bash\n\n# create topic\nkafka-topics --zookeeper zookeeper:2181 \\\n  --create --if-not-exists --replication-factor 1 --partitions 1 --topic USER_PROFILE\n\n# produce sample data\nkafka-console-producer --broker-list kafka:9092 --topic USER_PROFILE \u003c\u003c EOF\n{\"userid\": 1000, \"firstname\": \"Alison\", \"lastname\": \"Smith\", \"countrycode\": \"GB\", \"rating\": 4.7}\nEOF\n\n# consume\nkafka-console-consumer --bootstrap-server kafka:9092 --topic USER_PROFILE --from-beginning\n```\n\nAccess KSQL CLI\n\n* using the server\n    ```bash\n    # access ksql-server\n    docker exec -it local-ksql-server bash\n    \n    # start ksql cli\n    ksql http://ksql-server:8088\n    ```\n\n* using a local instance\n    ```bash\n    # connect to local cli\n    docker exec -it local-ksql-cli ksql http://ksql-server:8088\n    ```\n\n* using a temporary instance\n    ```bash\n    # connect to remote server\n    docker run --rm \\\n      --network=kafka-scala-examples_local_kafka_network \\\n      -it confluentinc/cp-ksql-cli http://ksql-server:8088\n    ```\n\nExecute SQL statements\n```bash\n# create stream\nCREATE STREAM user_profile (\\\n  userid INT, \\\n  firstname VARCHAR, \\\n  lastname VARCHAR, \\\n  countrycode VARCHAR, \\\n  rating DOUBLE \\\n  ) WITH (KAFKA_TOPIC = 'USER_PROFILE', VALUE_FORMAT = 'JSON');\n\n# verify stream\nlist streams;\ndescribe user_profile;\n\n# query stream\nSELECT userid, firstname, lastname, countrycode, rating FROM user_profile EMIT CHANGES;\n```\n\nExpect the consumer and the query to show the generated data\n```bash\n# generate data\ndocker run --rm \\\n  -v $(pwd)/local/ksql:/datagen \\\n  --network=kafka-scala-examples_local_kafka_network \\\n  -it confluentinc/ksql-examples ksql-datagen \\\n  bootstrap-server=kafka:29092 \\\n  schemaRegistryUrl=http://schema-registry:8081 \\\n  schema=datagen/user_profile.avro \\\n  format=json \\\n  topic=USER_PROFILE \\\n  key=userid \\\n  maxInterval=5000 \\\n  iterations=100\n```\n\n## kafka-connect\n\n* [Kafka Connect](https://docs.confluent.io/current/connect/index.html)\n* Confluent's Kafka Connect [API](https://docs.confluent.io/current/connect/references/restapi.html) and [connectors](https://docs.confluent.io/current/connect/managing/connectors.html)\n* [Udemy Course](https://www.udemy.com/course/kafka-connect)\n* [Kafka Connect Fundamentals](https://www.confluent.io/blog/kafka-connect-tutorial)\n\nSetup PostgreSQL locally\n```bash\n# create shared network\ndocker-compose up\n\n# start postgres\ndocker-compose -f docker-compose.postgres.yml up\n\n# (mac|linux) view postgres ui\n# [schema=public|database=postgres|username=postgres|password=postgres]\n[open|xdg-open] http://localhost:8080\n```\n\nSetup connectors\n\n* `kafka-connect-spooldir` [[confluent](https://docs.confluent.io/current/connect/kafka-connect-spooldir/index.html) | [official](https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-spooldir/index.html)]\n* `kafka-connect-jdbc` [[confluent](https://docs.confluent.io/current/connect/kafka-connect-jdbc/index.html)]\n\n```bash\n# list connector\nhttp -v :8083/connectors\n\n# init data to generate schema\ncp local/connect/data/resources-0.txt.orig local/connect/data/resources-0.txt\n\n# setup spooldir source connector\nhttp -v --json POST :8083/connectors \u003c local/connect/config/source-spooldir-connector.json\n\n# ingest data\necho \"{\\\"accountId\\\":\\\"123\\\",\\\"resourceType\\\":\\\"XXX\\\",\\\"value\\\":\\\"X1\\\"}\" \u003e local/connect/data/resources-1.txt\n\n# setup jdbc sink connector\n# topic = SCHEMA.DATABASE = \"public.postgres\"\nhttp -v --json POST :8083/connectors \u003c local/connect/config/sink-jdbc-connector.json\n\n# verify data\ndocker exec -it local-postgres bash -c \"psql -U postgres postgres\"\nselect * from public.postgres;\n\n# cleanup\ndocker-compose -f docker-compose.postgres.yml down -v\n```\n\n## extra\n\n### Resources\n\n* *Old [presentation](https://gitpitch.com/niqdev/kafka-scala-examples)*\n* [What is the actual role of Zookeeper in Kafka?](https://www.quora.com/What-is-the-actual-role-of-Zookeeper-in-Kafka-What-benefits-will-I-miss-out-on-if-I-don%E2%80%99t-use-Zookeeper-and-Kafka-together/answer/Gwen-Shapira)\n* [How to use Apache Kafka to transform a batch pipeline into a real-time one](https://medium.com/@stephane.maarek/how-to-use-apache-kafka-to-transform-a-batch-pipeline-into-a-real-time-one-831b48a6ad85)\n* [Kafka Partitioning](https://simplydistributed.wordpress.com/2016/12/13/kafka-partitioning)\n* [Should you put several event types in the same Kafka topic?](http://martin.kleppmann.com/2018/01/18/event-types-in-kafka-topic.html)\n* [How to choose the number of topics/partitions in a Kafka cluster?](https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster)\n* [Docker Tips and Tricks with KSQL and Kafka](https://rmoff.net/2018/12/15/docker-tips-and-tricks-with-ksql-and-kafka)\n* [Introduction to Topic Log Compaction in Apache Kafka](https://medium.com/swlh/introduction-to-topic-log-compaction-in-apache-kafka-3e4d4afd2262)\n\n### Tools\n\n* [Kafka Streams Topology Visualizer](https://zz85.github.io/kafka-streams-viz) (online)\n* [kafkacat](https://github.com/edenhill/kafkacat)\n* [Kafka-Utils](https://github.com/Yelp/kafka-utils)\n* [Insulator](https://andrewinci.github.io/Insulator) (GUI)\n* [KLoadGen](https://github.com/corunet/kloadgen)\n* [Kowl](https://github.com/cloudhut/kowl)\n* [UI for Apache Kafka](https://github.com/provectus/kafka-ui)\n* [Cruise Control](https://github.com/linkedin/cruise-control)\n* [CMAK](https://github.com/yahoo/CMAK)\n\n### Companies\n\n* [Confluent](https://www.confluent.io)\n* [Lenses](https://lenses.io)\n* [Conduktor](https://www.conduktor.io)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fniqdev%2Fkafka-scala-examples","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fniqdev%2Fkafka-scala-examples","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fniqdev%2Fkafka-scala-examples/lists"}