{"id":16177865,"url":"https://github.com/hifly81/kafka-examples","last_synced_at":"2025-04-07T15:06:30.311Z","repository":{"id":37389746,"uuid":"145122216","full_name":"hifly81/kafka-examples","owner":"hifly81","description":"Practical examples with Apache Kafka®","archived":false,"fork":false,"pushed_at":"2025-03-20T15:22:43.000Z","size":125450,"stargazers_count":75,"open_issues_count":3,"forks_count":15,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-31T14:11:34.193Z","etag":null,"topics":["avro","avro-schema","avroserializer","confluent","confluent-kafka","flink","graalvm","kafka","kafka-client","kafka-consumer","kafka-consumers","kafka-events","kafka-installation","kafka-producer","kafka-producers","kafka-streams","kafka-topic","ksql","ktable","microprofile"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hifly81.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":"CONTRIBUTING.adoc","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-17T13:13:03.000Z","updated_at":"2025-03-25T08:19:08.000Z","dependencies_parsed_at":"2023-10-02T17:50:26.883Z","dependency_job_id":"8d0838f2-1a84-44fb-a62d-182386ebfb10","html_url":"https://github.com/hifly81/kafka-examples","commit_stats":{"total_commits":486,"total_committers":5,"mean_commits":97.2,"dds":"0.028806584362139898","last_synced_commit":"aaba090629dece35f07a1d552ab637172610e858"},"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hifly81%2Fkafka-examples","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hifly81%2Fkafka-examples/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hifly81%2Fkafka-examples/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hifly81%2Fkafka-examples/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hifly81","download_url":"https://codeload.github.com/hifly81/kafka-examples/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247675596,"owners_count":20977376,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","avro-schema","avroserializer","confluent","confluent-kafka","flink","graalvm","kafka","kafka-client","kafka-consumer","kafka-consumers","kafka-events","kafka-installation","kafka-producer","kafka-producers","kafka-streams","kafka-topic","ksql","ktable","microprofile"],"created_at":"2024-10-10T05:10:55.798Z","updated_at":"2025-04-07T15:06:30.292Z","avatar_url":"https://github.com/hifly81.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"= Practical examples with Apache Kafka®\n:author: Giovanni Marigi\n:email: gio.marigi@gmail.com\n:revdate: March 20, 2025\n:revnumber: 1.2.1\n:version-label!:\n:toc: left\n:toclevels: 3\n\n\n\u003c\u003c\u003c\n// page break\n\n\n== General Information\n\nThis repository contains various examples designed to demonstrate the functionality of link:https://kafka.apache.org[Apache Kafka®].\nThe examples are mostly extracted from more complex projects and should not be considered complete or ready to be used in a production environment, unless after a serious refactoring and test work.\n\n\u003c\u003c\u003c\n\n== Export Documentation\n\nClone this repository:\n\n[source,bash]\n----\ngit clone git@github.com:hifly81/kafka-examples.git\n----\n\n\nInstall link:https://asciidoctor.org[_asciidoctor_]:\n\nLinux:\n\n[source,bash]\n----\ngem install asciidoctor --pre\n----\n\nMac:\n\n[source,bash]\n----\nbrew install asciidoctor\n----\n\nExport documentation:\n\n[source,bash]\n----\n# PDF\nasciidoctor-pdf README.adoc\n\n# HTML\nasciidoctor README.html\n----\n\n\u003c\u003c\u003c\n\n== Contributions\n\n - link:https://github.com/hifly81/kafka-examples/blob/master/CONTRIBUTING.adoc[How to Contribute]\n - link:https://star-history.com/#hifly81/kafka-examples\u0026Date[Star History Chart]\n\n== License\n\n - link:https://github.com/hifly81/kafka-examples/blob/master/LICENSE[MIT License]\n\n\u003c\u003c\u003c\n\n== Installation with Docker\n\nOfficial documentation on how to install Docker on Linux/Mac/Windows is available at this link:\nhttps://docs.docker.com/engine/install/\n\n=== Prerequisites\n\nList of software required on your local machine to run the examples:\n\n - _curl_\n - _wget_\n - _openssl_\n - _Java SE 17 or 21_\n - _keytool_ from Java distribution\n - _Apache Maven 3.x_\n - _Go Programming language_ _(for proxy example)_\n - _Python_ _(for python clients)_\n\n=== Docker Images Version\n\nDefault image version for required components is listed in file link:.env[.env]\n\nIf you needed to change the docker image version for the specific components, just update file link:.env[.env].\n\n=== Docker Images details\n\nApache Kafka® docker images are downloaded from Docker Hub link:https://hub.docker.com/r/apache/kafka[_apache/kafka_] and are based on Apache Kafka® version 3.8.x).\n\n\n=== Single-node cluster +\n\nTo run a single-node cluster (KRaft controller and Broker node combined) using Docker, run the link:docker-compose.yml[docker-compose.yml] file available in the root directory. It also contains a container with link:https://github.com/edenhill/kcat[kcat]:\n\n\n* broker: _apache/kafka_, listening on port _9092_\n* kcat: _confluentinc/cp-kcat_\n\n==== Start\n\n[source,bash]\n----\nscripts/bootstrap.sh\n----\n\n==== Stop +\n\n[source,bash]\n----\nscripts/tear-down.sh\n----\n\n=== Multi nodes cluster +\n\nTo run a cluster with 3 KRaft controller and 3 Broker nodes using Docker on different containers, use the file in apache-kafka github repository:\nhttps://raw.githubusercontent.com/apache/kafka/trunk/docker/examples/jvm/cluster/isolated/plaintext/docker-compose.yml\n\n* kafka-1: _apache/kafka_, listening on port _29092_\n* kafka-2: _apache/kafka_, listening on port _39092_\n* kafka-3: _apache/kafka_, listening on port _49092_\n\n==== Start\n\n[source,bash]\n----\nscripts/bootstrap-isolated.sh\n----\n\n==== Stop +\n\n[source,bash]\n----\nscripts/tear-down-isolated.sh\n----\n\n\u003c\u003c\u003c\n\n== Installation on Kubernetes using Confluent For Kubernetes\n\n=== Prerequisites\n\nList of software required on your local machine to run the examples:\n\n- _helm_\n- _kubectl_\n- _Minikube_\n- _Confluent for Kubernetes operator (CFK)_: https://docs.confluent.io/operator/current/overview.html\n\n=== Minikube installation in Linux with KVM/QEMU\n\nFollow instructions for _ArchLinux_ (also tested with _Fedora_)\n\nhttps://dev.to/xs/kubernetes-minikube-with-qemu-kvm-on-arch-312a\n\n=== Minikube installation in Mac/Windows\n\nhttps://minikube.sigs.k8s.io/docs/start/\n\n=== Deployment of a cluster with Confluent for Kubernetes Operator (CFK)\n\nStart _Minikube_ with _kvm2_ driver (Linux):\n\n[source,bash]\n----\nminikube delete\nminikube config set driver kvm2\n----\n\nStart _Minikube_ with _docker_ driver (Mac):\n\n[source,bash]\n----\nminikube delete\nminikube config set driver docker\n----\n\n[source,bash]\n----\ntouch /tmp/config \u0026\u0026 export KUBECONFIG=/tmp/config\nminikube start --memory 16384 --cpus 4\n----\n\nCreate a _k8s_ namespace named _confluent_:\n\n[source,bash]\n----\nkubectl create namespace confluent\nkubectl config set-context --current --namespace confluent\n----\n\nAdd confluent repository to _helm_:\n\n[source,bash]\n----\nhelm repo add confluentinc https://packages.confluent.io/helm\nhelm repo update\n----\n\n==== Install confluent-for-kubernetes operator (latest version) from Confluent’s Helm repo:\n\n[source,bash]\n----\nhelm upgrade --install confluent-operator confluentinc/confluent-for-kubernetes --set kRaftEnabled=true\n----\n\n==== Deploy Confluent components\n\n1 controller, 3 brokers:\n\n[source,bash]\n----\nkubectl apply -f confluent-for-kubernetes/k8s/confluent-platform-reducted.yaml\n----\n\nList pods:\n\n[source,bash]\n----\nkubectl get pods\n\nNAME                                  READY   STATUS    RESTARTS   AGE\nconfluent-operator-665db446b7-j52rj   1/1     Running   0          6m35s\nkafka-0                               1/1     Running   0          65s\nkafka-1                               1/1     Running   0          65s\nkafka-2                               1/1     Running   0          65s\nkraftcontroller-0                     1/1     Running   0          5m5s\n----\n\nVerify events and pods:\n\n[source,bash]\n----\nwatch -n 5 \"kubectl get events --sort-by='.lastTimestamp'\"\nwatch -n 5 \"kubectl get pods\"\n----\n\nalternately, you can install additional Confluent components: 1 controller, 3 brokers, 1 connect, 1 ksqldb, 1 schema registry, 1 rest proxy:\n\n[source,bash]\n----\nkubectl apply -f confluent-for-kubernetes/k8s/confluent-platform.yaml\n----\n\n==== Apache Kafka® Operations\n\nTopic create:\n\n[source,bash]\n----\nkubectl exec --stdin --tty kafka-0 -- /bin/bash\nkafka-topics --bootstrap-server localhost:9092 --create --topic test-1\nexit\n----\n\nTopic list:\n\n[source,bash]\n----\nkubectl exec --stdin --tty kafka-0 -- /bin/bash\nkafka-topics --bootstrap-server localhost:9092 --list\nexit\n----\n\nTopic describe:\n\n[source,bash]\n----\nkubectl exec --stdin --tty kafka-0 -- /bin/bash\nkafka-topics --bootstrap-server localhost:9092 --topic test-1 --describe\nexit\n----\n\nProduce messages to Topic:\n\n[source,bash]\n----\nkubectl exec --stdin --tty kafka-0 -- /bin/bash\nkafka-producer-perf-test --num-records 1000000 --record-size 1000 --throughput -1 --topic test-1 --producer-props bootstrap.servers=localhost:9092\nexit\n----\n\nConsume messages from Topic:\n\n[source,bash]\n----\nkubectl exec --stdin --tty kafka-0 -- /bin/bash\nkafka-console-consumer --bootstrap-server localhost:9092 --topic test-1 --from-beginning\nexit\n----\n\n==== Tear Down\n\nShut down Confluent components and the data:\n\n[source,bash]\n----\nkubectl delete -f confluent-for-kubernetes/k8s/topic.yml\nkubectl delete -f confluent-for-kubernetes/k8s/producer.yml\nkubectl delete -f confluent-for-kubernetes/k8s/confluent-platform.yaml\nhelm delete confluent-operator\n----\n\nDelete namespace _confluent_:\n\n[source,bash]\n----\nkubectl delete namespace confluent\n----\n\nDelete minikube:\n\n[source,bash]\n----\nminikube delete\n----\n\n\u003c\u003c\u003c\n\n== Apache Kafka® producers\n\nSome implementations of Apache Kafka® producers.\n\nFolder link:kafka-producer/[kafka-producer/]\n\nExecute tests:\n\n[source,bash]\n----\ncd kafka-producer\nmvn clean test\n----\n\n=== String Serializer +\n\nIt uses link:https://kafka.apache.org/36/javadoc/org/apache/kafka/common/serialization/StringSerializer.html[org.apache.kafka.common.serialization.StringSerializer] class for key and value serialization.\n\nCreate topic _topic1_:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic topic1 --replication-factor 1 --partitions 1\n----\n\nProduce on topic _topic1_:\n\n[source,bash]\n----\ncd kafka-producer\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.producer.serializer.string.Runner\"\n----\n\n=== Json Serializer +\n\nIt uses link:https://kafka.apache.org/36/javadoc/org/apache/kafka/common/serialization/StringSerializer.html[org.apache.kafka.common.serialization.StringSerializer] for key serialization and a link:kafka-producer/src/main/java/org/hifly/kafka/demo/producer/serializer/json/CustomDataJsonSerializer.java[org.hifly.kafka.demo.producer.serializer.json.CustomDataJsonSerializer] for value serialization.\n\nCreate topic _test_custom_data_:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic test_custom_data --replication-factor 1 --partitions 1\n----\n\nProduce on topic _test_custom_data_:\n\n[source,bash]\n----\ncd kafka-producer\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.producer.serializer.json.Runner\"\n----\n\n=== Partitioner +\n\nIt uses a custom partitioner for keys.\n\nMessages with key _Mark_ go to partition 1, with key _Antony_ to partition 2 and with key _Paul_ to partition 3.\n\nCreate topic _demo-test_ with 3 partitions:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic demo-test --replication-factor 1 --partitions 3\n----\n\nProduce on topic _demo-test_:\n\n[source,bash]\n----\ncd kafka-producer\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.producer.partitioner.custom.Runner\"\n----\n\n=== Create Timestamp vs LogAppend Timestamp\n\n==== Create Timestamp\n\nMessage timestamp is set on headers when the message has been produced. This is the default behaviour, _Create Timestamp_.\n\nCreate topic _topic1_:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic topic1 --replication-factor 1 --partitions 1\n----\n\nConsume from _topic1_ and print out the message timestamp:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-consumer.sh --topic topic1 --bootstrap-server broker:9092 --from-beginning --property print.timestamp=true\n----\n\nProduce records on _topic1_:\n\n[source,bash]\n----\ndocker exec broker /opt/kafka/bin/kafka-producer-perf-test.sh --topic topic1 --num-records 1000 --record-size 100 --throughput -1 --producer-props bootstrap.servers=broker:9092\n----\n\nCheck consumer log for message timestamp:\n\n[source,bash]\n----\nCreateTime:1697359570614\tYQHHNEBSEPDNSEIFGAMSUJXKOLTXSPLGHDIOYZJFNIDSPWHZMKVJAXDBZFCOXYKYRJOGYKDESSJMOIIOWVKYUAVWJLXSEPPFEILV\nCreateTime:1697359570621\tBASHCGRHSYGIFSYLVGRXCDVABWWTRQZTMMPBAXGHEPHTASSORYKGVPFGQYJKINSZUJLXQUUDVALUSBFRSXNQHSDFDBAKQZZNTYXF\nCreateTime:1697359570621\tHYGDPYGNRETYAXIXXYQKMKURDSJYIZNEDAHVIVHCJAPGOBQLHUZTKIWTVFEHVYPNGHIDSERMARFXCPYFEPQMFDOTDPWNKMYRMFIA\nCreateTime:1697359570621\tBIQAWWOIFIAKNYFEPTPMIXPQAXFEIKUFFXIDHILBPCBTHWDRMALHFNDCRHAYVLLMRCKJIPNPKGWCIWQCHNHSFSCTYSAKSLVZCCAI\n----\n\n==== LogAppend Timestamp\n\nMessage timestamp is set on headers when the record arrives at the broker, the broker will override the timestamp of the producer record with its own timestamp (the current time of the broker environment) as it appends the record to the log.\n\nCreate topic _topic2_ with _message.timestamp.type=LogAppendTime_:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic topic2 --replication-factor 1 --partitions 1 --config message.timestamp.type=LogAppendTime\n----\n\nConsume from _topic2_ and print out the message timestamp:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-consumer.sh --topic topic2 --bootstrap-server broker:9092 --from-beginning --property print.timestamp=true\n----\n\nProduce records on _topic2_:\n\n[source,bash]\n----\ndocker exec broker /opt/kafka/bin/kafka-producer-perf-test.sh --topic topic2 --num-records 1000 --record-size 100 --throughput -1 --producer-props bootstrap.servers=broker:9092\n----\n\nCheck consumer log for message timestamp:\n\n[source,bash]\n----\nLogAppendTime:1697359857981\tYQHHNEBSEPDNSEIFGAMSUJXKOLTXSPLGHDIOYZJFNIDSPWHZMKVJAXDBZFCOXYKYRJOGYKDESSJMOIIOWVKYUAVWJLXSEPPFEILV\nLogAppendTime:1697359857981\tBASHCGRHSYGIFSYLVGRXCDVABWWTRQZTMMPBAXGHEPHTASSORYKGVPFGQYJKINSZUJLXQUUDVALUSBFRSXNQHSDFDBAKQZZNTYXF\nLogAppendTime:1697359857981\tHYGDPYGNRETYAXIXXYQKMKURDSJYIZNEDAHVIVHCJAPGOBQLHUZTKIWTVFEHVYPNGHIDSERMARFXCPYFEPQMFDOTDPWNKMYRMFIA\nLogAppendTime:1697359857981\tBIQAWWOIFIAKNYFEPTPMIXPQAXFEIKUFFXIDHILBPCBTHWDRMALHFNDCRHAYVLLMRCKJIPNPKGWCIWQCHNHSFSCTYSAKSLVZCCAI\n----\n\n=== Interceptor\n\nFolder link:interceptors/[interceptors/]\n\nThis example shows how to create a custom producer interceptor. Java class link:interceptors/src/main/java/org/hifly/kafka/interceptor/producer/CreditCardProducerInterceptor.java[_CreditCardProducerInterceptor_] will mask a sensitive info on producer record (credit card number).\n\nCompile and package:\n\n[source,bash]\n----\ncd interceptors\nmvn clean package\n----\n\nRun a consumer:\n\n[source,bash]\n----\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.interceptor.consumer.Runner\"\n----\n\nRun a producer:\n\n[source,bash]\n----\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.interceptor.producer.Runner\"\n----\n\nVerify output:\n\n[source,bash]\n----\nrecord is:XXXXXX\nTopic: test_custom_data - Partition: 0 - Offset: 1\n----\n\n=== Python Producer\n\nFolder link:kafka-python-producer/[kafka-python-producer/]\n\nInstall confluent-kafka-python lib link:https://docs.confluent.io/kafka-clients/python/current/overview.html[_confluent-kafka_]:\n\n[source,bash]\n----\npip install confluent-kafka\n----\n\nor:\n\n[source,bash]\n----\npython3 -m pip install confluent-kafka\n----\n\nCreate _kafka-topic_ topic:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic kafka-topic --replication-factor 1 --partitions 1\n----\n\nRun producer:\n\n[source,bash]\n----\ncd kafka-python-producer\npython producer.py\n----\n\n\u003c\u003c\u003c\n\n== Apache Kafka® consumers\n\nImplementation of a consumer that can be used with different deserializer classes (for key and value).\n\nFolder link:kafka-consumer/[kafka-consumer/]\n\nJava class link:kafka-consumer/src/main/java/org/hifly/kafka/demo/consumer/core/ConsumerInstance.java[_ConsumerInstance_] can be customized with:\n\n - clientId _(string)_\n - groupId _(string)_\n - topics _(string separated by comma)_\n - key deserializer class _(string)_\n - value deserializer class _(string)_\n - partition assignment strategy _(org.apache.kafka.clients.consumer.RangeAssignor|org.apache.kafka.clients.consumer.RoundRobinAssignor|org.apache.kafka.clients.consumer.StickyAssignor|org.apache.kafka.clients.consumer.CooperativeStickyAssignor)_\n - isolation.level _(read_uncommitted|read_committed)_\n - poll timeout _(ms)_\n - consume duration _(ms)_\n - autoCommit _(true|false)_\n - commit sync _(true|false)_\n - subscribe mode _(true|false)_\n\nTopics can be passed as argument 1 of the main program:\n\n[source,bash]\n----\n-Dexec.args=\"users,users_clicks\"\n----\n\nPartition assignment strategy can be passed as argument 2 of the main program:\n\n[source,bash]\n----\n-Dexec.args=\"users,users_clicks org.apache.kafka.clients.consumer.RoundRobinAssignor\"\n----\n\nGroup id can be passed as argument 3 of the main program:\n\n[source,bash]\n----\n-Dexec.args=\"users,users_clicks org.apache.kafka.clients.consumer.RoundRobinAssignor group-1\"\n----\n\nExecute tests:\n\n[source,bash]\n----\ncd kafka-consumer\nmvn clean test\n----\n\n=== String Deserializer\n\nIt uses link:https://kafka.apache.org/37/javadoc/org/apache/kafka/common/serialization/StringDeserializer.html[_org.apache.kafka.common.serialization.StringDeserializer_] for key and value deserialization. Default topic is _topic1_.\n\n[source,bash]\n----\ncd kafka-consumer\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.consumer.core.Runner\"\n----\n\nSend messages to _topic1_:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic topic1 --property \"parse.key=true\" --property \"key.separator=:\"\n\u003e Frank:1\n----\n\n=== Consumer Partition Assignor\n\n==== Range (default)\n\nCreate 2 topics, _users_ and _users_clicks_ with the same number of partitions:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic users --replication-factor 1 --partitions 3\n\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic users_clicks --replication-factor 1 --partitions 3\n----\n\nRun 2 consumer instances (2 different shells/terminals) belonging to the same consumer group and subscribed to _user_ and _user_clicks_ topics. Consumers uses\nlink:https://kafka.apache.org/37/javadoc/org/apache/kafka/clients/consumer/RangeAssignor.html[_org.apache.kafka.clients.consumer.RangeAssignor_] to distribute partition ownership.\n\n[source,bash]\n----\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.consumer.core.Runner\" -Dexec.args=\"users,users_clicks org.apache.kafka.clients.consumer.RangeAssignor range-group-app\"\n----\n\nSend messages to both topics using the same key (_Frank_):\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic users --property \"parse.key=true\" --property \"key.separator=:\"\n\u003e Frank:1\n\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic users_clicks --property \"parse.key=true\" --property \"key.separator=:\"\n\u003e Frank:1\n----\n\nVerify that the same consumer instance will read both messages.\n\n[source,bash]\n----\nGroup id group-XX - Consumer id: consumer-group-XX-1-421db3e2-6501-45b1-acfd-275ce8d18368 - Topic: users - Partition: 1 - Offset: 0 - Key: frank - Value: 1\nGroup id group-XX - Consumer id: consumer-group-XX-1-421db3e2-6501-45b1-acfd-275ce8d18368 - Topic: users_clicks - Partition: 1 - Offset: 0 - Key: frank - Value: 1\n----\n\n==== Round Robin\n\nCreate 2 topics, _users_ and _users_clicks_ with same number of partitions:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic users --replication-factor 1 --partitions 3\n\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic users_clicks --replication-factor 1 --partitions 3\n----\n\nRun 2 consumer instances (2 different shells/terminals) belonging to the same consumer group and subscribed to _user_ and _user_clicks_ topics; consumers uses\nlink:https://kafka.apache.org/37/javadoc/org/apache/kafka/clients/consumer/RoundRobinAssignor.html[_org.apache.kafka.clients.consumer.RoundRobinAssignor_] to distribute partition ownership.\n\n[source,bash]\n----\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.consumer.core.Runner\" -Dexec.args=\"users,users_clicks org.apache.kafka.clients.consumer.RoundRobinAssignor rr-group-app\"\n----\n\nSend messages to both topics using the same key (_Frank_):\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic users --property \"parse.key=true\" --property \"key.separator=:\"\n\u003e Frank:1\n\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic users_clicks --property \"parse.key=true\" --property \"key.separator=:\"\n\u003e Frank:1\n----\n\nVerify that messages are read by different consumer instances.\n\n==== Static membership\n\nThis example will show how to configure different consumer instances to use a unique group instance id and define a static membership for topic partitions.\n\nAfter shutting down and then restarting the consumer instance, this will consume from the same partitions avoiding re-balancing.\n\nCreate topic _topic1_ with 12 partitions:\n\n[source,bash]\n----\ndocker exec broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic topic1 --replication-factor 1 --partitions 12\n----\n\nRun 3 different consumer instances (from 3 different terminals) belonging to the same consumer group:\n\nmember1:\n[source,bash]\n----\ncd kafka-consumer\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.consumer.staticmembership.Runner\" -Dexec.args=\"consumer-member1.properties\"\n----\n\nmember2:\n[source,bash]\n----\ncd kafka-consumer\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.consumer.staticmembership.Runner\" -Dexec.args=\"consumer-member2.properties\"\n----\n\nmember3:\n[source,bash]\n----\ncd kafka-consumer\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.consumer.staticmembership.Runner\" -Dexec.args=\"consumer-member3.properties\"\n----\n\nRun a producer perf test to send messages to _topic1_:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-producer-perf-test.sh --topic topic1 --num-records 10000 --throughput -1 --record-size 2000 --producer-props bootstrap.servers=broker:9092\n----\n\nConsumers will start reading messages from partitions (e.g.):\n\n - member1 (1,2,3,4)\n - member2 (5,6,7,8)\n - member3 (9,10,11,12)\n\nTry to shut down consumer instances (CTRL+C) and then re-start them again; verify that re-balancing will not happen and consumers will always read from the same partitions.\n\n=== Read from the closest replica\n\nThis example shows how to use the feature (since Apache Kafka® 2.4+) for consumers to read messages from the closest replica, even if it is not a leader of the partition.\n\nStart a cluster with 3 brokers on 3 different racks, _dc1_, _dc2_ and _dc3_:\n\n[source,bash]\n----\nscripts/bootstrap-racks.sh\n----\n\nCreate topic _topic-regional_ and assign partition leaderships only on broker 1 and 3 (_dc1_ and _dc3_):\n\n[source,bash]\n----\ndocker exec broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic topic-regional --replication-factor 3 --partitions 3\n----\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server broker:9092 --reassignment-json-file /tmp/assignment.json --execute\n\ndocker exec -it broker /opt/kafka/bin/kafka-leader-election.sh --bootstrap-server broker:9092 --topic topic-regional --election-type PREFERRED --partition 0\n\ndocker exec -it broker /opt/kafka/bin/kafka-leader-election.sh --bootstrap-server broker:9092 --topic topic-regional --election-type PREFERRED --partition 1\n\ndocker exec -it broker /opt/kafka/bin/kafka-leader-election.sh --bootstrap-server broker:9092 --topic topic-regional --election-type PREFERRED --partition 2\n----\n\nVerify partitions with topic describe command:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --topic topic-regional --describe\n\nTopic: topic-regional\tTopicId: p-sy0qiQTtSTLTJSG7s7Ew\tPartitionCount: 3\tReplicationFactor: 3\tConfigs:\n\tTopic: topic-regional\tPartition: 0\tLeader: 1\tReplicas: 1,2,3\tIsr: 2,3,1\tOffline:\n\tTopic: topic-regional\tPartition: 1\tLeader: 3\tReplicas: 3,2,1\tIsr: 3,1,2\tOffline:\n\tTopic: topic-regional\tPartition: 2\tLeader: 1\tReplicas: 1,3,2\tIsr: 1,2,3\tOffline:\n----\n\nRun a consumer that will read messages from broker2 from rack _dc2_:\n\n[source,bash]\n----\ncd kafka-consumer\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.consumer.rack.Runner\"\n----\n\nProduce 50 messages:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-producer-perf-test.sh --topic topic-regional --num-records 50 --throughput 10 --record-size 1 --producer-props bootstrap.servers=broker:9092\n----\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-racks.sh\n----\n\n=== Consumers and retry topics\n\nFolder link:kafka-consumer-retry-topics/[kafka-consumer-retry-topics/]\n\nThis solution could be implemented on consumer side to handle errors in processing records without blocking the input topic.\n\n . Consumer processes records and commit the offset (_auto-commit_).\n . If a record can't be processed _(simple condition here to raise an error, is the existence of a specific message HEADER named ERROR)_, it is sent to a retry topic, if the number of retries is not yet exhausted.\n . When the number of retries is exhausted, record is sent to a DLQ topic.\n . Number of retries is set at Consumer instance level.\n\nCreate topics _retry-topic_ and _dlq-topic_:\n\n[source,bash]\n----\ndocker exec broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic retry-topic --replication-factor 1 --partitions 1\n\ndocker exec broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic dlq-topic --replication-factor 1 --partitions 1\n----\n\nRun consumer managing retry topics:\n\n[source,bash]\n----\ncd kafka-consumer-retry-topics\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.consumer.retry.ConsumerRetries\"\n----\n\nSend records:\n\n[source,bash]\n----\ndocker exec kcat bash -c \"echo 'alice,{\"col_foo\":1}'|kcat -b broker:9092 -t input-topic -P -K ,\"\n\ndocker exec kcat bash -c \"echo 'alice,{\"col_foo\":1}'|kcat -b broker:9092 -t input-topic -P -H ERROR=xxxxx -K ,\"\ndocker exec kcat bash -c \"echo 'alice,{\"col_foo\":1}'|kcat -b broker:9092 -t input-topic -P -H ERROR=xxxxx -K ,\"\ndocker exec kcat bash -c \"echo 'alice,{\"col_foo\":1}'|kcat -b broker:9092 -t input-topic -P -H ERROR=xxxxx -K ,\"\ndocker exec kcat bash -c \"echo 'alice,{\"col_foo\":1}'|kcat -b broker:9092 -t input-topic -P -H ERROR=xxxxx -K ,\"\n----\n\nVerify in consumer log if messages are sent to retry and dlq topics:\n\n[source,bash]\n----\nGroup id c9a19a62-0284-4251-be22-5d691243646a - Consumer id: consumer-c9a19a62-0284-4251-be22-5d691243646a-1-86fb972e-b5c8-4621-8464-9c1a747a920b - Topic: input-topic - Partition: 0 - Offset: 0 - Key: alice - Value: {col_foo:1}\nGroup id c9a19a62-0284-4251-be22-5d691243646a - Consumer id: consumer-c9a19a62-0284-4251-be22-5d691243646a-1-86fb972e-b5c8-4621-8464-9c1a747a920b - Topic: input-topic - Partition: 0 - Offset: 1 - Key: alice - Value: {col_foo:1}\nError message detected: number of retries 3 left for key alice\nsend to RETRY topic: retry-topic\nGroup id c9a19a62-0284-4251-be22-5d691243646a - Consumer id: consumer-c9a19a62-0284-4251-be22-5d691243646a-1-86fb972e-b5c8-4621-8464-9c1a747a920b - Topic: input-topic - Partition: 0 - Offset: 2 - Key: alice - Value: {col_foo:1}\nError message detected: number of retries 2 left for key alice\nsend to RETRY topic: retry-topic\nGroup id c9a19a62-0284-4251-be22-5d691243646a - Consumer id: consumer-c9a19a62-0284-4251-be22-5d691243646a-1-86fb972e-b5c8-4621-8464-9c1a747a920b - Topic: input-topic - Partition: 0 - Offset: 3 - Key: alice - Value: {col_foo:1}\nError message detected: number of retries 1 left for key alice\nsend to RETRY topic: retry-topic\nGroup id c9a19a62-0284-4251-be22-5d691243646a - Consumer id: consumer-c9a19a62-0284-4251-be22-5d691243646a-1-86fb972e-b5c8-4621-8464-9c1a747a920b - Topic: input-topic - Partition: 0 - Offset: 4 - Key: alice - Value: {col_foo:1}\nError message detected: number of retries 0 left for key alice\nnumber of retries exhausted, send to DLQ topic: dlq-topic\n----\n\n=== Interceptor\n\nFolder link:interceptors/[interceptors/]\n\nThis example shows how to create a custom consumer interceptor. Java class _CreditCardConsumerInterceptor_ will intercept records before deserialization and print headers.\n\nRun a consumer:\n\n[source,bash]\n----\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.interceptor.consumer.Runner\"\n----\n\nRun a producer:\n\n[source,bash]\n----\ncd interceptors\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.interceptor.producer.Runner\"\n----\n\nVerify output:\n\n[source,bash]\n----\nrecord headers:RecordHeaders(headers = [], isReadOnly = false)\nGroup id consumer-interceptor-g2 - Consumer id: consumer-consumer-interceptor-g2-1-0e20b2b6-3269-4bc5-bfdb-ca787cf68aa8 - Topic: test_custom_data - Partition: 0 - Offset: 0 - Key: null - Value: XXXXXX\nConsumer 23d06b51-5780-4efc-9c33-a93b3caa3b48 - partition 0 - lastOffset 1\n----\n\n=== Python Consumer\n\nFolder link:kafka-python-consumer/[kafka-python-consumer/]\n\nInstall confluent kafka python lib link:https://docs.confluent.io/kafka-clients/python/current/overview.html[_confluent-kafka_]:\n\n[source,bash]\n----\npip install confluent-kafka\n----\n\nCreate topic _kafka-topic_:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic kafka-topic --replication-factor 1 --partitions 1\n----\n\nRun producer:\n\n[source,bash]\n----\ncd kafka-python-producer\npython producer.py\n----\n\nRun consumer:\n\n[source,bash]\n----\ncd kafka-python-consumer\npython consumer.py\n----\n\n\u003c\u003c\u003c\n\n== Admin \u0026 Management\n\n=== Apache Kafka® CLI Tools\n\nApache Kafka® CLI are located in _$KAFKA_HOME/bin_ directory.\n\nDocker images provided are already shipped with CLI.\n\n. _kafka-acls_ - manage acls\n. _kafka-topics_ - create, delete, describe, or change a topic\n. _kafka-configs_ - create, delete, describe, or change cluster settings\n. _kafka-consumer-groups_ - manage consumer groups\n. _kafka-console-consumer_ - read data from topics and outputs it to standard output\n. _kafka-console-producer_ - produce data to topics\n. _kafka-consumer-perf-test_ - consume high volumes of data through your cluster\n. _kafka-producer-perf-test_ - produce high volumes of data through your cluster\n. _kafka-avro-console-producer_ - produce Avro data to topics with a schema _(only with confluent installation)_\n. _kafka-avro-console-consumer_ - read Avro data from topics with a schema and outputs it to standard output _(only with confluent installation)_\n\n=== Topics: segments and retention\n\nCreate a topic _cars_ with retention for old segments set to 5 minutes and size of segments set to 100 KB.\n\nBe aware that link:https://docs.confluent.io/platform/current/installation/configuration/broker-configs.html#log-retention-check-interval-ms[_log.retention.check.interval.ms_] is set by default to 5 minutes and this is the frequency in milliseconds that the log cleaner checks whether any log is eligible for deletion.\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic cars --replication-factor 1 --partitions 1 --config segment.bytes=100000 --config segment.ms=604800000 --config retention.ms=300000 --config retention.bytes=-1\n----\n\nLaunch a producer performance session:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-producer-perf-test.sh --topic cars --num-records 99999999999999 --throughput -1 --record-size 1 --producer-props bootstrap.servers=broker:9092\n----\n\nCheck the log dir for _cars_ topic and wait for deletion of old segments (5 minutes + log cleaner trigger delta)\n\n[source,bash]\n----\ndocker exec -it broker watch ls -ltr /tmp/kraft-combined-logs/cars-0/\n----\n\n=== Apache Kafka® Admin Client\n\nFolder link:admin-client/[admin-client]\n\nIt uses link:https://kafka.apache.org/37/javadoc/org/apache/kafka/clients/admin/AdminClient.html[_org.apache.kafka.clients.admin.AdminClient_] to execute Admin API.\n\nOperations currently added:\n\n - list of cluster nodes\n - list topics\n\n[source,bash]\n----\ncd admin-client\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.admin.AdminClientWrapper\" -Dexec.args=\"admin.properties\"\n----\n\n=== Compression\n\nFolder link:compression/[compression/]\n\nThis example will show that messages sent to the same topic with different _compression.type_.\nMessages with different compression can be read by the same consumer instance.\n\nCompressions supported on producer are:\n\n - _none_ (no compression)\n - _gzip_\n - _snappy_\n - _lz4_\n - _zstd_\n\nSend messages with different compression type and with batching disabled:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic topic1 --producer.config compression/client-none.properties --property \"parse.key=true\" --property \"key.separator=:\"\n0:none\n----\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic topic1 --producer.config compression/client-gzip.properties --property \"parse.key=true\" --property \"key.separator=:\"\n1:gzip\n----\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic topic1 --producer.config compression/client-snappy.properties --property \"parse.key=true\" --property \"key.separator=:\"\n2:snappy\n----\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic topic1 --producer.config compression/client-lz4.properties --property \"parse.key=true\" --property \"key.separator=:\"\n3:lz4\n----\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic topic1 --producer.config compression/client-zstd.properties --property \"parse.key=true\" --property \"key.separator=:\"\n4:zstd\n----\n\nRun a consumer on _topic1_ topic:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-consumer.sh --topic topic1 --bootstrap-server broker:9092 --from-beginning\nnone\ngzip\nsnappy\nlz4\nzstd\n----\n\n\u003c\u003c\u003c\n\n== Schema Registry\n\n=== Confluent Avro Generic Record\n\nIt uses link:https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/serializers/KafkaAvroSerializer.java[_io.confluent.kafka.serializers.KafkaAvroSerializer_] for value serializer, sending an Avro link:https://avro.apache.org/docs/1.11.1/api/java/org/apache/avro/generic/GenericRecord.html[_GenericRecord_].\n\nConfluent Schema Registry is needed to run the example.\n\nMore Info at: https://github.com/confluentinc/schema-registry\n\nAvro schema link:kafka-producer/src/main/resources/car.avsc[_car.avsc_]:\n\n[source,json]\n----\n{\n \"type\": \"record\",\n \"name\": \"Car\",\n \"namespace\": \"org.hifly.kafka.demo.producer.serializer.avro\",\n \"fields\": [\n  {\n   \"name\": \"model\",\n   \"type\": \"string\"\n  },\n  {\n   \"name\": \"brand\",\n   \"type\": \"string\"\n  }\n ]\n}\n----\n\nStart Confluent Schema Registry:\n\n[source,bash]\n----\nscripts/bootstrap-cflt-schema-registry.sh\n----\n\nConsume messages:\n\n[source,bash]\n----\ncd kafka-consumer\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.consumer.deserializer.avro.Runner\" -Dexec.args=\"CONFLUENT\"\n----\n\nProduce messages:\n\n[source,bash]\n----\ncd kafka-producer\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.producer.serializer.avro.Runner\" -Dexec.args=\"CONFLUENT\"\n----\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-cflt-schema-registry.sh\n----\n\n=== Apicurio Avro Generic Record\n\nIt uses link:https://github.com/Apicurio/apicurio-registry/blob/main/serdes/avro-serde/src/main/java/io/apicurio/registry/serde/avro/AvroKafkaSerializer.java[_io.apicurio.registry.utils.serde.AvroKafkaSerializer_] for value serializer, sending an Avro link:https://avro.apache.org/docs/1.11.1/api/java/org/apache/avro/generic/GenericRecord.html[_GenericRecord_].\n\nApicurio Schema Registry is needed to run the example.\n\nInfo at: https://github.com/Apicurio/apicurio-registry\n\nAvro schema link:kafka-producer/src/main/resources/car.avsc[_car.avsc_]:\n\n[source,json]\n----\n{\n \"type\": \"record\",\n \"name\": \"Car\",\n \"namespace\": \"org.hifly.kafka.demo.producer.serializer.avro\",\n \"fields\": [\n  {\n   \"name\": \"model\",\n   \"type\": \"string\"\n  },\n  {\n   \"name\": \"brand\",\n   \"type\": \"string\"\n  }\n ]\n}\n----\n\nStart Apicurio:\n\n[source,bash]\n----\nscripts/bootstrap-apicurio.sh\n----\n\nConsume messages:\n\n[source,bash]\n----\ncd kafka-consumer\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.consumer.deserializer.avro.Runner\" -Dexec.args=\"APICURIO\"\n----\n\nProduce messages:\n\n[source,bash]\n----\ncd kafka-producer\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.producer.serializer.avro.Runner\" -Dexec.args=\"APICURIO\"\n----\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-apicurio.sh\n----\n\n=== Hortonworks Avro Generic Record\n\nIt uses _com.hortonworks.registries.schemaregistry.serdes.avro.kafka.KafkaAvroSerializer_ for value serializer, sending an Avro link:https://avro.apache.org/docs/1.11.1/api/java/org/apache/avro/generic/GenericRecord.html[_GenericRecord_].\n\nHortonworks Schema Registry is needed to run the example. +\n\nInfo at: https://registry-project.readthedocs.io/en/latest/schema-registry.html#running-kafka-example\n\nAvro schema link:kafka-producer/src/main/resources/car.avsc[_car.avsc_]:\n\n[source,json]\n----\n{\n \"type\": \"record\",\n \"name\": \"Car\",\n \"namespace\": \"org.hifly.kafka.demo.producer.serializer.avro\",\n \"fields\": [\n  {\n   \"name\": \"model\",\n   \"type\": \"string\"\n  },\n  {\n   \"name\": \"brand\",\n   \"type\": \"string\"\n  }\n ]\n}\n----\n\nStart Hortonworks Schema Registry:\n\n[source,bash]\n----\nscripts/bootstrap-hortonworks-sr.sh\n----\n\n[source,bash]\n----\ncd kafka-producer\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.producer.serializer.avro.Runner\" -Dexec.args=\"HORTONWORKS\"\n----\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-hortonworks-sr.sh\n----\n\n=== Confluent Avro Specific Record\n\nFolder: link:confluent-avro-specific-record/[confluent-avro-specific-record]\n\nImplementation of a producer and a consumer using Avro link:https://avro.apache.org/docs/1.11.1/api/java/org/apache/avro/specific/SpecificRecord.html[_SpecificRecord_] for serializing and deserializing.\n\nConfluent Schema Registry is needed to run the example. +\n\n[source,bash]\n----\nscripts/bootstrap-cflt-schema-registry.sh\n----\n\nCreate _cars_ topic:\n\n[source,bash]\n----\ndocker exec -it broker kafka-topics --bootstrap-server broker:9092 --create --topic cars --replication-factor 1 --partitions 1\n----\n\nAvro schema link:confluent-avro-specific-record/src/main/resources/car_v1.avsc[_car_v1.avsc_]:\n\n[source,json]\n----\n{\"schema\": \"{\\\"type\\\": \\\"record\\\",\\\"name\\\": \\\"Car\\\",\\\"namespace\\\": \\\"org.hifly.kafka.demo.avro\\\",\\\"fields\\\": [{\\\"name\\\": \\\"model\\\",\\\"type\\\": \\\"string\\\"},{\\\"name\\\": \\\"brand\\\",\\\"type\\\": \\\"string\\\"}]}\"}\n----\n\nRegister first version of schema:\n\n[source,bash]\n----\ncurl -X POST -H \"Content-Type: application/vnd.schemaregistry.v1+json\" \\\n--data @confluent-avro-specific-record/src/main/resources/car_v1.avsc \\\nhttp://localhost:8081/subjects/cars-value/versions\n----\n\nRun the consumer:\n\n[source,bash]\n----\ncd confluent-avro-specific-record\nmvn clean compile package \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.avro.RunnerConsumer\"\n----\n\nRun the producer:\n\n[source,bash]\n----\ncd confluent-avro-specific-record\nmvn clean compile package \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.avro.RunnerProducer\"\n----\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-cflt-schema-registry.sh\n----\n\n=== Confluent Schema Registry: Schema Evolution\n\nFor documentation see the official docs at: https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html\n\n==== Backward\n\nChanges allowed:\n\n - Delete fields\n - Add optional fields\n\nConfluent Schema Registry is needed to run the example. +\n\n[source,bash]\n----\nscripts/bootstrap-cflt-schema-registry.sh\n----\n\nCreate _car_ topic:\n\n[source,bash]\n----\ndocker exec -it broker kafka-topics --bootstrap-server broker:9092 --create --topic cars --replication-factor 1 --partitions 1\n----\n\nAvro schema link:avro/car_v1.avsc[_car_v1.avsc_]:\n\n[source,json]\n----\n{\"schema\": \"{ \\\"type\\\": \\\"record\\\", \\\"name\\\": \\\"Car\\\", \\\"namespace\\\": \\\"org.hifly.kafka.demo.producer.serializer.avro\\\",\\\"fields\\\": [   {\\\"name\\\": \\\"model\\\",\\\"type\\\": \\\"string\\\"},{\\\"name\\\": \\\"brand\\\",\\\"type\\\": \\\"string\\\"}] }\" }\n----\n\nRegister a first version of schema:\n\n[source,bash]\n----\ncurl -X POST -H \"Content-Type: application/vnd.schemaregistry.v1+json\" \\\n--data @avro/car_v1.avsc \\\nhttp://localhost:8081/subjects/cars-value/versions\n----\n\nSet compatibility on _BACKWARD_:\n\n[source,bash]\n----\ncurl -X PUT -H \"Content-Type: application/vnd.schemaregistry.v1+json\" \\\n--data '{\"compatibility\": \"BACKWARD\"}' \\\nhttp://localhost:8081/config/cars-value\n----\n\nVerify compatibility for _cars-value_ subject:\n\n[source,bash]\n----\ncurl -X GET http://localhost:8081/config/cars-value\n----\n\nRun the producer:\n\n[source,bash]\n----\ncd confluent-avro-specific-record\nmvn clean compile package \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.avro.RunnerProducer\"\n----\n\nRun the consumer (don't stop it):\n\n[source,bash]\n----\ncd confluent-avro-specific-record\nmvn clean compile package \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.avro.RunnerConsumer\"\n----\n\nView the latest schema for _cars-value_ subject:\n\n[source,bash]\n----\ncurl -X GET http://localhost:8081/subjects/cars-value/versions/latest | jq .\n----\n\nAvro schema link:avro/car_v2.avsc[_car_v2.avsc_]:\n\n[source,json]\n----\n{\"schema\": \"{ \\\"type\\\": \\\"record\\\", \\\"name\\\": \\\"Car\\\", \\\"namespace\\\": \\\"org.hifly.kafka.demo.producer.serializer.avro\\\",\\\"fields\\\": [   {\\\"name\\\": \\\"engine\\\",\\\"type\\\": \\\"string\\\", \\\"default\\\":\\\"diesel\\\"}, {\\\"name\\\": \\\"model\\\",\\\"type\\\": \\\"string\\\"},{\\\"name\\\": \\\"brand\\\",\\\"type\\\": \\\"string\\\"}] }\" }\n----\n\nRegister a new version of schema, with the addition of a field with default value:\n\n[source,bash]\n----\ncurl -X POST -H \"Content-Type: application/vnd.schemaregistry.v1+json\" \\\n--data @avro/car_v2.avsc \\\nhttp://localhost:8081/subjects/cars-value/versions\n----\n\nProduce data with using the new schema:\n\n[source,bash]\n----\nsh produce-avro-records.sh\n----\n\nVerify that consumer will not break and continue to process messages.\n\nAvro schema link:avro/car_v3.avsc[_car_v3.avsc_]:\n\n[source,json]\n----\n{\"schema\": \"{ \\\"type\\\": \\\"record\\\", \\\"name\\\": \\\"Car\\\", \\\"namespace\\\": \\\"org.hifly.kafka.demo.producer.serializer.avro\\\",\\\"fields\\\": [   {\\\"name\\\": \\\"engine\\\",\\\"type\\\": \\\"string\\\"}, {\\\"name\\\": \\\"model\\\",\\\"type\\\": \\\"string\\\"},{\\\"name\\\": \\\"brand\\\",\\\"type\\\": \\\"string\\\"}] }\" }\n----\n\nRegister a new version of schema, with the addition of a field with a required value:\n\n[source,bash]\n----\ncurl -X POST -H \"Content-Type: application/vnd.schemaregistry.v1+json\" \\\n--data @avro/car_v3.avsc \\\nhttp://localhost:8081/subjects/cars-value/versions\n----\n\nyou will get an error:\n\n[source,bash]\n----\n{\"error_code\":42201,\"message\":\"Invalid schema\n----\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-cflt-schema-registry.sh\n----\n\n=== Confluent Schema Registry: Multiple Event Types in same Topic\n\nFolder: link:confluent-avro-multi-event/[confluent-avro-multi-event]\n\nThis example shows how to use Avro unions with schema references.\n\nIn this example a topic named _car-telemetry_ will be configured with a schema _car-telemetry.avsc_ and will store different Avro messages:\n\n - car-info messages from schema _car-info.avsc_\n - car-telemetry messages from schema _car-telemetry-data.avsc_\n\nlink:confluent-avro-multi-event/src/main/resources/car-telemetry-data.avsc[_car-telemetry.avsc_]:\n\n[source,json]\n----\n[\n  \"org.hifly.kafka.demo.avro.references.CarInfo\",\n  \"org.hifly.kafka.demo.avro.references.CarTelemetryData\"\n]\n----\n\nlink:confluent-avro-multi-event/src/main/resources/car-telemetry-data.avsc[_car-telemetry-data.avsc_]:\n\n[source,json]\n----\n{\n  \"type\": \"record\",\n  \"name\": \"CarTelemetryData\",\n  \"namespace\": \"org.hifly.kafka.demo.avro.references\",\n  \"fields\": [\n    {\n      \"name\": \"speed\",\n      \"type\": \"double\"\n    },\n    {\n      \"name\": \"latitude\",\n      \"type\": \"string\"\n    },\n    {\n      \"name\": \"longitude\",\n      \"type\": \"string\"\n    }\n  ]\n}\n----\n\nlink:confluent-avro-multi-event/src/main/resources/car-info.avsc[_car-info.avsc_]:\n\n[source,json]\n----\n{\n  \"type\": \"record\",\n  \"name\": \"CarInfo\",\n  \"namespace\": \"org.hifly.kafka.demo.avro.references\",\n  \"fields\": [\n    {\n      \"name\": \"model\",\n      \"type\": \"string\"\n    },\n    {\n      \"name\": \"brand\",\n      \"type\": \"string\"\n    }\n  ]\n}\n----\n\nConfluent Schema Registry is needed to run the example. +\n\n[source,bash]\n----\nscripts/bootstrap-cflt-schema-registry.sh\n----\n\nRegister the subjects using link:https://docs.confluent.io/platform/current/schema-registry/develop/maven-plugin.html[Confluent Schema Registry maven plugin]:\n\n[source,bash]\n----\ncd confluent-avro-multi-event\nmvn schema-registry:register\n\n[INFO] --- kafka-schema-registry-maven-plugin:7.4.0:register (default-cli) @ confluent-avro-references ---\n[INFO] Registered subject(car-info) with id 1 version 1\n[INFO] Registered subject(car-telemetry-data) with id 2 version 1\n[INFO] Registered subject(car-telemetry-value) with id 3 version 1\n----\n\nVerify the subjects:\n\n[source,bash]\n----\ncurl -X GET http://localhost:8081/subjects\n\n[\"car-info\",\"car-telemetry-data\",\"car-telemetry-value\"]\n----\n\nVerify the resulting schema for _car-telemetry-value_ subject:\n\n[source,bash]\n----\ncurl -X GET http://localhost:8081/subjects/car-telemetry-value/versions/1\n\n{\"subject\":\"car-telemetry-value\",\"version\":1,\"id\":3,\"references\":[{\"name\":\"io.confluent.examples.avro.references.CarInfo\",\"subject\":\"car-info\",\"version\":1},{\"name\":\"io.confluent.examples.avro.references.CarTelemetryData\",\"subject\":\"car-telemetry-data\",\"version\":1}],\"schema\":\"[\\\"org.hifly.kafka.demo.avro.references.CarInfo\\\",\\\"org.hifly.kafka.demo.avro.references.CarTelemetryData\\\"]\"}\n----\n\nGenerate Java Pojo from avro schemas:\n\n[source,bash]\n----\ncd confluent-avro-multi-event\nmvn clean package\n----\n\nRun a Consumer:\n\n[source,bash]\n----\ncd confluent-avro-multi-event\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.avro.references.RunnerConsumer\"\n----\n\nOn a different shell, run a Producer:\n\n[source,bash]\n----\ncd confluent-avro-multi-event\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.avro.references.RunnerProducer\"\n----\n\nVerify records on Consumer:\n\n[source,bash]\n----\nCar Info event {\"model\": \"Ferrari\", \"brand\": \"F40\"} - offset-\u003e 4\nCar Telemetry event {\"speed\": 156.8, \"latitude\": \"42.8\", \"longitude\": \"22.6\"} - offset-\u003e 5\n----\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-cflt-schema-registry.sh\n----\n\n=== Confluent Schema Registry: Nested objects\n\nFolder: link:confluent-avro-hierarchy-event/[confluent-avro-hierarchy-event]\n\nThis example shows how to use Avro nested objects.\n\nIn this example a topic named _car-telemetry_ will be configured with a schema _car-telemetry-data.avsc_ with a nested schema reference from _car.avsc_\n\nlink:confluent-avro-hierarchy-event/src/main/resources/car-telemetry-data.avsc[_car-telemetry-data.avsc_]:\n\n[source,json]\n----\n{\n  \"type\": \"record\",\n  \"name\": \"CarTelemetryData\",\n  \"namespace\": \"org.hifly.kafka.demo.avro.references\",\n  \"fields\": [\n    {\n      \"name\": \"speed\",\n      \"type\": \"double\"\n    },\n    {\n      \"name\": \"latitude\",\n      \"type\": \"string\"\n    },\n    {\n      \"name\": \"longitude\",\n      \"type\": \"string\"\n    },\n    {\n      \"name\": \"info\",\n      \"type\": \"org.hifly.kafka.demo.avro.references.CarInfo\"\n    }\n\n  ]\n}\n----\n\nlink:confluent-avro-hierarchy-event/src/main/resources/car-info.avsc[_car-info.avsc_]:\n\n[source,json]\n----\n{\n  \"type\": \"record\",\n  \"name\": \"CarInfo\",\n  \"namespace\": \"org.hifly.kafka.demo.avro.references\",\n  \"fields\": [\n    {\n      \"name\": \"model\",\n      \"type\": \"string\"\n    },\n    {\n      \"name\": \"brand\",\n      \"type\": \"string\"\n    }\n  ]\n}\n----\n\nConfluent Schema Registry is needed to run the example. +\n\n[source,bash]\n----\nscripts/bootstrap-cflt-schema-registry.sh\n----\n\nRegister the subjects using link:https://docs.confluent.io/platform/current/schema-registry/develop/maven-plugin.html[Confluent Schema Registry maven plugin]:\n\n[source,bash]\n----\ncd confluent-avro-hierarchy-event\nmvn schema-registry:register\n\n[INFO] --- kafka-schema-registry-maven-plugin:7.4.0:register (default-cli) @ confluent-avro-hierarchy-event ---\n[INFO] Registered subject(car-info) with id 4 version 2\n[INFO] Registered subject(car-telemetry-value) with id 5 version 3\n\n----\n\nGenerate Java Pojo from avro schemas:\n\n[source,bash]\n----\ncd confluent-avro-hierarchy-event\nmvn clean package\n----\n\nRun a Consumer:\n\n[source,bash]\n----\ncd confluent-avro-hierarchy-event\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.avro.references.app.RunnerConsumer\"\n----\n\nOn a different shell, run a Producer:\n\n[source,bash]\n----\ncd confluent-avro-hierarchy-event\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.avro.references.app.RunnerProducer\"\n----\n\nVerify records on Consumer:\n\n[source,bash]\n----\nRecord:{\"speed\": 156.8, \"latitude\": \"42.8\", \"longitude\": \"22.6\", \"info\": {\"model\": \"Ferrari\", \"brand\": \"F40\"}}\n----\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-cflt-schema-registry.sh\n----\n\n\u003c\u003c\u003c\n\n== Apache Kafka® Connect\n\n=== Unix commands Source Connector\n\nFolder: link:kafka-unixcommand-connector/[kafka-unixcommand-connector]\n\nImplementation of a sample Kafka Connect Source Connector; it executes _unix commands_ (e.g. _fortune_, _ls -ltr, netstat_) and sends its output to a topic.\n\nIMPORTANT: unix commands are executed on connect worker node.\n\nThis connector relies on Confluent Schema Registry to convert messages using an Avro converter:\nlink:https://github.com/confluentinc/schema-registry/blob/master/avro-converter/src/main/java/io/confluent/connect/avro/AvroConverter.java[_io.confluent.connect.avro.AvroConverter_].\n\nConnector link:kafka-unixcommand-connector/config/source.quickstart.json[source.quickstart.json]:\n\n[source,json]\n----\n{\n    \"name\" : \"unixcommandsource\",\n    \"config\": {\n        \"connector.class\" : \"org.hifly.kafka.demo.connector.UnixCommandSourceConnector\",\n        \"command\" : \"fortune\",\n        \"topic\": \"unixcommands\",\n        \"poll.ms\" : 5000,\n        \"tasks.max\": 1\n    }\n}\n----\n\nParameters for source connector:\n\n- _command_ – unix command to execute (e.g. ls -ltr, fortune)\n- _topic_ – output topic\n- _poll.ms_ – poll interval in milliseconds between every execution\n\nCreate the connector package:\n\n[source,bash]\n----\ncd kafka-unixcommand-connector\nmvn clean package\n----\n\nRun the Docker container:\n\n[source,bash]\n----\nscripts/bootstrap-unixcommand-connector.sh\n----\n\nThis will create an image based on link:https://hub.docker.com/r/confluentinc/cp-kafka-connect-base/tags[_confluentinc/cp-kafka-connect-base_] using a custom link:kafka-unixcommand-connector/Dockerfile[_Dockerfile_].\n\nIt will use the confluent-hub utility link:https://docs.confluent.io/kafka-connectors/confluent-hub/client.html[_confluent-hub install_] to install the plugin in connect.\n\n\nDeploy the connector:\n\n[source,bash]\n----\ncurl -X POST -H Accept:application/json -H Content-Type:application/json http://localhost:8083/connectors/ -d @kafka-unixcommand-connector/config/source.quickstart.json\n----\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-unixcommand-connector.sh\n----\n\n=== Custom SMT: composite key from json records.\n\nFolder: link:kafka-smt-custom[kafka-smt-custom]\n\nImplementation of a custom link:https://docs.confluent.io/platform/current/connect/transforms/overview.html[Single Message Transformation (SMT)];\nit creates a key from a list of json fields from message record value. Fields are configurable using SMT property _fields_.\n\nExample:\n\nOriginal record:\n\n[source,bash]\n----\nkey: null\nvalue: {\"FIELD1\": \"01\",\"FIELD2\": \"20400\",\"FIELD3\": \"001\",\"FIELD4\": \"0006084655017\",\"FIELD5\": \"20221117\",\"FIELD6\": 9000018}\n----\n\nResult after SMT:\n\n[source,bash]\n----\n\"transforms.createKey.fields\": \"FIELD1,FIELD2,FIELD3\"\n\nkey: 0120400001\nvalue: {\"FIELD1\": \"01\",\"FIELD2\": \"20400\",\"FIELD3\": \"001\",\"FIELD4\": \"0006084655017\",\"FIELD5\": \"20221117\",\"FIELD6\": 9000018}\n----\n\nThe example applies the SMT to a link:https://www.mongodb.com/docs/kafka-connector/current/sink-connector/[MongoDB sink connector].\n\nRun the example:\n\n[source,bash]\n----\nscripts/bootstrap-smt-connector.sh\n----\n\nA MongoDB sink connector will be created with this link:kafka-smt-custom/config/connector_mongo.json[config]:\n\n[source,bash]\n----\n{\n  \"name\": \"mongo-sink\",\n  \"config\": {\n    \"connector.class\": \"com.mongodb.kafka.connect.MongoSinkConnector\",\n    \"topics\": \"test\",\n    \"connection.uri\": \"mongodb://admin:password@mongo:27017\",\n    \"key.converter\": \"org.apache.kafka.connect.storage.StringConverter\",\n    \"value.converter\": \"org.apache.kafka.connect.storage.StringConverter\",\n    \"key.converter.schemas.enable\": false,\n    \"value.converter.schemas.enable\": false,\n    \"database\": \"Tutorial2\",\n    \"collection\": \"pets\",\n    \"transforms\": \"createKey\",\n    \"transforms.createKey.type\": \"org.hifly.kafka.smt.KeyFromFields\",\n    \"transforms.createKey.fields\": \"FIELD1,FIELD2,FIELD3\"\n  }\n}\n----\n\nOriginal json messages will be sent to _test_ topic.\n\nSink connector will apply the SMT and store the records in MongoDB _pets_ collection from _Tutorial2_ database, using a key generated by the SMT.\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-smt-connector.sh\n----\n\n=== SMT: log records with AOP\n\nFolder: link:kafka-smt-aspectj/[kafka-smt-aspectj]\n\nUsage of a predefined SMT to a link:https://www.mongodb.com/docs/kafka-connector/current/sink-connector/[MongoDB sink connector].\n\n_apply_ method for SMT classes in package _org.apache.kafka.connect.transforms_ is intercepted by a Java AOP Aspect\nimplemented using link:https://www.eclipse.org/aspectj/[AspectJ] framework.\n\nThe _@Aspect_, implemented in class link:kafka-smt-aspectj/src/main/java/org/hifly/kafka/smt/aspectj/SMTAspect.java[_org.hifly.kafka.smt.aspectj.SMTAspect_], logs the input arg (_SinkRecord_ object) to the standard output.\n\n\n[source,bash]\n----\n @Pointcut(\"execution(* org.apache.kafka.connect.transforms.*.apply(..)) \u0026\u0026 !execution(* org.apache.kafka.connect.runtime.PredicatedTransformation.apply(..))\")\n    public void standardMethod() {}\n\n    @Before(\"standardMethod()\")\n    public void log(JoinPoint jp) throws Throwable {\n\n        Object[] array = jp.getArgs();\n        if(array != null) {\n            for(Object tmp: array)\n                LOGGER.info(tmp.toString());\n        }\n    }\n----\n\nConnect log will show sink records entries:\n\n[source,bash]\n----\nSinkRecord{kafkaOffset=0, timestampType=CreateTime} ConnectRecord{topic='test', kafkaPartition=2, key=null, keySchema=Schema{STRING}, value={\"FIELD1\": \"01\",\"FIELD2\": \"20400\",\"FIELD3\": \"001\",\"FIELD4\": \"0006084655017\",\"FIELD5\": \"20221117\",\"FIELD6\": 9000018}, valueSchema=Schema{STRING}, timestamp=1683701851358, headers=ConnectHeaders(headers=)}\n----\n\nRun the example:\n\n[source,bash]\n----\nscripts/bootstrap-smt-aspectj.sh\n----\n\nConnect will start with aspectjweaver java agent:\n\n[source,bash]\n----\n-Dorg.aspectj.weaver.showWeaveInfo=true -Daj.weaving.verbose=true -javaagent:/usr/share/java/aspectjweaver-1.9.19.jar\n----\n\nAspects are deployed as standard jars and copied to Kafka Connect classpath _/etc/kafka-connect/jars/kafka-smt-aspectj-1.2.1.jar_\n\nA MongoDB sink connector will be created with this link:kafka-smt-aspectj/config/connector_mongo.json[config]:\n\n[source,bash]\n----\n{\n  \"name\": \"mongo-sink\",\n  \"config\": {\n    \"connector.class\": \"com.mongodb.kafka.connect.MongoSinkConnector\",\n    \"topics\": \"test\",\n    \"connection.uri\": \"mongodb://admin:password@mongo:27017\",\n    \"key.converter\": \"org.apache.kafka.connect.storage.StringConverter\",\n    \"value.converter\": \"org.apache.kafka.connect.storage.StringConverter\",\n    \"key.converter.schemas.enable\": false,\n    \"value.converter.schemas.enable\": false,\n    \"database\": \"Tutorial2\",\n    \"collection\": \"pets\",\n    \"transforms\": \"Filter\",\n    \"transforms.Filter.type\": \"org.apache.kafka.connect.transforms.Filter\",\n    \"transforms.Filter.predicate\": \"IsFoo\",\n    \"predicates\": \"IsFoo\",\n    \"predicates.IsFoo.type\": \"org.apache.kafka.connect.transforms.predicates.TopicNameMatches\",\n    \"predicates.IsFoo.pattern\": \"test\"\n\n  }\n}\n----\n\nOriginal json messages will be sent to _test_ topic.\n\nSink connector will apply the SMT and store the records in MongoDB _pets_ collection from _Tutorial2_ database.\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-smt-aspectj.sh\n----\n\n=== Postgres to Mongo\n\nIn this example a JDBC source connector will copy rows from a Postgres table to a MongoDB collection. Rows containing a JSON CLOB not properly parsable will be sent to DLQ topic.\n\nFolder: link:postgres-to-mongo[postgres-to-mongo]\n\nlink:https://www.mongodb.com/docs/kafka-connector/current/sink-connector/[MongoDB sink connector] example configured to send bad messages to a DLQ topic named _dlq-mongo-accounts_.\n\nMongoDB Sink Connector has been configured to use a link:https://www.mongodb.com/docs/kafka-connector/current/sink-connector/configuration-properties/id-strategy/[id strategy] to determine the __id_ value for each document.\n\nMongoDB Sink Connector has been configured to use a link:https://www.mongodb.com/docs/kafka-connector/current/sink-connector/configuration-properties/write-strategies/#delete-write-model[delete strategy] when it receives a tombstone event.\n\nRun the example:\n\n[source,bash]\n----\nscripts/bootstrap-postgres-to-mongo.sh\n----\n\nA JDBC source connector will be created with this link:postgres-to-mongo/config/jdbc_psql_source.json[config]:\n\n[source,bash]\n----\n{\n  \"name\": \"jdbc-source-connector\",\n  \"config\": {\n    \"connector.class\": \"io.confluent.connect.jdbc.JdbcSourceConnector\",\n    \"tasks.max\": \"1\",\n    \"connection.url\": \"jdbc:postgresql://postgres:5432/postgres\",\n    \"connection.user\": \"postgres\",\n    \"connection.password\": \"postgres\",\n    \"table.whitelist\": \"accounts\",\n    \"mode\": \"incrementing\",\n    \"incrementing.column.name\": \"seq_id\",\n    \"topic.prefix\": \"jdbc_\",\n    \"poll.interval.ms\": \"5000\",\n    \"numeric.mapping\": \"best_fit\",\n    \"value.converter\": \"io.confluent.connect.avro.AvroConverter\",\n    \"key.converter\": \"org.apache.kafka.connect.json.JsonConverter\",\n    \"key.converter.schemas.enable\": \"false\",\n    \"value.converter.schema.registry.url\": \"http://schema-registry:8081\",\n    \"transforms\": \"createKey,nestKey\",\n    \"transforms.createKey.type\": \"org.apache.kafka.connect.transforms.ValueToKey\",\n    \"transforms.createKey.fields\": \"id\",\n    \"transforms.nestKey.type\": \"org.apache.kafka.connect.transforms.ReplaceField$Key\",\n    \"transforms.nestKey.renames\": \"id:originalId\"\n  }\n}\n----\n\nA MongoDB sink connector will be created with this link:postgres-to-mongo/config/connector_mongo_sink.json[config]:\n\n[source,bash]\n----\n{\n  \"name\": \"mongo-sink-dlq\",\n  \"config\": {\n    \"connector.class\": \"com.mongodb.kafka.connect.MongoSinkConnector\",\n    \"errors.tolerance\": \"all\",\n    \"topics\": \"jdbc_accounts\",\n    \"errors.deadletterqueue.topic.name\": \"dlq-mongo-accounts\",\n    \"errors.deadletterqueue.topic.replication.factor\": \"1\",\n    \"errors.deadletterqueue.context.headers.enable\": \"true\",\n    \"connection.uri\": \"mongodb://admin:password@mongo:27017\",\n    \"database\": \"Employee\",\n    \"collection\": \"account\",\n    \"mongo.errors.log.enable\":\"true\",\n    \"delete.on.null.values\": \"true\",\n    \"document.id.strategy.overwrite.existing\": \"true\",\n    \"document.id.strategy\": \"com.mongodb.kafka.connect.sink.processor.id.strategy.FullKeyStrategy\",\n    \"delete.writemodel.strategy\": \"com.mongodb.kafka.connect.sink.writemodel.strategy.DeleteOneDefaultStrategy\",\n    \"publish.full.document.only\": \"true\",\n    \"value.converter\": \"io.confluent.connect.avro.AvroConverter\",\n    \"key.converter\": \"org.apache.kafka.connect.json.JsonConverter\",\n    \"key.converter.schemas.enable\": \"false\",\n    \"value.converter.schema.registry.url\": \"http://schema-registry:8081\"\n  }\n}\n----\n\nValidate results, query documents in Mongo collection _account_ from database _Employee_:\n\n[source,bash]\n----\ndocker exec -it mongo mongosh \"mongodb://admin:password@localhost:27017\" --eval 'db.getSiblingDB(\"Employee\").account.find()'\n----\n\n[source,bash]\n----\n[\n  { _id: { originalId: '1' }, id: '1', ssn: 'AAAA' },\n  { _id: { originalId: '2' }, id: '2', ssn: 'BBBB' },\n  { _id: { originalId: '3' }, id: '3', ssn: 'CCCC' },\n  { _id: { originalId: '4' }, id: '4', ssn: 'DDDD' },\n  { _id: { originalId: '5' }, id: '5', ssn: 'EEEE' }\n]\n----\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-postgres-to-mongo.sh\n----\n\n=== HTTP Sink Connector example\n\nFolder: link:kafka-connect-sink-http/[kafka-connect-sink-http]\n\nExample of usage of link:https://docs.confluent.io/kafka-connectors/http/current/overview.html[HTTP Sink Connector].\n\nRun the example:\n\n[source,bash]\n----\nscripts/bootstrap-connect-sink-http.sh\n----\n\nA web application, exposing REST APIs, listening on port _8010_ will start up.\n\nA HTTP sink connector will be created with this link:kafka-connect-sink-http/config/http_sink.json[config]:\n\n[source,bash]\n----\n{\n  \"name\": \"SimpleHttpSink\",\n  \"config\":\n  {\n    \"topics\": \"topicA\",\n    \"tasks.max\": \"2\",\n    \"connector.class\": \"io.confluent.connect.http.HttpSinkConnector\",\n    \"http.api.url\": \"http://host.docker.internal:8010/api/message\",\n    \"value.converter\": \"org.apache.kafka.connect.storage.StringConverter\",\n    \"confluent.topic.bootstrap.servers\": \"broker:9092\",\n    \"confluent.topic.replication.factor\": \"1\",\n    \"reporter.bootstrap.servers\": \"broker:9092\",\n    \"reporter.result.topic.name\": \"success-responses\",\n    \"reporter.result.topic.replication.factor\": \"1\",\n    \"reporter.error.topic.name\": \"error-responses\",\n    \"reporter.error.topic.replication.factor\": \"1\",\n    \"consumer.override.max.poll.interval.ms\": \"5000\"\n  }\n}\n----\n\nSend json messages to _topicA_ topic:\n\n[source,bash]\n----\ndocker exec -it broker kafka-console-producer --broker-list broker:9092 --topic topicA --property \"parse.key=true\" --property \"key.separator=:\"\n\u003e 1:{\"FIELD1\": \"01\",\"FIELD2\": \"20400\",\"FIELD3\": \"001\",\"FIELD4\": \"0006084655017\",\"FIELD5\": \"20221117\",\"FIELD6\": 9000018}\n----\n\nSink connector will execute an HTTP POST Request to the endpoint _http://localhost:8010/api/message_\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-connect-sink-http.sh\n----\n\n=== S3 Sink Connector example\n\nFolder: link:kafka-connect-sink-s3/[kafka-connect-sink-s3]\n\nExample of usage of link:https://docs.confluent.io/kafka-connectors/s3-sink/current/overview.html[S3 Sink Connector].\n\nRun the example:\n\n[source,bash]\n----\nscripts/bootstrap-connect-sink-s3.sh\n----\n\nlink:https://min.io/[MinIO] will start listening on port 9000 (admin/minioadmin)\n\nA S3 sink connector will be created with this link:kafka-connect-sink-s3/config/s3_sink.json[config]:\n\n[source,bash]\n----\n{\n  \"name\": \"sink-s3\",\n  \"config\":\n  {\n    \"topics\": \"gaming-player-activity\",\n    \"tasks.max\": \"1\",\n    \"connector.class\": \"io.confluent.connect.s3.S3SinkConnector\",\n    \"store.url\": \"http://minio:9000\",\n    \"s3.region\": \"us-west-2\",\n    \"s3.bucket.name\": \"gaming-player-activity-bucket\",\n    \"s3.part.size\": \"5242880\",\n    \"flush.size\": \"100\",\n    \"storage.class\": \"io.confluent.connect.s3.storage.S3Storage\",\n    \"format.class\": \"io.confluent.connect.s3.format.avro.AvroFormat\",\n    \"schema.generator.class\": \"io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator\",\n    \"partitioner.class\": \"io.confluent.connect.storage.partitioner.DefaultPartitioner\",\n    \"schema.compatibility\": \"NONE\"\n  }\n}\n----\n\nSink connector will read messages from topic _gaming-player-activity_ and store them in a S3 bucket _gaming-player-activity-bucket_ using _io.confluent.connect.s3.format.avro.AvroFormat_ as format class.\n\nSink connector will generate a new object storage entry every 100 messages (_flush_size_).\n\nTo generate random records for topic _gaming-player-activity_ we will use link:https://github.com/ugol/jr[jr] tool.\n\nSend 1000 messages to _gaming-player-activity_ topic using jr:\n\n[source,bash]\n----\ndocker exec -it -w /home/jr/.jr jr jr template run gaming_player_activity -n 1000 -o kafka -t gaming-player-activity -s --serializer avro-generic\n----\n\nVerify that 10 entries are stored in MinIO into _gaming-player-activity-bucket_ bucket, connecting to MiniIO web console, http://localhost:9000 (admin/minioadmin):\n\nimage::images/minio.png[gaming-player-activity-bucket]\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-connect-sink-s3.sh\n----\n\n==== Parquet format\n\nSame example but Sink connector will read Avro messages from topic _gaming-player-activity_ and store them in a S3 bucket _gaming-player-activity-bucket_ using _io.confluent.connect.s3.format.parquet.ParquetFormat_ as format class.\n\nThe format of data stored in MinIO will be Parquet.\n\nRun the example:\n\n[source,bash]\n----\nscripts/bootstrap-connect-sink-s3-parquet.sh\n----\n\nA S3 sink connector will be created with this link:kafka-connect-sink-s3/config/s3_parquet_sink.json[config]:\n\n[source,bash]\n----\n{\n  \"name\": \"sink-parquet-s3\",\n  \"config\":\n  {\n    \"topics\": \"gaming-player-activity\",\n    \"tasks.max\": \"1\",\n    \"connector.class\": \"io.confluent.connect.s3.S3SinkConnector\",\n    \"store.url\": \"http://minio:9000\",\n    \"s3.region\": \"us-west-2\",\n    \"s3.bucket.name\": \"gaming-player-activity-bucket\",\n    \"s3.part.size\": \"5242880\",\n    \"flush.size\": \"100\",\n    \"storage.class\": \"io.confluent.connect.s3.storage.S3Storage\",\n    \"partitioner.class\": \"io.confluent.connect.storage.partitioner.DefaultPartitioner\",\n    \"format.class\": \"io.confluent.connect.s3.format.parquet.ParquetFormat\",\n    \"parquet.codec\": \"snappy\",\n    \"schema.registry.url\": \"http://schema-registry:8081\",\n    \"value.converter\": \"io.confluent.connect.avro.AvroConverter\",\n    \"key.converter\": \"org.apache.kafka.connect.storage.StringConverter\",\n    \"value.converter.schema.registry.url\": \"http://schema-registry:8081\"\n  }\n}\n----\n\nSend 1000 messages to _gaming-player-activity_ topic using jr:\n\n[source,bash]\n----\ndocker exec -it -w /home/jr/.jr jr jr template run gaming_player_activity -n 1000 -o kafka -t gaming-player-activity -s --serializer avro-generic\n----\n\nVerify that 10 entries are stored in MinIO into _gaming-player-activity-bucket_ bucket, connecting to MiniIO web console, http://localhost:9000 (admin/minioadmin):\n\nimage::images/minio2.png[gaming-player-activity-bucket]\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-connect-sink-s3.sh\n----\n\n=== SAP HANA Source Connector example\n\nFolder: link:kafka-connect-source-sap-hana/[kafka-connect-source-sap-hana]\n\nExample of usage of link:https://github.com/SAP/kafka-connect-sap[SAP HANA Source Connector].\n\nRun the example:\n\n[source,bash]\n----\nscripts/bootstrap-connect-source-sap-hana.sh\n----\n\nInsert rows in _LOCALDEV.TEST_ table:\n\n[source,bash]\n----\ndocker exec -i hana /usr/sap/HXE/HDB90/exe/hdbsql -i 90 -d HXE -u LOCALDEV -p Localdev1  \u003e /tmp/result.log  2\u003e\u00261 \u003c\u003c-EOF\nINSERT INTO TEST (111, 'foo', 100,50);\nINSERT INTO TEST (222, 'bar', 100,50);\nEOF\n----\n\nA SAP HANA source connector will be created with this link:kafka-connect-source-sap-hana/config/sap_hana_source.json[config]:\n\n[source,bash]\n----\n{\n  \"name\": \"sap-hana-source\",\n  \"config\":\n  {\n    \"topics\": \"testtopic\",\n    \"tasks.max\": \"1\",\n    \"connector.class\": \"com.sap.kafka.connect.source.hana.HANASourceConnector\",\n    \"connection.url\": \"jdbc:sap://sap:39041/?databaseName=HXE\u0026reconnect=true\u0026statementCacheSize=512\",\n    \"connection.user\": \"LOCALDEV\",\n    \"connection.password\" : \"Localdev1\",\n    \"value.converter.schema.registry.url\": \"http://schema-registry:8081\",\n    \"auto.create\": \"true\",\n    \"testtopic.table.name\": \"\\\"LOCALDEV\\\".\\\"TEST\\\"\",\n    \"key.converter\": \"io.confluent.connect.avro.AvroConverter\",\n    \"key.converter.schema.registry.url\": \"http://schema-registry:8081\",\n    \"value.converter\": \"io.confluent.connect.avro.AvroConverter\",\n    \"value.converter.schema.registry.url\": \"http://schema-registry:8081\"\n  }\n}\n----\n\nSource will read rows from _LOCALDEV.TEST_ table and store in _testtopic_ topic.\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-connect-source-sap-hana.sh\n----\n\n=== Outbox Table: Event Router with SMT and JDBC Source Connector\n\nFolder: link:kafka-connect-source-event-router/[kafka-connect-source-event-router]\n\nIn this example, some SMT transformations (chained) are used to create an Event Router starting from an input _outbox table_.\n\nThe outbox table contains different operations for the same aggregate (_Consumer Loan_); the different operations are sent on specific topics following these routing rules:\n\n - operation: CREATE --\u003e topic: _loan_\n - operation: INSTALLMENT_PAYMENT --\u003e topic: _loan_payment_\n - operation: EARLY_LOAN_CLOSURE --\u003e topic: _loan_\n\nRecords from the outbox table are fetched using a JDBC Source Connector.\n\nRun the example:\n\n[source,bash]\n----\nscripts/bootstrap-connect-event-router.sh\n----\n\nOutbox table:\n\nimage::images/outbox_table.png[Outbox table]\n\n[source,bash]\n----\ninsert into outbox_table (id, aggregate, operation, payload, event_time) values (1, 'Consumer Loan', 'CREATE', '{\\\"event\\\": {\\\"type\\\":\\\"Mortgage Opening\\\",\\\"timestamp\\\":\\\"2023-11-20T10:00:00\\\",\\\"data\\\":{\\\"mortgageId\\\":\\\"ABC123\\\",\\\"customer\\\":\\\"John Doe\\\",\\\"amount\\\":200000,\\\"duration\\\": 20}}}','2023-11-20 10:00:00');\n\ninsert into outbox_table (id, aggregate, operation, payload, event_time) values (2, 'Consumer Loan', 'INSTALLMENT_PAYMENT', '{\\\"event\\\": {\\\"type\\\":\\\"Mortgage Opening\\\",\\\"timestamp\\\":\\\"2023-11-20T10:00:00\\\",\\\"data\\\":{\\\"mortgageId\\\":\\\"ABC123\\\",\\\"customer\\\":\\\"John Doe\\\",\\\"amount\\\":200000,\\\"duration\\\": 20}}}','2023-12-01 09:30:00');\n\ninsert into outbox_table (id, aggregate, operation, payload, event_time) values (3, 'Consumer Loan', 'EARLY_LOAN_CLOSURE', '{\\\"event\\\":{\\\"type\\\":\\\"Early Loan Closure\\\",\\\"timestamp\\\":\\\"2023-11-25T14:15:00\\\",\\\"data\\\":{\\\"mortgageId\\\":\\\"ABC12\\\",\\\"closureAmount\\\":150000,\\\"closureDate\\\":\\\"2023-11-25\\\",\\\"paymentMethod\\\":\\\"Bank Transfer\\\",\\\"transactionNumber\\\":\\\"PQR456\\\"}}}','2023-11-25 09:30:00');\n----\n\nA link:https://docs.confluent.io/kafka-connectors/jdbc/current/source-connector/overview.html[JDBC Source Connector] will be created with this link:kafka-connect-source-event-router/config/connector_jdbc_source.json[config]:\n\n[source,bash]\n----\n{\n  \"name\" : \"pgsql-sample-source\",\n  \"config\": {\n    \"connector.class\": \"io.confluent.connect.jdbc.JdbcSourceConnector\",\n    \"connection.url\": \"jdbc:postgresql://postgres:5432/postgres\",\n    \"connection.user\": \"postgres\",\n    \"connection.password\": \"postgres\",\n    \"topic.prefix\": \"\",\n    \"poll.interval.ms\" : 3600000,\n    \"table.whitelist\" : \"public.outbox_table\",\n    \"mode\":\"bulk\",\n    \"key.converter\": \"org.apache.kafka.connect.storage.StringConverter\",\n    \"transforms\":\"valueToTopic,addPrefix,removeString1,removeString2\",\n    \"transforms.valueToTopic.type\":\"io.confluent.connect.transforms.ExtractTopic$Value\",\n    \"transforms.valueToTopic.field\":\"operation\",\n    \"transforms.addPrefix.type\": \"org.apache.kafka.connect.transforms.RegexRouter\",\n    \"transforms.addPrefix.regex\": \".*\",\n    \"transforms.addPrefix.replacement\": \"loan$0\",\n    \"transforms.removeString1.type\": \"org.apache.kafka.connect.transforms.RegexRouter\",\n    \"transforms.removeString1.regex\": \"(.*)CREATE(.*)\",\n    \"transforms.removeString1.replacement\": \"$1$2\",\n    \"transforms.removeString2.type\": \"org.apache.kafka.connect.transforms.RegexRouter\",\n    \"transforms.removeString2.regex\": \"(.*)INSTALLMENT(.*)\",\n    \"transforms.removeString2.replacement\": \"$1$2\",\n    \"topic.creation.default.replication.factor\": 1,\n    \"topic.creation.default.partitions\": 1\n  }\n}\n\n----\n\nVerify topic list:\n\n[source,bash]\n----\ndocker exec -it broker kafka-topics --bootstrap-server broker:9092 --list\n\n__consumer_offsets\n_schemas\ndocker-connect-configs\ndocker-connect-offsets\ndocker-connect-status\nloan\nloan_PAYMENT\n----\n\n[source,bash]\n----\ndocker exec -it broker /bin/bash\n[appuser@broker ~]$ cat /tmp/kraft-combined-logs/loan-0/00000000000000000000.log\n�����Wz���Wz�����������������Consumer Loan\nCREATE�{\\\"event\\\": {\\\"type\\\":\\\"Mortgage Opening\\\",\\\"timestamp\\\":\\\"2023-11-20T10:00:00\\\",\\\"data\\\":{\\\"mortgageId\\\":\\\"ABC123\\\",\\\"customer\\\":\\\"John Doe\\\",\\\"amount\\\":200000,\\\"duration\\\": 20}}}\u00262023-11-20 10:00:00k'�z\u003c��Wz���Wz�����������������Consumer Loan$EARLY_LOAN_CLOSURE�{\\\"event\\\":{\\\"type\\\":\\\"Early Loan Closure\\\",\\\"timestamp\\\":\\\"2023-11-25T14:15:00\\\",\\\"data\\\":{\\\"mortgageId\\\":\\\"ABC12\\\",\\\"closureAmount\\\":150000,\\\"closureDate\\\":\\\"2023-11-25\\\",\\\"paymentMethod\\\":\\\"Bank Transfer\\\",\\\"transactionNumber\\\":\\\"PQR456\\\"}}}\u00262023-11-25 09:30:00\n----\n\n----\ndocker exec -it broker /bin/bash\n[appuser@broker ~]$ cat /tmp/kraft-combined-logs/loan_PAYMENT-0/00000000000000000000.log\n,�A��Wz���Wz�����������������Consumer Loan\u0026INSTALLMENT_PAYMENT�{\\\"event\\\": {\\\"type\\\":\\\"Mortgage Opening\\\",\\\"timestamp\\\":\\\"2023-11-20T10:00:00\\\",\\\"data\\\":{\\\"mortgageId\\\":\\\"ABC123\\\",\\\"customer\\\":\\\"John Doe\\\",\\\"amount\\\":200000,\\\"duration\\\": 20}}}\u00262023-12-01 09:30:00\n----\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-connect-event-router.sh\n----\n\n=== CDC with Debezium PostgreSQL Source Connector\n\nFolder: link:cdc-debezium-postgres/[cdc-debezium-postgres]\n\nUsage of link:https://debezium.io/documentation/reference/stable/connectors/postgresql.html[Debezium Source Connector for PostgreSQL] to send RDMS table updates into a topic.\n\nThe _debezium/debezium-connector-postgresql:1.7.1_ connector has been installed into connect docker image using confluent hub (see link:cdc-debezium-postgres/docker-compose.yml[_docker-compose.yml_] file).\n\nMore details on the connector are available at: https://docs.confluent.io/debezium-connect-postgres-source/current/overview.html.\n\nRun cluster:\n\n[source,bash]\n----\nscripts/bootstrap-cdc.sh\n----\n\nThe connector uses _pgoutput_ plugin for replication. This plug-in is always present in PostgreSQL server. The Debezium connector interprets the raw replication event stream directly into change events.\n\nVerify the existence of _account_ table and data in PostgreSQL:\n\n[source,bash]\n----\ndocker exec -it postgres psql -h localhost -p 5432 -U postgres -c 'select * from accounts;'\n----\n\n[source,bash]\n----\n user_id | username | password |    email     |         created_on         |         last_login\n---------+----------+----------+--------------+----------------------------+----------------------------\n       1 | foo      | bar      | foo@bar.com  | 2023-10-16 10:48:08.595034 | 2023-10-16 10:48:08.595034\n       2 | foo2     | bar2     | foo2@bar.com | 2023-10-16 10:48:08.596646 | 2023-10-16 10:48:08.596646\n       3 | foo3     | bar3     | foo3@bar.com | 2023-10-16 10:51:22.671384 | 2023-10-16 10:51:22.671384\n       4 | foo4     | bar4     | foo4@bar.com | 2024-02-28 12:12:08.665137 | 2024-02-28 12:12:08.665137\n----\n\nDeploy the connector:\n\n[source,bash]\n----\ncurl -v -X POST -H 'Content-Type: application/json' -d @cdc-debezium-postgres/config/debezium-source-pgsql.json http://localhost:8083/connectors\n----\n\nRun a consumer on _postgres.public.accounts_ topic and see the records:\n\n[source,bash]\n----\ndocker exec -it broker kafka-console-consumer --topic postgres.public.accounts --bootstrap-server broker:9092 --from-beginning --property print.key=true --property print.value=false\n----\n\nInsert a new record into _account_ table:\n\n[source,bash]\n----\ndocker exec -it postgres psql -h localhost -p 5432 -U postgres -c \"insert into accounts (user_id, username, password, email, created_on, last_login) values (3, 'foo3', 'bar3', 'foo3@bar.com', current_timestamp, current_timestamp);\"\n----\n\nVerify in consumer log the existence of 3 records:\n\n[source,bash]\n----\nStruct{user_id=1}\nStruct{user_id=2}\nStruct{user_id=3}\n----\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-cdc.sh\n----\n\n=== CDC with Debezium Informix Source Connector\n\nFolder: link:cdc-debezium-informix/[cdc-debezium-informix]\n\nUsage of link:https://debezium.io/documentation/reference/stable/connectors/informix.html[Debezium Source Connector for Informix] to send RDMS table updates into a topic.\n\nRun environment:\n\n[source,bash]\n----\nscripts/bootstrap-cdc-informix.sh\n----\n\nPerform the following tasks to prepare for using the Change Data Capture API and create tables on _iot_ database:\n\n[source,bash]\n----\ndocker exec -it ifx /bin/bash\nexport DBDATE=Y4MD\ndbaccess iot /opt/ibm/informix/etc/syscdcv1.sql\ndbaccess iot /tmp/informix_ddl_sample.sql\nexit\n----\n\nDeploy the connector:\n\n[source,bash]\n----\ncurl -v -X POST -H 'Content-Type: application/json' -d @cdc-debezium-informix/config/debezium-source-informix.json http://localhost:8083/connectors\n----\n\nRun a consumer on _test.informix.cust_db_ topic and see the records (expect to see 6 records):\n\n[source,bash]\n----\nkafka-avro-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic test.informix.cust_db --property schema.registry.url=http://localhost:8081\n----\n\n[source,bash]\n----\n{\"before\":null,\"after\":{\"test.informix.cust_db.Value\":{\"c_key\":\"\\u0004W\",\"c_status\":{\"string\":\"Z\"},\"c_date\":{\"int\":19100}}},\"source\":{\"version\":\"2.6.1.Final\",\"connector\":\"informix\",\"name\":\"test\",\"ts_ms\":1713272938000,\"snapshot\":{\"string\":\"first\"},\"db\":\"iot\",\"sequence\":null,\"ts_us\":1713272938000000,\"ts_ns\":1713272938000000000,\"schema\":\"informix\",\"table\":\"cust_db\",\"commit_lsn\":{\"string\":\"21484679168\"},\"change_lsn\":null,\"txId\":null,\"begin_lsn\":null},\"op\":\"r\",\"ts_ms\":{\"long\":1713272939104},\"ts_us\":{\"long\":1713272939104761},\"ts_ns\":{\"long\":1713272939104761000},\"transaction\":null}\n\n{\"before\":null,\"after\":{\"test.informix.cust_db.Value\":{\"c_key\":\"\\b®\",\"c_status\":{\"string\":\"Z\"},\"c_date\":{\"int\":18735}}},\"source\":{\"version\":\"2.6.1.Final\",\"connector\":\"informix\",\"name\":\"test\",\"ts_ms\":1713272938000,\"snapshot\":{\"string\":\"true\"},\"db\":\"iot\",\"sequence\":null,\"ts_us\":1713272938000000,\"ts_ns\":1713272938000000000,\"schema\":\"informix\",\"table\":\"cust_db\",\"commit_lsn\":{\"string\":\"21484679168\"},\"change_lsn\":null,\"txId\":null,\"begin_lsn\":null},\"op\":\"r\",\"ts_ms\":{\"long\":1713272939105},\"ts_us\":{\"long\":1713272939105769},\"ts_ns\":{\"long\":1713272939105769000},\"transaction\":null}\n\n{\"before\":null,\"after\":{\"test.informix.cust_db.Value\":{\"c_key\":\"\\r\\u0005\",\"c_status\":{\"string\":\"Z\"},\"c_date\":{\"int\":18370}}},\"source\":{\"version\":\"2.6.1.Final\",\"connector\":\"informix\",\"name\":\"test\",\"ts_ms\":1713272938000,\"snapshot\":{\"string\":\"true\"},\"db\":\"iot\",\"sequence\":null,\"ts_us\":1713272938000000,\"ts_ns\":1713272938000000000,\"schema\":\"informix\",\"table\":\"cust_db\",\"commit_lsn\":{\"string\":\"21484679168\"},\"change_lsn\":null,\"txId\":null,\"begin_lsn\":null},\"op\":\"r\",\"ts_ms\":{\"long\":1713272939105},\"ts_us\":{\"long\":1713272939105848},\"ts_ns\":{\"long\":1713272939105848000},\"transaction\":null}\n\n{\"before\":null,\"after\":{\"test.informix.cust_db.Value\":{\"c_key\":\"\\u0011\\\\\",\"c_status\":{\"string\":\"Z\"},\"c_date\":{\"int\":18004}}},\"source\":{\"version\":\"2.6.1.Final\",\"connector\":\"informix\",\"name\":\"test\",\"ts_ms\":1713272938000,\"snapshot\":{\"string\":\"true\"},\"db\":\"iot\",\"sequence\":null,\"ts_us\":1713272938000000,\"ts_ns\":1713272938000000000,\"schema\":\"informix\",\"table\":\"cust_db\",\"commit_lsn\":{\"string\":\"21484679168\"},\"change_lsn\":null,\"txId\":null,\"begin_lsn\":null},\"op\":\"r\",\"ts_ms\":{\"long\":1713272939105},\"ts_us\":{\"long\":1713272939105931},\"ts_ns\":{\"long\":1713272939105931000},\"transaction\":null}\n\n{\"before\":null,\"after\":{\"test.informix.cust_db.Value\":{\"c_key\":\"\\u0015³\",\"c_status\":{\"string\":\"Z\"},\"c_date\":{\"int\":17639}}},\"source\":{\"version\":\"2.6.1.Final\",\"connector\":\"informix\",\"name\":\"test\",\"ts_ms\":1713272938000,\"snapshot\":{\"string\":\"true\"},\"db\":\"iot\",\"sequence\":null,\"ts_us\":1713272938000000,\"ts_ns\":1713272938000000000,\"schema\":\"informix\",\"table\":\"cust_db\",\"commit_lsn\":{\"string\":\"21484679168\"},\"change_lsn\":null,\"txId\":null,\"begin_lsn\":null},\"op\":\"r\",\"ts_ms\":{\"long\":1713272939105},\"ts_us\":{\"long\":1713272939105984},\"ts_ns\":{\"long\":1713272939105984000},\"transaction\":null}\n\n{\"before\":null,\"after\":{\"test.informix.cust_db.Value\":{\"c_key\":\"\\u001A\\n\",\"c_status\":{\"string\":\"Z\"},\"c_date\":{\"int\":17274}}},\"source\":{\"version\":\"2.6.1.Final\",\"connector\":\"informix\",\"name\":\"test\",\"ts_ms\":1713272938000,\"snapshot\":{\"string\":\"last\"},\"db\":\"iot\",\"sequence\":null,\"ts_us\":1713272938000000,\"ts_ns\":1713272938000000000,\"schema\":\"informix\",\"table\":\"cust_db\",\"commit_lsn\":{\"string\":\"21484679168\"},\"change_lsn\":null,\"txId\":null,\"begin_lsn\":null},\"op\":\"r\",\"ts_ms\":{\"long\":1713272939106},\"ts_us\":{\"long\":1713272939106252},\"ts_ns\":{\"long\":1713272939106252000},\"transaction\":null}\n----\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-cdc-informix.sh\n----\n\n=== CDC with Debezium MongoDB Source Connector and Outbox Event Router\n\nFolder: link:cdc-debezium-mongo/[cdc-debezium-mongo]\n\nUsage of link:https://debezium.io/documentation/reference/stable/connectors/mongodb.html[Debezium Source Connector for MongoDB] to send updates into a topic. This example will use Debezium Event Router to implement a scenario for the Outbox pattern.\n\nRun environment:\n\n[source,bash]\n----\nscripts/bootstrap-cdc-mongo.sh\n----\n\nScript will create a user _data-platform-cdc_ with the privileges required to run CDC:\n\n[source,bash]\n----\nuse admin\ndb.createRole({\n   role: \"CDCRole\",\n   privileges: [\n      { resource: { cluster: true }, actions: [\"find\", \"changeStream\"] },\n      { resource: { db: \"outbox\", collection: \"loans\" }, actions: [ \"find\", \"changeStream\" ] }\n   ],\n   roles: []\n});\n\ndb.createUser({\n  user: \"data-platform-cdc\",\n  pwd: \"password\",\n  roles: [\n    { role: \"read\", db: \"admin\" },\n    { role: \"clusterMonitor\", db: \"admin\" },\n    { role: \"read\", db: \"config\" },\n    { role: \"read\", db: \"outbox\" },\n    { role: \"CDCRole\", db: \"admin\"}\n  ]\n});\n----\n\nScript will also insert a document into _outbox_ database and _loans_ collection:\n\n[source,bash]\n----\n{\n  \"aggregateId\": \"012313\",\n  \"aggregateType\": \"Consumer Loan\",\n  \"topicName\": \"CONSUMER_LOAN\",\n  \"eventDate\": \"2024-08-20T09:42:02.665Z\",\n  \"eventId\": 1,\n  \"eventType\": \"INSTALLMENT_PAYMENT\",\n  \"payload\": {\n    \"amount\": \"200000\"\n  }\n}\n----\n\nScript will deploy a source connector. Topic destination will be extracted from field _topicName_ in document. Message Key will be set using filed _aggregateId_:\n\n[source,json]\n----\n{\n  \"name\": \"mongo-debezium-connector\",\n  \"config\": {\n    \"connector.class\": \"io.debezium.connector.mongodb.MongoDbConnector\",\n    \"mongodb.connection.string\": \"mongodb://mongo:27017/?replicaSet=rs0\",\n    \"topic.prefix\": \"test\",\n    \"database.include.list\" : \"outbox\",\n    \"collection.include.list\" : \"outbox.loans\",\n    \"mongodb.user\" : \"data-platform-cdc\",\n    \"mongodb.password\" : \"password\",\n    \"value.converter\": \"org.apache.kafka.connect.json.JsonConverter\",\n    \"value.converter.schemas.enable\": \"true\",\n    \"transforms\": \"outbox,unwrap\",\n    \"transforms.outbox.type\": \"io.debezium.connector.mongodb.transforms.outbox.MongoEventRouter\",\n    \"transforms.outbox.tracing.span.context.field\": \"propagation\",\n    \"transforms.outbox.tracing.with.context.field.only\": \"false\",\n    \"transforms.outbox.tracing.operation.name\": \"debezium-read\",\n    \"transforms.outbox.collection.field.event.key\": \"aggregateId\",\n    \"transforms.outbox.collection.field.event.id\": \"aggregateId\",\n    \"transforms.outbox.collection.field.event.payload\": \"payload\",\n    \"transforms.outbox.collection.expand.json.payload\": \"true\",\n    \"transforms.outbox.collection.fields.additional.placement\": \"aggregateType:header:aggregateType,eventDate:header:eventTime,eventType:header:type,eventId:header:id\",\n    \"transforms.outbox.route.by.field\": \"topicName\",\n    \"transforms.outbox.route.topic.replacement\": \"${routedByValue}\",\n    \"transforms.unwrap.type\": \"io.debezium.connector.mongodb.transforms.ExtractNewDocumentState\",\n    \"transforms.unwrap.drop.tombstones\": \"false\",\n    \"transforms.unwrap.operation.header\": \"false\",\n    \"transforms.unwrap.delete.handling.mode\": \"drop\",\n    \"transforms.unwrap.array.encoding\": \"array\"\n  }\n}\n----\n\nRun a consumer on _CONSUMER_LOAN_ topic and see the records _(headers - key - value)_:\n\n[source,bash]\n----\ndocker exec -it broker kafka-console-consumer --bootstrap-server broker:9092 --topic CONSUMER_LOAN --from-beginning --property print.key=true --property print.headers=true\n----\n\n[source,bash]\n----\nid:012313,aggregateType:Consumer Loan,eventTime:2024-08-20T09:42:02.665Z,type:INSTALLMENT_PAYMENT,id:1\t012313\t{\"amount\":\"200000\"}\n----\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-cdc-mongo.sh\n----\n\n=== Tasks distributions using a Datagen Source Connector\n\nFolder: link:kafka-connect-task-distribution/[kafka-connect-task-distribution]\n\nThis example will show how tasks are automatically balanced between Running worker nodes.\n\nA connect cluster will be created with 2 workers, _connect_ and _connect2_ and using a link:https://docs.confluent.io/kafka-connectors/datagen/current/overview.html[_Datagen_ Source Connector] with 4 tasks continuously inserting data.\n\nAfter some seconds _connect2_ will be stopped and all tasks will be redistributed to _connect_ worker node.\n\nRun sample:\n\n[source,bash]\n----\nscripts/bootstrap-connect-tasks.sh\n----\n\nYou will first see tasks distributed between the 2 Running workers:\n\n[source,bash]\n----\n{\"datagen-sample\":{\"status\":{\"name\":\"datagen-sample\",\"connector\":{\"state\":\"RUNNING\",\"worker_id\":\"connect:8083\"},\"tasks\":[{\"id\":0,\"state\":\"RUNNING\",\"worker_id\":\"connect:8083\"},{\"id\":1,\"state\":\"RUNNING\",\"worker_id\":\"connect2:8083\"},{\"id\":2,\"state\":\"RUNNING\",\"worker_id\":\"connect:8083\"},{\"id\":3,\"state\":\"RUNNING\",\"worker_id\":\"connect2:8083\"}],\"type\":\"source\"}}}\n----\n\nAfter stopping _connect2_, you will see tasks only distributed to _connect_ worker:\n\n[source,bash]\n----\n{\"datagen-sample\":{\"status\":{\"name\":\"datagen-sample\",\"connector\":{\"state\":\"RUNNING\",\"worker_id\":\"connect:8083\"},\"tasks\":[{\"id\":0,\"state\":\"RUNNING\",\"worker_id\":\"connect:8083\"},{\"id\":1,\"state\":\"RUNNING\",\"worker_id\":\"connect:8083\"},{\"id\":2,\"state\":\"RUNNING\",\"worker_id\":\"connect:8083\"},{\"id\":3,\"state\":\"RUNNING\",\"worker_id\":\"connect:8083\"}],\"type\":\"source\"}}}\n----\n\nTeardown:\n\n[source,bash]\n----\nscripts/tear-down-connect-tasks.sh\n----\n\n\u003c\u003c\u003c\n\n== Apache Kafka® Streams\n\nFolder: link:kafka-streams/[kafka-streams]\n\nImplementation of a series of Apache Kafka® Streams topologies.\n\nExecute tests:\n\n[source,bash]\n----\ncd kafka-streams\nmvn clean test\n----\n\n=== Events counter Stream +\n\nCount number of events grouped by key.\n\nCreate topics:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic counter-input-topic --replication-factor 1 --partitions 2\n\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic counter-output-topic --replication-factor 1 --partitions 2\n----\n\nRun the topology:\n\n[source,bash]\n----\ncd kafka-streams\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.streams.stream.StreamCounter\"\n----\n\nSend messages to _counter-input-topic_ topics:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic counter-input-topic --property \"parse.key=true\" --property \"key.separator=:\"\n\"John\":\"transaction_1\"\n\"Mark\":\"transaction_1\"\n\"John\":\"transaction_2\"\n----\n\nRead from _counter-output-topic_ topic:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-consumer.sh --topic counter-output-topic --bootstrap-server broker:9092 --from-beginning --property print.key=true --property key.separator=\" : \" --value-deserializer \"org.apache.kafka.common.serialization.LongDeserializer\"\n----\n\n=== Sum values Stream +\n\nSum values grouping by key.\n\nCreate topics:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic sum-input-topic --replication-factor 1 --partitions 2\n\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic sum-output-topic --replication-factor 1 --partitions 2\n----\n\nRun the topology:\n\n[source,bash]\n----\ncd kafka-streams\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.streams.stream.StreamSum\"\n----\n\nSend messages to _sum-input-topic_ topics:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic sum-input-topic --property \"parse.key=true\" --property \"key.separator=:\"\n\"John\":1\n\"Mark\":2\n\"John\":5\n----\n\nRead from _sum-output-topic_ topic:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-consumer.sh --topic sum-output-topic --bootstrap-server broker:9092 --from-beginning --property print.key=true --property key.separator=\" : \" --value-deserializer \"org.apache.kafka.common.serialization.IntegerDeserializer\"\n----\n\n=== Cars sensor Stream +\n\nThe stream filters out speed data from car data sensor records. Speed limit is set to 150km/h and only events exceeding the limits are filtered out. +\n\nA KTable stores the car info data. +\nA left join between the KStream and the KTable produces a new aggregated object published to an output topic.\n\nCreate _carinfo-topic_, _carsensor-topic_ and _carsensor-output-topic_ topics:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic carinfo-topic --replication-factor 1 --partitions 2\n\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic carsensor-topic --replication-factor 1 --partitions 2\n\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic carsensor-output-topic --replication-factor 1 --partitions 2\n----\n\nRun the topology:\n\n[source,bash]\n----\ncd kafka-streams\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.streams.stream.CarSensorStream\"\n----\n\nSend messages to _carinfo-topic_ and _carsensor-topic_ topics:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic carinfo-topic --property \"parse.key=true\" --property \"key.separator=:\"\n1:{\"id\":\"1\",\"brand\":\"Ferrari\",\"model\":\"F40\"}\n----\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic carsensor-topic --property \"parse.key=true\" --property \"key.separator=:\"\n1:{\"id\":\"1\",\"speed\":350}\n----\n\nRead from _carsensor-output-topic_ topic:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-consumer.sh --topic carsensor-output-topic --bootstrap-server broker:9092 --from-beginning --property print.key=true --property key.separator=\" : \"\n----\n\n=== Cars brand Stream +\n\nThe stream splits the original data into 2 different topics, one for Ferrari cars and one for all other car brands.\n\nCreate _cars-input-topic_, _ferrari-input-topic_ and _cars-output-topic_ topics:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic cars-input-topic --replication-factor 1 --partitions 2\n\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic ferrari-input-topic --replication-factor 1 --partitions 2\n\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic cars-output-topic --replication-factor 1 --partitions 2\n----\n\nRun the topology:\n\n[source,bash]\n----\ncd kafka-streams\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.streams.stream.CarBrandStream\"\n----\n\nSend messages to _cars-input-topic_ topic:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic cars-input-topic --property \"parse.key=true\" --property \"key.separator=:\"\n1:{\"id\":\"1\",\"brand\":\"Ferrari\",\"model\":\"F40\"}\n2:{\"id\":\"2\",\"brand\":\"Bugatti\",\"model\":\"Chiron\"}\n----\n\nRead from _ferrari-input-topic_ and _cars-output-topic_ topics:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-consumer.sh --topic ferrari-input-topic --bootstrap-server broker:9092 --from-beginning --property print.key=true --property key.separator=\" : \"\n----\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-consumer.sh --topic cars-output-topic --bootstrap-server broker:9092 --from-beginning --property print.key=true --property key.separator=\" : \"\n----\n\n=== JSONArray Fields removal Processor +\n\nFolder: link:kafka-streams-processor/[kafka-streams-processor]\n\nRemove a specific json field from the record and forward it to the next topology node.\nThis example uses Kafka Streams Processor API.\n\nExecute tests:\n\n[source,bash]\n----\ncd kafka-streams-processor\nmvn clean test\n----\n\nCreate _processor-input-topic_ and _processor-output-topic_ topics:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic processor-input-topic --replication-factor 1 --partitions 2\n\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic processor-output-topic --replication-factor 1 --partitions 2\n----\n\nRun the topology:\n\n[source,bash]\n----\ncd kafka-streams\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.streams.processor.JSONArrayRemoveProcessorApplication\"\n----\n\nSend messages to _processor-input-topic_ topics:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic processor-input-topic --property \"parse.key=true\" --property \"key.separator=:\"\n1:{\"id\":\"1\",\"brand\":\"Ferrari\",\"model\":\"F40\"}\n----\n\nRead from _processor-output-topic_ topic:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-consumer.sh --topic processor-output-topic --bootstrap-server broker:9092 --from-beginning --property print.key=true --property key.separator=\" : \"\n----\n\n=== Expired Messages Processor +\n\nFolder: link:kafka-streams-processor/[kafka-streams-processor]\n\nRemove old entries based on time (expiration time set to 30 seconds) using a punctuator.\nThis example uses Kafka Streams Processor API.\n\nExecute tests:\n\n[source,bash]\n----\ncd kafka-streams-processor\nmvn clean test\n----\n\nCreate _expired-messages-input-topic_ and _expired-messages-output-topic_ topics:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic expired-messages-input-topic --replication-factor 1 --partitions 2\n\ndocker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker:9092 --create --topic expired-messages-output-topic --replication-factor 1 --partitions 2\n----\n\nRun the topology:\n\n[source,bash]\n----\ncd kafka-streams-processor\nmvn clean compile \u0026\u0026 mvn exec:java -Dexec.mainClass=\"org.hifly.kafka.demo.streams.processor.ExpiredMessagesApplication\"\n----\n\nSend messages to _expired-messages-input-topic_ topics:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-producer.sh --broker-list broker:9092 --topic expired-messages-input-topic --property \"parse.key=true\" --property \"key.separator=:\"\n1:{\"id\":\"1\",\"remote-device\":\"R01\",\"time\":\"2021-11-02T02:50:12.208Z\"}\n----\n\nRead from _expired-messages-output-topic_ topic:\n\n[source,bash]\n----\ndocker exec -it broker /opt/kafka/bin/kafka-console-consumer.sh --topic expired-messages-output-topic --bootstrap-server broker:9092 --from-beginning --property print.key=true --property key.separator=\" : \"\n----\n\nFind out in kafka streams application log, expiration entries:\n\n[source,bash]\n----\n[expired-messages-app-073d6f11-585b-4e69-b91f-bc998bdf49f3-StreamThread-1] INFO org.hifly.kafka.demo.streams.processor.ExpiredMessagesProcessor - 1 is expired --\u003e Sessions between:99849\n----\n\n=== Interactive queries\n\nFolder: link:kafka-streams/[kafka-streams]\n\nClass link:kafka-streams/src/main/java/org/hifly/kafka/demo/streams/queries/QueryController.java[_org.hifly.kafka.demo.streams.queries.QueryController_] shows how to execute queries against the local state store. Controller is attached at the following streams example:\n\n - link:kafka-streams/src/main/java/org/hifly/kafka/demo/streams/stream/StreamCounter.java[StreamCounter]\n\nWhen running the _StreamCounter_ example, check in the terminal for entries like:\n\n[source,bash]\n----\nquery result \u003ckey\u003e: \u003cvalue\u003e\n----\n\n=== Kafka Streams application deployed on Kubernetes\n\nFolder: link:kafka-streams-k8s/[kafka-streams-k8s]\n\nIn this example a stateful Kafka Stream Application, using RocksDB will be deployed on a Kubernetes cluster using a StatefulSet.\n\nThe application uses a window store of 5 minutes duration to count the number of words.\n\nIn order to run the example you need to provision first a Kafka cluster on Kubernetes. You can follow the example in this repo in section __\"Installation on Kubernetes using Confluent For Kubernetes\" __for running a cluster on Minikube.\n\nThe application in this demo requires a kafka broker bound on address: \"kafka:9071\".\n\nAfter installed the kafka cluster on k8s, you need first to create a Docker Image for the demo application and register in your Kubernetes Docker Registry. The next commands show how to do it with Minikube and docker driver.\n\n[source,bash]\n----\ncd kafka-streams-k8s\nmvn clean package\n# Valid if you are suing minikube with docker driver\neval $(minikube docker-env)\ndocker build -t kafka-streams-app .\n----\n\nVerify that the image is listed in your Kubernetes Docker Registry:\n\n[source,bash]\n----\n$ docker images\nREPOSITORY                                TAG         IMAGE ID       CREATED         SIZE\nkafka-streams-app                         latest      9facf1537335   2 seconds ago   322MB\nconfluentinc/confluent-operator           0.1033.33   1223f152dab7   6 weeks ago     146MB\n----\n\nCreate the topics required for the demo:\n\n[source,bash]\n----\nkubectl exec --stdin --tty kafka-0 -- /bin/bash\n\nkafka-topics --bootstrap-server localhost:9092 --create --topic words-input-topic --partitions 6\nkafka-topics --bootstrap-server localhost:9092 --create --topic words-counter-output-topic --partitions 6\nexit\n----\n\nNow you can deploy the kafka stream application on Kubernetes suing a StatefulSet and a Headless Service:\n\n[source,bash]\n----\nkubectl apply -f k8s/statefulset.yml\nkubectl apply -f k8s/svc.yml\nkubectl apply -f k8s/svc-promethues.yml\n----\n\nVerify that 3 instances of the application are running:\n\n[source,bash]\n----\n$ kubectl get pods\nNAME                                 READY   STATUS    RESTARTS   AGE\nconfluent-operator-76c899cf8-pl6ng   1/1     Running   0          3m6s\nkafka-0                              1/1     Running   0          96s\nkafka-1                              1/1     Running   0          96s\nkafka-2                              1/1     Running   0          96s\nkafka-streams-app-0                  1/1     Running   0          43s\nkafka-streams-app-1                  1/1     Running   0          40s\nkafka-streams-app-2                  1/1     Running   0          38s\nkraftcontroller-0                    1/1     Running   0          2m49s\n----\n\nVerify the PV for rocksdb (1GB each):\n\n[source,bash]\n----\n$ kubectl get pv\nNAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                           STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE\npvc-52323f68-fab8-40b1-b1d1-4c7da9e214d8   1Gi        RWO            Delete           Bound    confluent/data0-kafka-2                         standard       \u003cunset\u003e                          5m42s\npvc-7858f673-19fa-4e8c-80b0-c90b2481db8a   1Gi        RWO            Delete           Bound    confluent/data0-kafka-0                         standard       \u003cunset\u003e                          5m42s\npvc-af3d40d8-4ca8-457a-a8a2-e4f9d0d640a6   1G         RWO            Delete           Bound    confluent/data0-kraftcontroller-0               standard       \u003cunset\u003e                          6m56s\npvc-b226b4f3-53a3-4c98-ab02-49e88397ea8c   1Gi        RWO            Delete           Bound    confluent/rocksdb-storage-kafka-streams-app-2   standard       \u003cunset\u003e                          3m9s\npvc-c3c8459c-9228-491d-91ae-c5af4d47e0e7   1Gi        RWO            Delete           Bound    confluent/rocksdb-storage-kafka-streams-app-0   standard       \u003cunset\u003e                          3m9s\npvc-cf5471eb-85e1-42d3-a8a3-a74fbe7c9b5f   1Gi        RWO            Delete           Bound    confluent/data0-kafka-1                         standard       \u003cunset\u003e                          5m42s\npvc-f43467a9-b644-4ef5-8890-3c956a3805d3   1Gi        RWO            Delete           Bound    confluent/rocksdb-storage-kafka-streams-app-1   standard       \u003cunset\u003e                          3m9s\n----\n\nCheck the log of the application to evaluate the application status (RUNNING):\n\n[source,bash]\n----\n$ kubectl logs -f kafka-streams-app-0\n\n[wordscounter_app-9752ab10-9474-4b55-b73d-bb4985137ee6-StreamThread-2] INFO org.apache.kafka.streams.processor.internals.StreamThread - stream-thread [wordscounter_app-9752ab10-9474-4b55-b73d-bb4985137ee6-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING\n[wordscounter_app-9752ab10-9474-4b55-b73d-bb4985137ee6-StreamThread-2] ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhifly81%2Fkafka-examples","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhifly81%2Fkafka-examples","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhifly81%2Fkafka-examples/lists"}