{"id":22492632,"url":"https://github.com/garystafford/kstreams-on-msk","last_synced_at":"2025-03-27T20:33:37.105Z","repository":{"id":43268090,"uuid":"468624366","full_name":"garystafford/kstreams-on-msk","owner":"garystafford","description":"Kafka KStreams example using Amazon MSK with IAM Auth","archived":false,"fork":false,"pushed_at":"2022-07-06T12:57:19.000Z","size":144,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2023-08-05T02:22:54.213Z","etag":null,"topics":["amazon-msk","java","kafka","kstreams"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/garystafford.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-11T05:44:58.000Z","updated_at":"2022-03-16T07:33:22.000Z","dependencies_parsed_at":"2022-08-22T16:11:15.622Z","dependency_job_id":null,"html_url":"https://github.com/garystafford/kstreams-on-msk","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garystafford%2Fkstreams-on-msk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garystafford%2Fkstreams-on-msk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garystafford%2Fkstreams-on-msk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garystafford%2Fkstreams-on-msk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/garystafford","download_url":"https://codeload.github.com/garystafford/kstreams-on-msk/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245920438,"owners_count":20694090,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-msk","java","kafka","kstreams"],"created_at":"2024-12-06T18:20:01.143Z","updated_at":"2025-03-27T20:33:37.068Z","avatar_url":"https://github.com/garystafford.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Streams Example for Amazon MSK with AWS IAM\n\nSimple [`WordCountLambdaExample.java`](https://github.com/confluentinc/kafka-streams-examples/blob/7.0.0-post/src/main/java/io/confluent/examples/streams/WordCountLambdaExample.java)\ncode example from Confluent, adopted for use\nwith [IAM access control for Amazon MSK](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html).\nThe example code demonstrates the use of\nthe [Apache Kafka Streams API](https://kafka.apache.org/documentation/streams/) (_aka Kafka Streams or KStreams_).\n\nFor this demonstration, the appliation was built with [Gradle 7.4](https://gradle.org/releases/) as a fat\njar (`shadowJar`) using [`com.github.johnrengelman.shadow`](https://github.com/johnrengelman/shadow). The application\nwas compiled using Java [OpenJDK 8u322](https://mail.openjdk.java.net/pipermail/jdk8u-dev/2022-January/014522.html). The\napplication was run from within an OpenJDK 8u322 container on an Amazon EKS cluster. The Kafka API commands were run from\nwithin a second container, containing the Kafka APIs, also running on the same Amazon EKS cluster. The Amazon MSK cluster was\nrunning Apache Kafka version 2.8.1.\n\n## Build, Copy, and Run the KStreams Application on EKS/MSK\n\nCommands to create the Kafka topics, build the application with Gradle, copy fat jar to an Amazon EKS pod container, run\nthe application, and produce input messages.\n\n```shell\n# build kstreams application fat jar\ngradle clean shadowJar\n```\n\n```shell\n# get kstream application pod name running on eks cluster\nexport AWS_ACCOUNT=$(aws sts get-caller-identity --output text --query 'Account')\nexport EKS_REGION=\"us-east-1\"\nexport CLUSTER_NAME=\"eks-demo-cluster\"\nexport NAMESPACE=\"kafka\"\nexport APPLICATION=\"kstreams-demo\"\nexport KAFKA_POD=$(\n  kubectl get pods -n $NAMESPACE -l app=$APPLICATION | \\\n    awk 'FNR == 2 {print $1}')\n```\n\n```shell\n# in separate terminal window, exec into kafka container running on eks cluster to run the kafka api commands\nkubectl exec -it $KAFKA_POD -n kafka -c kafka-connect -- bash\n\n# *** CHANGE ME - msk bootstrap servers ***\nexport BOOTSTRAP_SERVERS=\"b-2.msk-demo-cluster...kafka.us-east-1.amazonaws.com:9098,b-1.msk-demo-cluster...kafka.us-east-1.amazonaws.com:9098\"\n\n# create two topics\nbin/kafka-topics.sh \\\n  --bootstrap-server $BOOTSTRAP_SERVERS \\\n  --command-config config/client-iam.properties \\\n  --create \\\n  --topic streams-plaintext-input \\\n  --partitions 1 \\\n  --replication-factor 1\n\nbin/kafka-topics.sh \\\n  --bootstrap-server $BOOTSTRAP_SERVERS \\\n  --command-config config/client-iam.properties \\\n  --create \\\n  --topic streams-wordcount-output \\\n  --partitions 1 \\\n  --replication-factor 1\n\n# input text (produce messages) (NOTE kstreams app must be running first - see below)\nbin/kafka-console-producer.sh \\\n  --bootstrap-server $BOOTSTRAP_SERVERS \\\n  --producer.config config/client-iam.properties \\\n  --topic streams-plaintext-input\n\n# display word counts (consume messages) processed by kstreams app (NOTE kstreams app must be running first - see below)\nbin/kafka-console-consumer.sh \\\n  --bootstrap-server $BOOTSTRAP_SERVERS \\\n  --consumer.config config/client-iam.properties \\\n  --topic streams-wordcount-output \\\n  --from-beginning \\\n  --property print.key=true \\\n  --property value.deserializer=org.apache.kafka.common.serialization.LongDeserializer\n```\n\n```shell\n# copy kstreams application fat jar to java container running on eks cluster\nkubectl cp -n kafka -c kstreams-app build/libs/KStreamsDemo-1.0-SNAPSHOT-all.jar $KAFKA_POD:/kafka_2.13-3.1.0\n\n# in separate terminal window, exec into java container running on eks cluster to run the kstreams application\nkubectl exec -it $KAFKA_POD -n kafka -c kstreams-app -- bash\n\n# *** CHANGE ME - msk bootstrap servers ***\nexport BOOTSTRAP_SERVERS=\"b-2.msk-demo-cluster...kafka.us-east-1.amazonaws.com:9098,b-1.msk-demo-cluster...kafka.us-east-1.amazonaws.com:9098\"\n\n# run kstreams application (will run continuously)\n# with debug/verbose output\njava -verbose -Xdebug -cp KStreamsDemo-1.0-SNAPSHOT-all.jar io.confluent.examples.streams.WordCountLambdaExampleIAMAuth $BOOTSTRAP_SERVERS\n\n# without verbose output\njava -cp KStreamsDemo-1.0-SNAPSHOT-all.jar io.confluent.examples.streams.WordCountLambdaExampleIAMAuth $BOOTSTRAP_SERVERS\n```\n\n### Optional Kafka API Commands\n\n```shell\n# list all topics\nbin/kafka-topics.sh --list \\\n  --bootstrap-server $BOOTSTRAP_SERVERS \\\n  --command-config config/client-iam.properties\n\n# display input text (consume messages)\nbin/kafka-console-consumer.sh \\\n  --bootstrap-server $BOOTSTRAP_SERVERS \\\n  --consumer.config config/client-iam.properties \\\n  --topic streams-plaintext-input \\\n  --from-beginning --max-messages 100 \\\n\n# describe topic / get topic size\nbin/kafka-log-dirs.sh --describe \\\n  --bootstrap-server $BOOTSTRAP_SERVERS \\\n  --command-config config/client-iam.properties \\\n  --topic-list streams-wordcount-output\n\n# delete topics\nbin/kafka-topics.sh --delete \\\n  --bootstrap-server $BOOTSTRAP_SERVERS \\\n  --command-config config/client-iam.properties \\\n  --topic streams-plaintext-input \n\nbin/kafka-topics.sh --delete \\\n  --bootstrap-server $BOOTSTRAP_SERVERS \\\n  --command-config config/client-iam.properties \\\n  --topic streams-wordcount-output\n```\n\n## Local Docker Version of Kafka\n\nLocal Dockerized version of Apache Kafka 2.8.1 (same as Amazon MSK version used in demo) and ZooKeeper for debugging\nproject. Using unauthenticated Kafka configuration.\n\n```shell\n# create Apache Kafka and ZooKeeper containers locally\ndocker-compose up -d\n\n# exec into Kafka container to interact with Kafka\ndocker exec -it kstreamsdemo_kafka_1 bash\n\ncd ./opt/bitnami/kafka/bin/\n\n# create two topics\nkafka-topics.sh \\\n  --bootstrap-server localhost:9092 \\\n  --create \\\n  --topic streams-plaintext-input \\\n  --partitions 1 \\\n  --replication-factor 1\n\nkafka-topics.sh \\\n  --bootstrap-server localhost:9092 \\\n  --create \\\n  --topic streams-wordcount-output \\\n  --partitions 1 \\\n  --replication-factor 1\n\n# list all topics\nkafka-topics.sh \\\n  --list \\\n  --bootstrap-server localhost:9092\n\n# input text (produce messages) (NOTE kstreams app must be running first - see below)\nkafka-console-producer.sh \\\n  --bootstrap-server localhost:9092 \\\n  --topic streams-plaintext-input\n\n# display word counts (consume messages) processed by kstreams app (NOTE kstreams app must be running first - see below)\nkafka-console-consumer.sh \\\n  --bootstrap-server localhost:9092 \\\n  --topic streams-wordcount-output \\\n  --from-beginning \\\n  --property print.key=true \\\n  --property value.deserializer=org.apache.kafka.common.serialization.LongDeserializer\n\n# run kstreams application locally using dockerized version of kafka with no auth\njava -cp build/libs/KStreamsDemo-1.0-SNAPSHOT-all.jar io.confluent.examples.streams.WordCountLambdaExampleNoAuth localhost:9092\n\n# optional: delete topics\nkafka-topics.sh --delete \\\n  --bootstrap-server localhost:9092 \\\n  --topic streams-plaintext-input \n\nkafka-topics.sh --delete \\\n  --bootstrap-server localhost:9092 \\\n  --topic streams-wordcount-output\n```\n\n---\n\n\u003ci\u003eThe contents of this repository represent my viewpoints and not of my past or current employers, including Amazon Web\nServices (AWS). All third-party libraries, modules, plugins, and SDKs are the property of their respective owners. The author(s) assumes no responsibility or liability for any errors or omissions in the content of this site. The information contained in this site is provided on an \"as is\" basis with no guarantees of completeness, accuracy, usefulness or timeliness.\u003c/i\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgarystafford%2Fkstreams-on-msk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgarystafford%2Fkstreams-on-msk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgarystafford%2Fkstreams-on-msk/lists"}