{"id":18499550,"url":"https://github.com/gusinacio/kafka-docker","last_synced_at":"2025-08-16T23:38:56.694Z","repository":{"id":71174071,"uuid":"316634836","full_name":"gusinacio/kafka-docker","owner":"gusinacio","description":"Docker compose a cluster with zookeeper and client for INE5418","archived":false,"fork":false,"pushed_at":"2020-11-28T20:02:00.000Z","size":3,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-17T00:49:36.179Z","etag":null,"topics":["docker","docker-compose","kafka"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gusinacio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-28T01:37:16.000Z","updated_at":"2023-02-10T18:40:19.000Z","dependencies_parsed_at":"2023-07-03T12:16:53.355Z","dependency_job_id":null,"html_url":"https://github.com/gusinacio/kafka-docker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gusinacio%2Fkafka-docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gusinacio%2Fkafka-docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gusinacio%2Fkafka-docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gusinacio%2Fkafka-docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gusinacio","download_url":"https://codeload.github.com/gusinacio/kafka-docker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254077094,"owners_count":22010661,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","kafka"],"created_at":"2024-11-06T13:46:27.207Z","updated_at":"2025-05-14T05:35:38.796Z","avatar_url":"https://github.com/gusinacio.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kafka-docker\n\nThis project uses `docker-compose` to execute the tutorial for INE5418 at UFSC.\n\n## Pre-requisites\n\n- Docker\n- docker-compose\n\n## Running\n\nTo start the cluster with 2 kafka brokers simply run:\n\n```\n$ docker-compose up -d\n```\n\nAfter this, you will have:\n\n- zookeper: zookeeper instance\n- broker_1: kafka instance\n- broker_2: kafka instance\n\n### Client executions\n\nWe create three scripts to run the three commands needed to accomplish the tutorial:\n\n- kafka-topics\n- kafka-console-producer\n- kafka-console-consumer\n\nThey run in docker too using following base_script:\n\n```\n#!/bin/bash\n\ndocker run -it --rm --network kafka-docker_app-tier -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 -e DISABLE_WELCOME_MESSAGE=true bitnami/kafka:2-debian-10 ${0##*/}.sh $@\n```\n\n## Tutorial Report\n\n### Start up\nWe start runing our cluster:\n\n![](https://i.imgur.com/hwSI8M8.png)\n\nAfter starting using the docker-compose command, we were asked to create a topic and list.\n\n### Managing topics\n\nWe cannot run in localhost because we are passing all these arguments inside a container. Instead of using localhost:2181 for the zookeeper instance, we use its docker hostname.\n\n```\n$ ./kafka-topics --create --zookeeper zookeeper:2181 -replication-factor 1 --partitions 1 --topic amazingTopic\n```\n\n![](https://i.imgur.com/cflG5Dn.png)\n\nWe list the topics using this:\n\n```\n$ ./kafka-topics --list --zookeeper zookeeper:2181\n```\n\n![](https://i.imgur.com/EAkC7Uv.png)\n\nAnd we described the topics:\n\n```\n$ ./kafka-topics --describe --zookeeper zookeeper:2181 --topic amazingTopic\n```\n\n![](https://i.imgur.com/nfWORfa.png)\n\nThen, we created a topic with multiple replicas:\n\n```\n$ ./kafka-topics --create --zookeeper zookeeper:2181 -replication-factor 2 --partitions 1 --topic redundantTopic\n```\n\n![](https://i.imgur.com/U4dQMlp.png)\n\nDescribing it:\n\n```\n$ ./kafka-topics --describe --zookeeper zookeeper:2181 --topic redundantTopic\n```\n\n![](https://i.imgur.com/Blk3TcT.png)\n\n### Message producer\n\nWe can produce in whatever broker we want, but like before, we won't use localhost and we will use the docker hostname instead.\n\nProduction of some messages:\n\n```\n$ ./kafka-console-producer --broker-list broker-1:9092 --topic amazingTopic\n```\n\n![](https://i.imgur.com/OUIf8GR.png)\n\n\nIf we wanted to use files, we would have to call the docker run withou the `-t` option like the following, that's why we created a custom `kafka-console-producer-pipe` script:\n\n```\n$ cat TESTE.txt | ./kafka-console-producer-pipe --broker-list broker-1:9092 --topic amazingTopic\n```\n\n![](https://i.imgur.com/1c0ISzL.png)\n\n### Message consumer\n\nConsuming a message is easy, using the docker hostname:\n\n```\n$ ./kafka-console-consumer --topic amazingTopic --bootstrap-server broker-1:9092 --from-beginning\n```\n\n![](https://i.imgur.com/uLwuHtV.png)\n\nWe could also limit the number max of messages seen:\n\n```\n$ ./kafka-console-consumer --topic amazingTopic --bootstrap-server broker-1:9092 --from-beginning --max-messages 1\n```\n\n![](https://i.imgur.com/votvbAc.png)\n\nIf you want to use a formatter, you could use:\n\n```\n$ ./kafka-console-consumer --topic amazingTopic --bootstrap-server broker-1:9092 --from-beginning --max-messages 1 --formatter 'kafka.coordinator.group.GroupMetadataManager$OffsetsMessageFormatter'\n```\n![](https://i.imgur.com/xwGywo8.png)\n\n\nTo consume messages from a specific consumer group, use the following:\n```\n$ ./kafka-console-consumer --topic amazingTopic --bootstrap-server broker-1:9092 --new-consumer --consumer-property group.id=my-group\n```\n\n## Shutdown\n\nAfter running all the tutorial, you can shutdown you docker-compose running:\n\n```\n$ docker-compose down\n```\n\n![](https://i.imgur.com/6ucT6pG.png)\n\n\nYou can clean all the saved configuration and topics created by deleting the volumes created:\n\n```\n$ docker volume ls -f \"name=kafka-docker*\" | grep \"kafka-docker\" | awk '{print $2}' | xargs docker volume rm\n```\n\n![](https://i.imgur.com/gzgUTFl.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgusinacio%2Fkafka-docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgusinacio%2Fkafka-docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgusinacio%2Fkafka-docker/lists"}