{"id":13588109,"url":"https://github.com/conduktor/kafka-stack-docker-compose","last_synced_at":"2025-05-13T19:16:04.457Z","repository":{"id":39900723,"uuid":"75800303","full_name":"conduktor/kafka-stack-docker-compose","owner":"conduktor","description":"docker compose files to create a fully working kafka stack","archived":false,"fork":false,"pushed_at":"2025-04-09T14:09:57.000Z","size":139,"stargazers_count":3438,"open_issues_count":6,"forks_count":1293,"subscribers_count":103,"default_branch":"master","last_synced_at":"2025-04-27T20:05:19.952Z","etag":null,"topics":["docker","docker-compose","kafka","kafka-rest-proxy","kafka-schema-registry","kafka-topics-ui","zookeeper"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/conduktor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-12-07T04:48:19.000Z","updated_at":"2025-04-27T02:06:15.000Z","dependencies_parsed_at":"2024-01-16T22:21:23.010Z","dependency_job_id":"7adf6268-7e10-4227-8fa6-37cb8e36dd31","html_url":"https://github.com/conduktor/kafka-stack-docker-compose","commit_stats":null,"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduktor%2Fkafka-stack-docker-compose","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduktor%2Fkafka-stack-docker-compose/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduktor%2Fkafka-stack-docker-compose/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduktor%2Fkafka-stack-docker-compose/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/conduktor","download_url":"https://codeload.github.com/conduktor/kafka-stack-docker-compose/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254010823,"owners_count":21999003,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","kafka","kafka-rest-proxy","kafka-schema-registry","kafka-topics-ui","zookeeper"],"created_at":"2024-08-01T15:06:30.909Z","updated_at":"2025-05-13T19:16:04.399Z","avatar_url":"https://github.com/conduktor.png","language":"Shell","readme":"[![Actions Status](https://github.com/conduktor/kafka-stack-docker-compose/workflows/CI/badge.svg)](https://github.com/conduktor/kafka-stack-docker-compose/actions)\n\n![logo](https://raw.githubusercontent.com/conduktor/conduktor.io-public/refs/heads/main/logo/logo-signature.png)\n# An open-source project by [Conduktor](https://conduktor.io/)\n\nThis project is sponsored by [Conduktor.io](https://www.conduktor.io/), the Enterprise Data Management \nPlatform for Streaming. \n\nOnce you have started your cluster, you can use Conduktor to easily manage it. \nJust connect against `localhost:9092`. If you are on Mac or Windows and want to connect from another container, use `host.docker.internal:29092`\n\n# kafka-stack-docker-compose\n\nThis replicates as well as possible real deployment configurations, where you have your zookeeper servers and Kafka servers distinct from each other. This solves all the networking hurdles that comes with Docker and Docker Compose, and is compatible cross platform.\n\n## Stack\n\n  - Conduktor Platform\n  - Zookeeper version\n  - Kafka version\n  - Kafka Schema Registry\n  - Kafka Rest Proxy\n  - Kafka Connect\n  - ksqlDB Server\n  - Zoonavigator\n\nFor a UI tool to access your local Kafka cluster, use the free version of [Conduktor](https://www.conduktor.io/get-started).\n\n# Requirements\n\nKafka will be exposed on `127.0.0.1`.\n\n## Apple M4 Support\n\nAt the time of writing there is an issue with Apple M4 chip machines and running certain Java based Docker images.\n\nModify the `conduktor.yml` file, uncomment the environment variable `CONSOLE_JAVA_OPTS: \"-XX:UseSVE=0\"`.\n\n## Full stack\n\nTo ease you journey with Kafka just connect to [localhost:8080](http://localhost:8080/)\n\n - Conduktor-platform: `$DOCKER_HOST_IP:8080`\n - Single Zookeeper: `$DOCKER_HOST_IP:2181`\n - Single Kafka: `$DOCKER_HOST_IP:9092`\n - Kafka Schema Registry: `$DOCKER_HOST_IP:8081`\n - Kafka Rest Proxy: `$DOCKER_HOST_IP:8082`\n - Kafka Connect: `$DOCKER_HOST_IP:8083`\n - KSQL Server: `$DOCKER_HOST_IP:8088`\n- (experimental) JMX port at `$DOCKER_HOST_IP:9001`\n\nRun with:\n```\ndocker compose -f full-stack.yml up\ndocker compose -f full-stack.yml down\n```\n\n## Single Zookeeper / Single Kafka\n\nThis configuration fits most development requirements.\n\n - Zookeeper will be available at `$DOCKER_HOST_IP:2181`\n - Kafka will be available at `$DOCKER_HOST_IP:9092`\n - (experimental) JMX port at `$DOCKER_HOST_IP:9999`\n\nRun with:\n```\ndocker compose -f zk-single-kafka-single.yml up\ndocker compose -f zk-single-kafka-single.yml down\n```\n\n## Single Zookeeper / Multiple Kafka\n\nIf you want to have three brokers and experiment with Kafka replication / fault-tolerance.\n\n- Zookeeper will be available at `$DOCKER_HOST_IP:2181`\n- Kafka will be available at `$DOCKER_HOST_IP:9092,$DOCKER_HOST_IP:9093,$DOCKER_HOST_IP:9094`\n\n\nRun with:\n```\ndocker compose -f zk-single-kafka-multiple.yml up\ndocker compose -f zk-single-kafka-multiple.yml down\n```\n\n## Multiple Zookeeper / Single Kafka\n\nIf you want to have three zookeeper nodes and experiment with zookeeper fault-tolerance.\n\n- Zookeeper will be available at `$DOCKER_HOST_IP:2181,$DOCKER_HOST_IP:2182,$DOCKER_HOST_IP:2183`\n- Kafka will be available at `$DOCKER_HOST_IP:9092`\n- (experimental) JMX port at `$DOCKER_HOST_IP:9999`\n\nRun with:\n```\ndocker compose -f zk-multiple-kafka-single.yml up\ndocker compose -f zk-multiple-kafka-single.yml down\n```\n\n\n## Multiple Zookeeper / Multiple Kafka\n\nIf you want to have three zookeeper nodes and three Kafka brokers to experiment with production setup.\n\n- Zookeeper will be available at `$DOCKER_HOST_IP:2181,$DOCKER_HOST_IP:2182,$DOCKER_HOST_IP:2183`\n- Kafka will be available at `$DOCKER_HOST_IP:9092,$DOCKER_HOST_IP:9093,$DOCKER_HOST_IP:9094`\n\nRun with:\n```\ndocker compose -f zk-multiple-kafka-multiple.yml up\ndocker compose -f zk-multiple-kafka-multiple.yml down\n```\n\n# FAQ\n\n## Kafka\n\n**Q: Kafka's log is too verbose, how can I reduce it?**\n\nA: Add the following line to your docker compose environment variables: `KAFKA_LOG4J_LOGGERS: \"kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO\"`. Full logging control can be accessed here: https://github.com/confluentinc/cp-docker-images/blob/master/debian/kafka/include/etc/confluent/docker/log4j.properties.template\n\n**Q: How do I delete data to start fresh?**\n\nA: Your data is persisted from within the docker compose folder, so if you want for example to reset the data in the full-stack docker compose, do a `docker compose -f full-stack.yml down`.\n\n**Q: Can I change the zookeeper ports?**\n\nA: yes. Say you want to change `zoo1` port to `12181` (only relevant lines are shown):\n```\n  zoo1:\n    ports:\n      - \"12181:12181\"\n    environment:\n        ZOO_PORT: 12181\n        \n  kafka1:\n    environment:\n      KAFKA_ZOOKEEPER_CONNECT: \"zoo1:12181\"\n```\n\n**Q: Can I change the Kafka ports?**\n\nA: yes. Say you want to change `kafka1` port to `12345` (only relevant lines are shown). Note only `LISTENER_DOCKER_EXTERNAL` changes:\n```\n  kafka1:\n    image: confluentinc/cp-kafka:7.2.1\n    hostname: kafka1\n    ports:\n      - \"12345:12345\"\n    environment:\n      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:19092,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:12345,DOCKER://host.docker.internal:29092\n```\n\n**Q: Kafka is using a lot of disk space for testing. Can I reduce it?**\n\nA: yes. This is for testing only!!! Reduce the KAFKA_LOG_SEGMENT_BYTES to 16MB and the KAFKA_LOG_RETENTION_BYTES to 128MB\n\n```\n  kafka1:\n    image: confluentinc/cp-kafka:7.2.1\n    ...\n    environment:\n      ...\n      # For testing small segments 16MB and retention of 128MB\n      KAFKA_LOG_SEGMENT_BYTES: 16777216\n      KAFKA_LOG_RETENTION_BYTES: 134217728\n```\n\n**Q: How do I expose Kafka?**\n\nA: If you want to expose Kafka outside of your local machine, you must set `KAFKA_ADVERTISED_LISTENERS` to the IP of the machine so that Kafka is externally accessible. To achieve this you can set `LISTENER_DOCKER_EXTERNAL` to the IP of the machine.\nFor example, if the IP of your machine is `50.10.2.3`, follow the sample mapping below:\n\n```\n  kafka1:\n    image: confluentinc/cp-kafka:7.2.1\n    ...\n    environment:\n      ...\n      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:19093,EXTERNAL://50.10.2.3:9093,DOCKER://host.docker.internal:29093\n```\n\n**Q: How do I add connectors to Kafka connect?**\n\nCreate a `connectors` directory and place your connectors there (usually in a subdirectory) `connectors/example/my.jar`\n\nThe directory is automatically mounted by the `kafka-connect` Docker container\n\nOR edit the bash command which pulls connectors at runtime\n\n```\nconfluent-hub install --no-prompt debezium/debezium-connector-mysql:latest\n        confluent-hub install \n```\n\n**Q: How to disable Confluent metrics?**\n\nAdd this environment variable\n```\nKAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE=false\n```\n","funding_links":[],"categories":["Shell","Testing \u0026 Development"],"sub_categories":["Local Development"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconduktor%2Fkafka-stack-docker-compose","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconduktor%2Fkafka-stack-docker-compose","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconduktor%2Fkafka-stack-docker-compose/lists"}