{"id":26862093,"url":"https://github.com/luismr/kafka-cluster-docker-compose","last_synced_at":"2026-05-06T20:35:05.332Z","repository":{"id":285306679,"uuid":"957672502","full_name":"luismr/kafka-cluster-docker-compose","owner":"luismr","description":"This setup provides a Kafka cluster using KRaft mode (ZooKeeper-free) with 3 brokers. Each broker is configured as both a broker and a controller, using plaintext communication and supporting automatic topic replication","archived":false,"fork":false,"pushed_at":"2025-03-30T23:52:04.000Z","size":776,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-31T00:21:45.650Z","etag":null,"topics":["docker","docker-compose","kafka","kafka-cluster","kafka-cluster-setup","kafka-kraft"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/luismr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-30T23:06:15.000Z","updated_at":"2025-03-30T23:52:07.000Z","dependencies_parsed_at":"2025-03-31T00:21:51.862Z","dependency_job_id":"f0523060-7343-47c1-a927-b26a7fa8e10d","html_url":"https://github.com/luismr/kafka-cluster-docker-compose","commit_stats":null,"previous_names":["luismr/kafka-cluster-docker-compose"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luismr%2Fkafka-cluster-docker-compose","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luismr%2Fkafka-cluster-docker-compose/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luismr%2Fkafka-cluster-docker-compose/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luismr%2Fkafka-cluster-docker-compose/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/luismr","download_url":"https://codeload.github.com/luismr/kafka-cluster-docker-compose/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246404400,"owners_count":20771605,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","kafka","kafka-cluster","kafka-cluster-setup","kafka-kraft"],"created_at":"2025-03-31T02:23:00.366Z","updated_at":"2026-05-06T20:35:05.327Z","avatar_url":"https://github.com/luismr.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Cluster Docker Compose Setup\n\n![Docker](https://img.shields.io/badge/Docker-27.2.0-orange)\n![Docker Compose](https://img.shields.io/badge/Docker%20Compose-v1.29.2--desktop.2-orange)\n![Kafka](https://img.shields.io/badge/Kafka-3.x-red)\n\n## Overview\nThis setup provides a Kafka cluster using KRaft mode (ZooKeeper-free) with 3 brokers. Each broker is configured as both a broker and a controller, using plaintext communication and supporting automatic topic replication.\n\n![Docker-compose Stack](docs/docker-compose-stack.png)\n\n## File Structure\n\nThe following is an overview of the key files and directories in this repository:\n\n```\n.\n├── docker-compose.yml\n├── scripts/\n├── docs/\n│   ├── docker-compose-stack.png\n│   ├── docker-compose-stack.json\n│   └── kafdrop.png\n├── .env.example\n└── LICENSE.md\n```\n\n- **docker-compose.yml**: Defines the services, networks, and volumes for the Kafka cluster setup.\n- **scripts/**: Contains scripts used for initializing and managing the Kafka brokers.\n- **docs/**: Contains documentation and images related to the project, such as diagrams and UI screenshots.\n  - **docker-compose-stack.png**: An image file depicting the Docker Compose stack.\n  - **docker-compose-stack.json**: A JSON file for editing the Docker Compose stack diagram in Excalidraw.\n  - **kafdrop.png**: An image file related to Kafdrop, possibly a diagram or screenshot.\n- **.env.example**: Template for environment variables, including the `KAFKA_CLUSTER_ID`.\n- **LICENSE.md**: Contains the MIT License for the project.\n\n## Checking Out the Repository\nTo clone the repository, use the following command:\n```bash\ngit clone git@github.com:luismr/kafka-cluster-docker-compose.git\n```\n\n## Setting Up the Network\nBefore starting the Kafka cluster, ensure the `kafka-net` network is created. You can create it manually with:\n```bash\ndocker network create kafka-net\n```\n\n## Setting the KAFKA_CLUSTER_ID\n\nThe `KAFKA_CLUSTER_ID` is a crucial environment variable for setting up the Kafka cluster in KRaft mode. You can set this variable in two ways:\n\n### Using an .env File\n1. Rename the `.env.example` file to `.env`.\n2. Open the `.env` file and set the `KAFKA_CLUSTER_ID` variable:\n   ```\n   KAFKA_CLUSTER_ID=your-unique-cluster-id\n   ```\n3. Save the file. Docker Compose will automatically load this file and use the variable when starting the containers.\n\n### Setting Environment Variables Directly\nAlternatively, you can set the `KAFKA_CLUSTER_ID` as an environment variable in your operating system:\n\n#### On Windows\n```cmd\nset KAFKA_CLUSTER_ID=your-unique-cluster-id\n```\n\n#### On Linux/Mac\n```bash\nexport KAFKA_CLUSTER_ID=your-unique-cluster-id\n```\n\nAfter setting the environment variable, you can start the Kafka cluster with Docker Compose:\n```bash\ndocker-compose up -d\n```\n\n## Configuration\n- **KAFKA_KRAFT_CLUSTER_ID**: Unique identifier for the Kafka cluster.\n- **KAFKA_NODE_ID**: Unique ID for each broker.\n- **KAFKA_PROCESS_ROLES**: Configures the broker to act as both a broker and a controller.\n- **KAFKA_LISTENERS**: Defines the internal and external listeners for each broker.\n- **KAFKA_LISTENER_SECURITY_PROTOCOL_MAP**: Specifies the security protocol for each listener (plaintext).\n- **KAFKA_INTER_BROKER_LISTENER_NAME**: Specifies the listener used for inter-broker communication.\n- **KAFKA_CONTROLLER_LISTENER_NAMES**: Specifies the listener used for controller communication.\n- **KAFKA_ADVERTISED_LISTENERS**: Advertises the listener addresses for client connections.\n- **KAFKA_LOG_DIRS**: Directory where Kafka logs are stored.\n- **KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR**: Sets the replication factor for the offsets topic.\n- **KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR**: Sets the replication factor for the transaction state log.\n- **KAFKA_TRANSACTION_STATE_LOG_MIN_ISR**: Minimum in-sync replicas for the transaction state log.\n\n## Running the Kafka Cluster\nTo start the Kafka cluster, use the following command:\n```bash\ndocker-compose up -d\n```\nThis command will start all the services defined in the `docker-compose.yml` file in detached mode.\n\n## Connecting to the Kafka Cluster\n\nYou can connect to the Kafka cluster using various client libraries. Below is an example using Python with the `kafka-python` library.\n\n### Python Example\n\nFirst, install the `kafka-python` library:\n\n```bash\npip install kafka-python\n```\n\nThen, use the following script to produce and consume messages:\n\n```python\nfrom kafka import KafkaProducer, KafkaConsumer\n\n# Producer\nproducer = KafkaProducer(bootstrap_servers=['localhost:19092', 'localhost:19093', 'localhost:19094'])\nproducer.send('my-topic', b'Hello, Kafka!')\nproducer.flush()\n\n# Consumer\nconsumer = KafkaConsumer('my-topic', bootstrap_servers=['localhost:19092', 'localhost:19093', 'localhost:19094'])\nfor message in consumer:\n    print(f\"Received message: {message.value.decode('utf-8')}\")\n```\n\nThis example connects to the Kafka brokers running on `localhost:19092`, `localhost:19093`, and `localhost:19094`.\n\n## Running Kafka in Standalone\n\nRunning Kafka in standalone mode is important for testing and development purposes. It allows you to quickly set up a Kafka broker without the need for a full cluster, making it easier to test configurations, produce and consume messages, and develop applications locally.\n\nTo run Kafka in standalone mode, use the following command:\n\n```bash\ndocker run -d \\\n  --name broker \\\n  -p 9092:9092 \\\n  apache/kafka:latest\n```\n\n## License\n\nThis project is licensed under the MIT License. For more details, please refer to the [LICENSE.md](LICENSE.md) file.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluismr%2Fkafka-cluster-docker-compose","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fluismr%2Fkafka-cluster-docker-compose","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluismr%2Fkafka-cluster-docker-compose/lists"}