{"id":26906754,"url":"https://github.com/dmartingarcia/docker-kafka","last_synced_at":"2026-04-11T21:34:37.093Z","repository":{"id":283924003,"uuid":"951351166","full_name":"dmartingarcia/docker-kafka","owner":"dmartingarcia","description":"a dockerised kafka environment","archived":false,"fork":false,"pushed_at":"2025-03-23T17:10:35.000Z","size":11,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-09T04:44:50.918Z","etag":null,"topics":["docker","docker-compose","kafka","kraft"],"latest_commit_sha":null,"homepage":"","language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dmartingarcia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-03-19T14:43:19.000Z","updated_at":"2025-03-23T17:10:39.000Z","dependencies_parsed_at":"2025-03-23T04:25:28.007Z","dependency_job_id":"aa6a1223-c33c-44dc-b664-752e22105e59","html_url":"https://github.com/dmartingarcia/docker-kafka","commit_stats":null,"previous_names":["dmartingarcia/docker-kafka"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dmartingarcia/docker-kafka","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmartingarcia%2Fdocker-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmartingarcia%2Fdocker-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmartingarcia%2Fdocker-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmartingarcia%2Fdocker-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dmartingarcia","download_url":"https://codeload.github.com/dmartingarcia/docker-kafka/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmartingarcia%2Fdocker-kafka/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31696743,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-11T21:17:31.016Z","status":"ssl_error","status_checked_at":"2026-04-11T21:17:24.556Z","response_time":54,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","kafka","kraft"],"created_at":"2025-04-01T11:18:36.473Z","updated_at":"2026-04-11T21:34:37.042Z","avatar_url":"https://github.com/dmartingarcia.png","language":"Makefile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Cluster with KRaFT\n\nThis project sets up a secure 3-node Kafka cluster using KRaFT (without Zookeeper) in Docker.\n\n## About KRaFT\n\nThis cluster uses Kafka's KRaFT (Kafka Raft) consensus mechanism, which replaced ZooKeeper as of Kafka 3.x. KRaFT advantages:\n- Simplified architecture (no ZooKeeper dependency)\n- Better scalability\n- Improved stability\n- Reduced operational complexity\n\nEach node in this cluster acts as both:\n- A controller (handling metadata)\n- A broker (handling data)\n\n## Architecture\n\nThe cluster consists of 3 Kafka nodes:\n- kafka1: Accessible at localhost:9092 (broker) and 9093 (controller)\n- kafka2: Accessible at localhost:9094 (broker) and 9095 (controller)\n- kafka3: Accessible at localhost:9096 (broker) and 9097 (controller)\n\nEach node runs both broker and controller roles using KRaFT consensus protocol.\n\n## Prerequisites\n\n- Docker Engine 20.10+\n- Docker Compose v2.0+\n- Make (optional)\n\n## Project Structure\n\n```\nkafka-docker/\n├── config/\n│   └── server.properties.template\n├── docker/\n│   └── Dockerfile\n├── docker-compose.yml\n├── docker-entrypoint.sh\n├── .env\n├── Makefile\n└── README.md\n```\n\n## Configuration\n\nThe cluster configuration is managed through environment variables in the `docker-compose.yml` file:\n\n```properties\nKAFKA_CLUSTER_ID=KRaft-Cluster-1\nKAFKA_VERSION=4.0.0\nKAFKA_NUM_PARTITIONS=3\nKAFKA_REPLICATION_FACTOR=3\nKAFKA_MIN_INSYNC_REPLICAS=2\n```\n\n## Security Considerations\n\n\u003e ⚠️ **Important Security Notice**: The current setup doesn't include authentication and encryption.\n\u003e For production environments, you should enable:\n\n1. **TLS Encryption**:\n   - Generate SSL certificates for each broker\n   - Configure SSL listeners\n   - Enable SSL for inter-broker communication\n\n2. **Authentication**:\n   - Set up SASL authentication (PLAIN, SCRAM, or OAUTHBEARER)\n   - Configure ACLs for access control\n   - Enable audit logging\n\nExample secure configuration:\n\n```properties\n# Security settings to add in server.properties\nlisteners=SASL_SSL://:9092,CONTROLLER://:9093\nsecurity.protocol=SASL_SSL\nssl.keystore.location=/path/to/kafka.keystore.jks\nssl.keystore.password=${SSL_KEYSTORE_PASSWORD}\nssl.key.password=${SSL_KEY_PASSWORD}\nsasl.enabled.mechanisms=SCRAM-SHA-512\nsasl.mechanism.controller.protocol=SCRAM-SHA-512\n```\n\n## Getting Started\n\n1. **Clone and setup:**\n   ```bash\n   git clone git@github.com:dmartingarcia/docker-kafka.git\n   cd kafka-docker\n   ```\n\n2. **Start the cluster:**\n   ```bash\n   make build    # Build images\n   make run      # Start cluster\n   ```\n\n3. **Verify the setup:**\n   ```bash\n   make status   # Check container status\n   make logs     # View logs\n   ```\n\n## Management Commands\n\n### Cluster Operations\n```bash\nmake build           # Build images\nmake run            # Start cluster\nmake down           # Stop cluster\nmake restart        # Restart cluster\nmake status         # Show status\nmake logs           # View logs\nmake clean          # Remove containers\nmake clean-volumes  # Remove volumes\nmake purge          # Complete cleanup\n```\n\n### Kafka Operations\n```bash\nmake topics                    # List topics\nmake create-topic TOPIC=test   # Create topic\nmake describe-topic TOPIC=test # Show topic details\nmake produce TOPIC=test        # Produce messages\nmake consume TOPIC=test        # Consume messages\nmake describe-groups          # List consumer groups\nmake cluster-status          # Show metadata\n```\n\n## Web UI Access\n\nThe Kafka UI is available at http://localhost:8080, providing:\n- Cluster overview\n- Topic management\n- Message browser\n- Consumer group monitoring\n\n## Data Persistence\n\nData is persisted in Docker volumes:\n- `kafka1_data`: Node 1 data\n- `kafka2_data`: Node 2 data\n- `kafka3_data`: Node 3 data\n\n## License\n\nMIT License - See LICENSE file for details\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmartingarcia%2Fdocker-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdmartingarcia%2Fdocker-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmartingarcia%2Fdocker-kafka/lists"}