{"id":22018431,"url":"https://github.com/flavienbwk/python-kafka-docker","last_synced_at":"2026-04-18T00:01:45.616Z","repository":{"id":56529978,"uuid":"309151119","full_name":"flavienbwk/python-kafka-docker","owner":"flavienbwk","description":"A simple implementation of a Kafka cluster (3 brokers) with 1 producer and 1 consumer, deployed with Docker.","archived":false,"fork":false,"pushed_at":"2020-11-02T10:08:50.000Z","size":304,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-23T09:35:25.563Z","etag":null,"topics":["docker","docker-compose","kafka","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flavienbwk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-01T17:33:43.000Z","updated_at":"2023-08-25T11:06:14.000Z","dependencies_parsed_at":"2022-08-15T20:30:32.226Z","dependency_job_id":null,"html_url":"https://github.com/flavienbwk/python-kafka-docker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/flavienbwk/python-kafka-docker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2Fpython-kafka-docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2Fpython-kafka-docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2Fpython-kafka-docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2Fpython-kafka-docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flavienbwk","download_url":"https://codeload.github.com/flavienbwk/python-kafka-docker/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2Fpython-kafka-docker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31950891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-17T17:29:20.459Z","status":"ssl_error","status_checked_at":"2026-04-17T17:28:47.801Z","response_time":62,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","kafka","python3"],"created_at":"2024-11-30T05:12:10.550Z","updated_at":"2026-04-18T00:01:45.599Z","avatar_url":"https://github.com/flavienbwk.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Python Kafka Docker implementation\n\nA simple implementation of a Kafka cluster (3 brokers) with 1 producer and 1 consumer, deployed with Docker.\n\nThis project includes :\n\n- 1 Kafka cluster of 3 brokers\n- 1 ZooKeeper cluster of 3 workers\n- 1 Confluent Control Center or Kowl (Kafka UI)\n- 1 Python example of producer\n- 1 Python example of consumer\n\nWe won't use [Kafka Schema Registry](https://www.youtube.com/watch?v=5fjw62LGYNg) and Avro in this implementation. To maintain consistency among your data formats, I recommend you to directly use [DataClasses](https://realpython.com/python-data-classes/) if both your producers and consumers run with Python.\n\n## 1. Configuration\n\nFirst, you need to setup your cluster to be accessible from other containers on your computer. Get your Docker network interface IP or external IPv4 :\n\n```bash\nip a | grep docker0\n```\n\n![Grep Docker IP](./images/docker_interface.png)\n\n\u003e In the image above, the IP to choose is `192.168.254.1`\n\nSet it changing `YOUR_IP` inside `.env`, `producer/.env` and `consumer/.env`\n\n\u003e Interestingly, I was not able to make Kafka listen on `0.0.0.0` as it triggers an error. That's why we need to specify the exact IP of our machine.\n\n## 2. Starting Kafka\n\nYou have two choices : starting the open-source Kafka version or the Enterprise one. The latter will allow you to benefit from all the features in the Confluent Control Center, especially the metrics.\n\nTo start the open-source cluster :\n\n```bash\ndocker-compose up -d\n```\n\nTo start the enterprise cluster :\n\n```bash\ndocker-compose -f docker-compose.enterprise.yml up -d\n```\n\n## 3. Adding and feeding a topic (producer)\n\nLet's run our producer. It will push a random sentence in the `sentences` topic every 3 seconds.\n\n```bash\ndocker-compose -f ./producer/docker-compose.yml up\n```\n\n![Producer run example](./images/producer.png)\n\n## 4. Pulling messages (consumer)\n\nLet's run our consumer. It will print messages when received :\n\n```bash\ndocker-compose -f ./consumer/docker-compose.yml up\n```\n\n![Consumer run example](./images/consumer.png)\n\n## 5. (optional) Watch the UI\n\nConnect to [`localhost:8080`](http://localhost:8080) to visualize your cluster activity\n\n\u003e Connect to [`localhost:9021`](http://localhost:9021) to visualize the **Enterprise** cluster activity\n\n![Kowl UI example](./images/UI.png)\n\n## 6. TODOs for this repo\n\n- [ ] Provide a fully working example of an SSL configuration\n- [x] Provide an example on how to delete a topic\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflavienbwk%2Fpython-kafka-docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflavienbwk%2Fpython-kafka-docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflavienbwk%2Fpython-kafka-docker/lists"}