https://github.com/flavienbwk/python-kafka-docker
A simple implementation of a Kafka cluster (3 brokers) with 1 producer and 1 consumer, deployed with Docker.
https://github.com/flavienbwk/python-kafka-docker
docker docker-compose kafka python3
Last synced: 2 months ago
JSON representation
A simple implementation of a Kafka cluster (3 brokers) with 1 producer and 1 consumer, deployed with Docker.
- Host: GitHub
- URL: https://github.com/flavienbwk/python-kafka-docker
- Owner: flavienbwk
- Created: 2020-11-01T17:33:43.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2020-11-02T10:08:50.000Z (over 5 years ago)
- Last Synced: 2025-03-23T09:35:25.563Z (over 1 year ago)
- Topics: docker, docker-compose, kafka, python3
- Language: Python
- Homepage:
- Size: 297 KB
- Stars: 2
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Python Kafka Docker implementation
A simple implementation of a Kafka cluster (3 brokers) with 1 producer and 1 consumer, deployed with Docker.
This project includes :
- 1 Kafka cluster of 3 brokers
- 1 ZooKeeper cluster of 3 workers
- 1 Confluent Control Center or Kowl (Kafka UI)
- 1 Python example of producer
- 1 Python example of consumer
We won't use [Kafka Schema Registry](https://www.youtube.com/watch?v=5fjw62LGYNg) and Avro in this implementation. To maintain consistency among your data formats, I recommend you to directly use [DataClasses](https://realpython.com/python-data-classes/) if both your producers and consumers run with Python.
## 1. Configuration
First, you need to setup your cluster to be accessible from other containers on your computer. Get your Docker network interface IP or external IPv4 :
```bash
ip a | grep docker0
```
> In the image above, the IP to choose is `192.168.254.1`
Set it changing `YOUR_IP` inside `.env`, `producer/.env` and `consumer/.env`
> Interestingly, I was not able to make Kafka listen on `0.0.0.0` as it triggers an error. That's why we need to specify the exact IP of our machine.
## 2. Starting Kafka
You have two choices : starting the open-source Kafka version or the Enterprise one. The latter will allow you to benefit from all the features in the Confluent Control Center, especially the metrics.
To start the open-source cluster :
```bash
docker-compose up -d
```
To start the enterprise cluster :
```bash
docker-compose -f docker-compose.enterprise.yml up -d
```
## 3. Adding and feeding a topic (producer)
Let's run our producer. It will push a random sentence in the `sentences` topic every 3 seconds.
```bash
docker-compose -f ./producer/docker-compose.yml up
```
## 4. Pulling messages (consumer)
Let's run our consumer. It will print messages when received :
```bash
docker-compose -f ./consumer/docker-compose.yml up
```
## 5. (optional) Watch the UI
Connect to [`localhost:8080`](http://localhost:8080) to visualize your cluster activity
> Connect to [`localhost:9021`](http://localhost:9021) to visualize the **Enterprise** cluster activity
## 6. TODOs for this repo
- [ ] Provide a fully working example of an SSL configuration
- [x] Provide an example on how to delete a topic