https://github.com/spotify/docker-kafka
Kafka (and Zookeeper) in Docker
https://github.com/spotify/docker-kafka
Last synced: 9 months ago
JSON representation
Kafka (and Zookeeper) in Docker
- Host: GitHub
- URL: https://github.com/spotify/docker-kafka
- Owner: spotify
- License: apache-2.0
- Archived: true
- Created: 2014-10-04T13:43:33.000Z (over 11 years ago)
- Default Branch: master
- Last Pushed: 2019-12-18T12:41:25.000Z (over 6 years ago)
- Last Synced: 2024-09-27T03:42:48.057Z (over 1 year ago)
- Language: Shell
- Size: 1.21 MB
- Stars: 1,399
- Watchers: 134
- Forks: 642
- Open Issues: 62
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Kafka in Docker
===
This repository provides everything you need to run Kafka in Docker.
For convenience also contains a packaged proxy that can be used to get data from
a legacy Kafka 7 cluster into a dockerized Kafka 8.
Why?
---
The main hurdle of running Kafka in Docker is that it depends on Zookeeper.
Compared to other Kafka docker images, this one runs both Zookeeper and Kafka
in the same container. This means:
* No dependency on an external Zookeeper host, or linking to another container
* Zookeeper and Kafka are configured to work together out of the box
Run
---
```bash
docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip \`docker-machine active\`` --env ADVERTISED_PORT=9092 spotify/kafka
```
```bash
export KAFKA=`docker-machine ip \`docker-machine active\``:9092
kafka-console-producer.sh --broker-list $KAFKA --topic test
```
```bash
export ZOOKEEPER=`docker-machine ip \`docker-machine active\``:2181
kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic test
```
Running the proxy
-----------------
Take the same parameters as the spotify/kafka image with some new ones:
* `CONSUMER_THREADS` - the number of threads to consume the source kafka 7 with
* `TOPICS` - whitelist of topics to mirror
* `ZK_CONNECT` - the zookeeper connect string of the source kafka 7
* `GROUP_ID` - the group.id to use when consuming from kafka 7
```bash
docker run -p 2181:2181 -p 9092:9092 \
--env ADVERTISED_HOST=`boot2docker ip` \
--env ADVERTISED_PORT=9092 \
--env CONSUMER_THREADS=1 \
--env TOPICS=my-topic,some-other-topic \
--env ZK_CONNECT=kafka7zookeeper:2181/root/path \
--env GROUP_ID=mymirror \
spotify/kafkaproxy
```
In the box
---
* **spotify/kafka**
The docker image with both Kafka and Zookeeper. Built from the `kafka`
directory.
* **spotify/kafkaproxy**
The docker image with Kafka, Zookeeper and a Kafka 7 proxy that can be
configured with a set of topics to mirror.
Public Builds
---
https://registry.hub.docker.com/u/spotify/kafka/
https://registry.hub.docker.com/u/spotify/kafkaproxy/
Build from Source
---
docker build -t spotify/kafka kafka/
docker build -t spotify/kafkaproxy kafkaproxy/
Todo
---
* Not particularily optimzed for startup time.
* Better docs