Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/vkuznet/kafka-app
Example of local kafka pipeline (setup and client)
https://github.com/vkuznet/kafka-app
Last synced: 30 days ago
JSON representation
Example of local kafka pipeline (setup and client)
- Host: GitHub
- URL: https://github.com/vkuznet/kafka-app
- Owner: vkuznet
- License: mit
- Created: 2021-05-19T18:59:42.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2021-05-23T14:44:34.000Z (over 3 years ago)
- Last Synced: 2024-10-30T06:40:25.696Z (3 months ago)
- Language: Go
- Size: 22.5 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
### Setup local kafka pipeline
To setup local kafka pipeline we need the following:
- a node with running docker daemon
- a docker compose installation
- a kafka docker github repo
```
# install docker compose
curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /wma/vk/docker-compose
export PATH=$PATH:/wma/vk
chmod +x docker-compose
# clone kafka docker github repo
git clone [email protected]:wurstmeister/kafka-docker.git
cd kafka-docker
# vim docker-compose.yml
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
build: .
ports:
- "9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 188.185.115.163
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
# start kafka
docker-compose ps
docker-compose down
docker-compose up -d
docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------------------------------------------
kafka-docker_kafka_1 start-kafka.sh Up 0.0.0.0:9094->9094/tcp,:::9094->9094/tcp
kafka-docker_zookeeper_1 /bin/sh -c /usr/sbin/sshd ... Up 0.0.0.0:2181->2181/tcp,:::2181->2181/tcp, 22/tcp,
2888/tcp, 3888/tcp
# get logs
docker-compose logs
# view existing topics with kafkacat
# see https://github.com/edenhill/kafkacat
docker run -it --network=host edenhill/kafkacat:1.6.0 -b localhost:9094 -L
# run client
cd ..
git clone [email protected]:vkuznet/kafka-app.git
cd kafka-app
go build
# we'll start our app which produce and consume messages on topic ("test")
# and using broker address localhost:9094
./kafka-app
# if we want we can consume messages using kafka CLI
docker run -it --network=host edenhill/kafkacat:1.6.0 -b localhost:9094 -C -t test -e -J
```
### References
- [Docker compose](https://docs.docker.com/compose/install/)
- [Kafka docker](https://github.com/wurstmeister/kafka-docker)
- [Kafka local setup](https://www.kimsereylam.com/kafka/docker/2020/10/16/setup-local-kafka-with-docker.html)
- [Kafka CLI](https://kimsereylam.com/kafka/2020/03/27/kafkacat-the-cli-for-kafka.html)
- [Go client](https://www.sohamkamani.com/golang/working-with-kafka/)