Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/bygui86/go-kafka-sarama
Sample GoLang project with Kafka
https://github.com/bygui86/go-kafka-sarama
docker dockerfile go golang kafka kafka-consumer kafka-producer zookeeper
Last synced: 3 days ago
JSON representation
Sample GoLang project with Kafka
- Host: GitHub
- URL: https://github.com/bygui86/go-kafka-sarama
- Owner: bygui86
- Created: 2019-03-26T07:48:57.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2021-02-20T15:39:35.000Z (almost 4 years ago)
- Last Synced: 2024-11-08T06:07:38.435Z (about 2 months ago)
- Topics: docker, dockerfile, go, golang, kafka, kafka-consumer, kafka-producer, zookeeper
- Language: Go
- Size: 56.6 KB
- Stars: 2
- Watchers: 0
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Go Kafka
## Instructions
1. Prepare environment
```bash
mkdir -p $GOPATH/src/github.com/byui86
cd $GOPATH/src/github.com/byui86
git clone [email protected]:bygui86/go-kafka.git
cd go-kafka
go get ./...
```
2. Setup environment variables
`TODO: replace with docker network`
```bash
export DOCKER_HOST_NAME="dockerhost"
export DOCKER_HOST_IP="$(ifconfig en0 | grep inet | grep -v inet6 | awk '{print $2}')"
```
3. Run Zookeeper on Docker
```bash
docker run -d --name zookeeper --restart always --network bridge --add-host=$DOCKER_HOST_NAME:$DOCKER_HOST_IP -p 2181:2181 -p 2888:2888 -p 3888:3888 zookeeper
```
3. Run Kafka on Docker
```bash
docker run -d --name kafka --restart always -e KAFKA_ZOOKEEPER_CONNECT=dockerhost:2181 -e KAFKA_ADVERTISED_HOST_NAME=$DOCKER_HOST_IP \
--network bridge --add-host=$DOCKER_HOST_NAME:$DOCKER_HOST_IP -p 9092:9092 -p 7203:7203 wurstmeister/kafka
```
4. Create Kafka topic
```bash
docker exec -ti kafka opt/kafka_2.12-2.1.1/bin/kafka-topics.sh --create --topic gomessages --replication-factor 1 --partitions 1 --zookeeper $DOCKER_HOST_NAME:2181
```
5. List all Kafka topics
```bash
docker exec -ti kafka opt/kafka_2.12-2.1.1/bin/kafka-topics.sh --list --zookeeper $DOCKER_HOST_NAME:2181
```
6. Run producer on Docker
```bash
cd $GOPATH/src/github.com/bygui86/go-kafka/producer
docker build . --tag go-kafka-producer:1.0
docker run -ti --rm --name producer go-kafka-producer:1.0
```
7. Run consumer on Docker
```bash
cd $GOPATH/src/github.com/bygui86/go-kafka/consumer
docker build . --tag go-kafka-consumer:1.0
docker run -ti --rm --name consumer go-kafka-consumer:1.0
```
---
## Running applications without Docker
1. Run producer
```
cd $GOPATH/src/github.com/bygui86/go-kafka/producer
go get ./...
go run producer.go
```
2. Run consumer
```
cd $GOPATH/src/github.com/bygui86/go-kafka/consumer
go get ./...
go run consumer.go
```
---
## Links
- https://medium.com/@yusufs/getting-started-with-kafka-in-golang-14ccab5fa26
- https://medium.com/rahasak/kafka-and-zookeeper-with-docker-65cff2c2c34f
- https://medium.com/@itseranga/kafka-producer-with-golang-fab7348a5f9a
- https://medium.com/@itseranga/kafka-consumer-with-golang-a93db6131ac2