https://github.com/cjimti/notes-kafka

Getting started with Kafka on Docker
https://github.com/cjimti/notes-kafka

docker kafka kafka-container mq tutorial

Last synced: 1 day ago
JSON representation

Getting started with Kafka on Docker

• Host: GitHub
• URL: https://github.com/cjimti/notes-kafka
• Owner: cjimti
• License: mit
• Created: 2018-02-28T06:55:44.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2018-02-28T16:55:08.000Z (almost 7 years ago)
• Last Synced: 2025-01-01T14:29:37.851Z (4 days ago)
• Topics: docker, kafka, kafka-container, mq, tutorial
• Homepage:
• Size: 2.93 KB
• Stars: 1
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Kafka Notes

I do nearly all my development with local [Docker containers](https://hub.docker.com/u/cjimti/) these days.

I moved from big bulky Vagrant managed VirtualBoxs to small simple

Docker containers about a year ago and have not looked back. I run

containers with individual `docker run` commands or use `docker-compose`

for slightly more complex setups.

My notes for getting started with [Kafka](https://kafka.apache.org/) on

a Mac. This is for development and testing. After experimenting with a

few other options [Confluent](https://www.confluent.io/) / [Confluent

Open Source](https://www.confluent.io/product/confluent-open-source/)

seems to provide the best setup for my taste.

### Architecture

```plain

     Stream Processors

           \ /

 Producers <-> Consumers

           / \

        Connectors

```

### Setup

Run [Zookeeper](https://zookeeper.apache.org/) , Kafka and

[Schema-registry](https://docs.confluent.io/current/schema-registry/docs/index.html) as seperate Docker containers:

```bash

docker run -d --name zookeeper \

    -p 2181:2181 \

    confluent/zookeeper

    

docker run -d --name kafka \

    -p 9092:9092 \

    --link zookeeper:zookeeper \

    confluent/kafka

    

docker run -d --name schema-registry \

    -p 8081:8081 --link zookeeper:zookeeper \

    --link kafka:kafka \

    confluent/schema-registry

```

Open a shell into the running Kafka container

`docker exec -it kafka`

From inside the new Kafka container make a **test** topic:

```bash

/usr/bin/kafka-topics --create \

    --zookeeper zookeeper:2181 \

    --replication-factor 1 \

    --partitions 1 \

    --topic test

```

List available topics:

```bash

/usr/bin/kafka-topics --list \

    --zookeeper zookeeper:2181

```

### Test

Open two more terminals on your Mac. One will be used for a consumer and one will be used for a publisher

##### Kafka Consumer

Listen for messages on the new **test** topic.

```bash

docker exec -it kafka bash

/usr/bin/kafka-console-consumer --zookeeper zookeeper:2181 \

    --topic test

```

##### Kafka Producer

Produce messages for the test topic in a seperate terminal.

```bash

docker exec -it kafka bash

/usr/bin/kafka-console-producer --broker-list localhost:9092 \

    --topic test

```

## Docker Compose

The following is Docker compose version of this setup:

```yaml

version: "3.2"

networks:

  kafkanet:

    driver: bridge

services:

  zookeeper:

    image: "confluent/zookeeper"

    container_name: "zookeeper"

    networks:

      - kafkanet

    ports:

      - 2881:2181

  kafka:

    image: "confluent/kafka"

    container_name: "kafka"

    depends_on:

      - zookeeper

    networks:

      - kafkanet

    ports:

      - 9092:9092

  schema-registry:

    image: "confluent/schema-registry"

    container_name: "schema-registry"

    depends_on:

      - kafka

    networks:

      - kafkanet

    ports:

      - 8081:8081

         

```

This Kafka notes repository contains a compose file. Clone and

run this stack with `docker-compose`.

```bash

git clone [email protected]:cjimti/notes-kafka.git

cd notes-kafka

docker-compose up

```