Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/streamthoughts/kafka-monitoring-stack-docker-compose

This project provides Docker compose files to deploy an Apache Kafka platform with a monitoring stack using Prometheus and Grafana
https://github.com/streamthoughts/kafka-monitoring-stack-docker-compose

docker grafana kafka kafka-cluster monitor-kafka monitoring prometheus

Last synced: 2 months ago
JSON representation

This project provides Docker compose files to deploy an Apache Kafka platform with a monitoring stack using Prometheus and Grafana

Host: GitHub
URL: https://github.com/streamthoughts/kafka-monitoring-stack-docker-compose
Owner: streamthoughts
License: apache-2.0
Created: 2019-06-27T19:40:58.000Z (over 5 years ago)
Default Branch: master
Last Pushed: 2024-03-22T19:20:17.000Z (10 months ago)
Last Synced: 2024-05-18T22:21:44.700Z (8 months ago)
Topics: docker, grafana, kafka, kafka-cluster, monitor-kafka, monitoring, prometheus
Language: Shell
Homepage:
Size: 474 KB
Stars: 124
Watchers: 4
Forks: 73
Open Issues: 3
Metadata Files:
- Readme: README.adoc
- License: LICENSE

Awesome Lists containing this project

awesome-kafka - Kafka-prometheus

README

        = Kafka Monitoring Stack for Docker Compose (Prometheus / Grafana)

image:https://img.shields.io/badge/License-Apache%202.0-blue.svg[https://github.com/streamthoughts/kafka-monitoring-stack-docker-compose/blob/master/LICENSE]

image:https://img.shields.io/github/issues-raw/streamthoughts/kafka-monitoring-stack-docker-compose[GitHub issues]

image:https://img.shields.io/github/stars/streamthoughts/kafka-monitoring-stack-docker-compose?style=social[GitHub Repo stars]

:toc:

:toc-placement!:

This repository demonstrates how to use Prometheus and Grafana for monitoring an Apache Kafka cluster.

toc::[]

== Stack

* Kafka (Confluent)

* Zookeeper

* Kafka Schema Registry (Confluent)

* KSQLDB

* Kafka Connect

* Prometheus

* Grafana

* AKHQ (https://akhq.io/)

* ZooNavigator (https://zoonavigator.elkozmon.com/en/stable/)

== Usage

[source, bash]

----

$ ./up -h

--------------------------------------------------------------------------------

      ____        __           _____ __                            _

     / __ \____ _/ /_____ _   / ___// /_________  ____ _____ ___  (_)___  ____ _

    / / / / __ `/ __/ __ `/   \__ \/ __/ ___/ _ \/ __ `/ __ `__ \/ / __ \/ __ `/

   / /_/ / /_/ / /_/ /_/ /   ___/ / /_/ /  /  __/ /_/ / / / / / / / / / / /_/ /

  /_____/\__,_/\__/\__,_/   /____/\__/_/   \___/\__,_/_/ /_/ /_/_/_/ /_/\__, /

                                                                        /____/

 Powered by Apache Kafka!

--------------------------------------------------------------------------------

Usage: ./up [options]

	 -n <[NAME]>      	: Name of the stack to deploy (required)

	 -s <[SERVICE]>   	: Service names to deploy (run all services if empty)

	 -h               	: Print this Help.

The table below lists the available docker-compose stacks:

NAME (./up -n )              |	FILE (docker compose -f  up -d)

===================================+==================================================

zk-kafka-multiple-nodes-sasl       |	zk-kafka-multiple-nodes-sasl-stack.yml

zk-kafka-multiple-nodes            |	zk-kafka-multiple-nodes-stack.yml

zk-kafka-single-node-full          |	zk-kafka-single-node-full-stack.yml

zk-kafka-single-node-sasl          |	zk-kafka-single-node-sasl-stack.yml

zk-kafka-single-node               |	zk-kafka-single-node-stack.yml

zkless-kafka-multiple-nodes        |	zkless-kafka-multiple-nodes-stack.yml

----

=== Examples

**To deploy a Kafka Cluster (Kraft)**

[source, bash]

----

$ ./up -n zkless-kafka-multiple-nodes

----

== Getting Started

**1. Clone the Kafka Monitoring Suite repository.**

[source,bash]

----

$ git clone https://github.com/streamthoughts/kafka-monitoring-stack-docker-compose.git

$ cd kafka-monitoring-stack-docker-compose

----

**2. Start Confluent/Kafka cluster.**

Deploy one of the provided docker-compose stack:

_Note: Depending on your network speed, this may take few minutes to download all images._

=== Single Zookeeper/Kafka with Prometheus/Grafana

Start/Stop with:

[source,bash]

----

$ ./zk-kafka-single-node-stack-start.sh

$ ./zk-kafka-single-node-stack-stop.sh

# or directly

$ docker-compose -f zk-kafka-single-node-stack.yml up -d

$ docker-compose -f zk-kafka-single-node-stack.yml down

----

=== Single Zookeeper / Multiple Kafka with Prometheus/Grafana

[source,bash]

----

$ docker-compose -f zk-kafka-multiple-nodes-stack.yml up -d

$ docker-compose -f zk-kafka-multiple-nodes-stack.yml down

----

=== Single Zookeeper/Kafka with Prometheus/Grafana (SASL/PLAINTEXT)

[source,bash]

----

$ ./zk-kafka-single-node-sasl-stack-start.sh

$ ./zk-kafka-single-node-sasl-stack-stop.sh

----

=== Single Zookeeper / Multiple Kafka with Prometheus/Grafana (SASL/PLAINTEXT)

[source,bash]

----

$ docker-compose -f  zk-kafka-multiple-nodes-sasl-stack.yml up -d

$ docker-compose -f  zk-kafka-multiple-nodes-sasl-stack.yml down

----

=== Full Stack (single node Kafka Cluster)

Start/Stop with:

[source,bash]

----

$ ./full-single-node-stack-start.sh

$ ./full-single-node-stack-stop.sh

# or directly

$ docker-compose -f  zk-kafka-single-node-full-stack.yml up -d

$ docker-compose -f  zk-kafka-single-node-full-stack.yml down

----

**3. Create Topic.**

Create `demo-topic` with 6 partitions and 3 replicas.

[source,bash]

----

$ docker exec -it kafka101 \

kafka-topics \

--create \

--partitions 6 \

--replication-factor 3 \

--topic demo-topic \

--bootstrap-server kafka101:29092

----

**4. Produce messages.**

Open a new terminal window, generate some message to simulate producer load.

[source,bash]

----

$ docker exec -it kafka101 \

kafka-producer-perf-test \

--throughput 500 \

--num-records 100000000 \

--topic demo-topic \

--record-size 100 \

--producer-props bootstrap.servers=kafka101:29092

----

**5. Consume messages.**

Open a new terminal window, generate some message to simulate consumer load.

[source,bash]

----

$ docker exec -it kafka101 \

kafka-consumer-perf-test \

--messages 100000000 \

--timeout 1000000 \

--topic demo-topic \

--reporting-interval 1000 \

--show-detailed-stats \

--bootstrap-server kafka101:29092

----

**6. Open Grafana.**

Open your favorite web browser and open one of the provided Grafana dashboards :

* Kafka Cluster / Global Health Check

image:./assets/kafka-cluster-healthcheck.png[kafka-cluster-healthcheck]

* Kafka Cluster / Performance

image:./assets/kafka-cluster-performance.png[kafka-cluster-performance]

* Kafka Cluster / Zookeeper Connections

* Kafka Cluster / JVM & OS

* Kafka Cluster / Hard disk usage

* Kafka Cluster / Topic Logs

image:./assets/kafka-cluster-logs.png[kafka-cluster-log]

=== Accessing Grafana Web UI

Grafana is accessible at the address : http://localhost:3000

Security are :

* user : `admin`

* password : `kafka`

=== Accessing Prometheus Web UI

Prometheus is accessible at the address : http://localhost:9090

== 💡 Contributions

Any feedback, bug reports and PRs are greatly appreciated!

== 🙏 Show your support

Please ⭐ this repository to support us!

== Licence

This code base is available under the Apache License, version 2.