Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/streamthoughts/kafka-monitoring-stack-docker-compose
This project provides Docker compose files to deploy an Apache Kafka platform with a monitoring stack using Prometheus and Grafana
https://github.com/streamthoughts/kafka-monitoring-stack-docker-compose
docker grafana kafka kafka-cluster monitor-kafka monitoring prometheus
Last synced: 3 months ago
JSON representation
This project provides Docker compose files to deploy an Apache Kafka platform with a monitoring stack using Prometheus and Grafana
- Host: GitHub
- URL: https://github.com/streamthoughts/kafka-monitoring-stack-docker-compose
- Owner: streamthoughts
- License: apache-2.0
- Created: 2019-06-27T19:40:58.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2024-03-22T19:20:17.000Z (3 months ago)
- Last Synced: 2024-03-25T08:59:28.807Z (3 months ago)
- Topics: docker, grafana, kafka, kafka-cluster, monitor-kafka, monitoring, prometheus
- Language: Shell
- Homepage:
- Size: 474 KB
- Stars: 123
- Watchers: 4
- Forks: 72
- Open Issues: 3
-
Metadata Files:
- Readme: README.adoc
- License: LICENSE
Lists
- awesome-kafka - Kafka-prometheus
README
= Kafka Monitoring Stack for Docker Compose (Prometheus / Grafana)
image:https://img.shields.io/badge/License-Apache%202.0-blue.svg[https://github.com/streamthoughts/kafka-monitoring-stack-docker-compose/blob/master/LICENSE]
image:https://img.shields.io/github/issues-raw/streamthoughts/kafka-monitoring-stack-docker-compose[GitHub issues]
image:https://img.shields.io/github/stars/streamthoughts/kafka-monitoring-stack-docker-compose?style=social[GitHub Repo stars]
:toc:
:toc-placement!:
This repository demonstrates how to use Prometheus and Grafana for monitoring an Apache Kafka cluster.
toc::[]
== Stack
* Kafka (Confluent)
* Zookeeper
* Kafka Schema Registry (Confluent)
* KSQLDB
* Kafka Connect
* Prometheus
* Grafana
* AKHQ (https://akhq.io/)
* ZooNavigator (https://zoonavigator.elkozmon.com/en/stable/)
== Usage
[source, bash]
----
$ ./up -h
--------------------------------------------------------------------------------
____ __ _____ __ _
/ __ \____ _/ /_____ _ / ___// /_________ ____ _____ ___ (_)___ ____ _
/ / / / __ `/ __/ __ `/ \__ \/ __/ ___/ _ \/ __ `/ __ `__ \/ / __ \/ __ `/
/ /_/ / /_/ / /_/ /_/ / ___/ / /_/ / / __/ /_/ / / / / / / / / / / /_/ /
/_____/\__,_/\__/\__,_/ /____/\__/_/ \___/\__,_/_/ /_/ /_/_/_/ /_/\__, /
/____/
Powered by Apache Kafka!
--------------------------------------------------------------------------------
Usage: ./up [options]
-n <[NAME]> : Name of the stack to deploy (required)
-s <[SERVICE]> : Service names to deploy (run all services if empty)
-h : Print this Help.
The table below lists the available docker-compose stacks:
NAME (./up -n ) | FILE (docker compose -f up -d)
===================================+==================================================
zk-kafka-multiple-nodes-sasl | zk-kafka-multiple-nodes-sasl-stack.yml
zk-kafka-multiple-nodes | zk-kafka-multiple-nodes-stack.yml
zk-kafka-single-node-full | zk-kafka-single-node-full-stack.yml
zk-kafka-single-node-sasl | zk-kafka-single-node-sasl-stack.yml
zk-kafka-single-node | zk-kafka-single-node-stack.yml
zkless-kafka-multiple-nodes | zkless-kafka-multiple-nodes-stack.yml
----
=== Examples
**To deploy a Kafka Cluster (Kraft)**
[source, bash]
----
$ ./up -n zkless-kafka-multiple-nodes
----
== Getting Started
**1. Clone the Kafka Monitoring Suite repository.**
[source,bash]
----
$ git clone https://github.com/streamthoughts/kafka-monitoring-stack-docker-compose.git
$ cd kafka-monitoring-stack-docker-compose
----
**2. Start Confluent/Kafka cluster.**
Deploy one of the provided docker-compose stack:
_Note: Depending on your network speed, this may take few minutes to download all images._
=== Single Zookeeper/Kafka with Prometheus/Grafana
Start/Stop with:
[source,bash]
----
$ ./zk-kafka-single-node-stack-start.sh
$ ./zk-kafka-single-node-stack-stop.sh
# or directly
$ docker-compose -f zk-kafka-single-node-stack.yml up -d
$ docker-compose -f zk-kafka-single-node-stack.yml down
----
=== Single Zookeeper / Multiple Kafka with Prometheus/Grafana
[source,bash]
----
$ docker-compose -f zk-kafka-multiple-nodes-stack.yml up -d
$ docker-compose -f zk-kafka-multiple-nodes-stack.yml down
----
=== Single Zookeeper/Kafka with Prometheus/Grafana (SASL/PLAINTEXT)
[source,bash]
----
$ ./zk-kafka-single-node-sasl-stack-start.sh
$ ./zk-kafka-single-node-sasl-stack-stop.sh
----
=== Single Zookeeper / Multiple Kafka with Prometheus/Grafana (SASL/PLAINTEXT)
[source,bash]
----
$ docker-compose -f zk-kafka-multiple-nodes-sasl-stack.yml up -d
$ docker-compose -f zk-kafka-multiple-nodes-sasl-stack.yml down
----
=== Full Stack (single node Kafka Cluster)
Start/Stop with:
[source,bash]
----
$ ./full-single-node-stack-start.sh
$ ./full-single-node-stack-stop.sh
# or directly
$ docker-compose -f zk-kafka-single-node-full-stack.yml up -d
$ docker-compose -f zk-kafka-single-node-full-stack.yml down
----
**3. Create Topic.**
Create `demo-topic` with 6 partitions and 3 replicas.
[source,bash]
----
$ docker exec -it kafka101 \
kafka-topics \
--create \
--partitions 6 \
--replication-factor 3 \
--topic demo-topic \
--bootstrap-server kafka101:29092
----
**4. Produce messages.**
Open a new terminal window, generate some message to simulate producer load.
[source,bash]
----
$ docker exec -it kafka101 \
kafka-producer-perf-test \
--throughput 500 \
--num-records 100000000 \
--topic demo-topic \
--record-size 100 \
--producer-props bootstrap.servers=kafka101:29092
----
**5. Consume messages.**
Open a new terminal window, generate some message to simulate consumer load.
[source,bash]
----
$ docker exec -it kafka101 \
kafka-consumer-perf-test \
--messages 100000000 \
--timeout 1000000 \
--topic demo-topic \
--reporting-interval 1000 \
--show-detailed-stats \
--bootstrap-server kafka101:29092
----
**6. Open Grafana.**
Open your favorite web browser and open one of the provided Grafana dashboards :
* Kafka Cluster / Global Health Check
image:./assets/kafka-cluster-healthcheck.png[kafka-cluster-healthcheck]
* Kafka Cluster / Performance
image:./assets/kafka-cluster-performance.png[kafka-cluster-performance]
* Kafka Cluster / Zookeeper Connections
* Kafka Cluster / JVM & OS
* Kafka Cluster / Hard disk usage
* Kafka Cluster / Topic Logs
image:./assets/kafka-cluster-logs.png[kafka-cluster-log]
=== Accessing Grafana Web UI
Grafana is accessible at the address : http://localhost:3000
Security are :
* user : `admin`
* password : `kafka`
=== Accessing Prometheus Web UI
Prometheus is accessible at the address : http://localhost:9090
== 💡 Contributions
Any feedback, bug reports and PRs are greatly appreciated!
== 🙏 Show your support
Please ⭐ this repository to support us!
== Licence
This code base is available under the Apache License, version 2.