{"id":23086816,"url":"https://github.com/papirosko/kafka-demo","last_synced_at":"2025-04-03T16:19:47.698Z","repository":{"id":185847804,"uuid":"671945401","full_name":"papirosko/kafka-demo","owner":"papirosko","description":"A demo how to setup kafka cluster with ui and metrics","archived":false,"fork":false,"pushed_at":"2023-08-03T16:39:18.000Z","size":19,"stargazers_count":3,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-09T04:44:54.511Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/papirosko.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-07-28T14:06:58.000Z","updated_at":"2025-01-05T08:43:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"d67b23c8-a878-4eb3-b91a-39b4131db8fa","html_url":"https://github.com/papirosko/kafka-demo","commit_stats":null,"previous_names":["papirosko/kafka-demo"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/papirosko%2Fkafka-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/papirosko%2Fkafka-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/papirosko%2Fkafka-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/papirosko%2Fkafka-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/papirosko","download_url":"https://codeload.github.com/papirosko/kafka-demo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247033815,"owners_count":20872532,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-16T19:31:16.764Z","updated_at":"2025-04-03T16:19:47.676Z","avatar_url":"https://github.com/papirosko.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Plan\nIn this article we will setup a simple kafka cluster with 2 nodes in it.\n\nWe will validate that cluster is working by producing and consuming messages. We will setup\n`kafka-ui` to see the cluster details. We will also export cluster metrics and watch them in grafana.\n\nWe will use `docker compose` for this to make things simpler. \n\n\nThe docker image for the kafka will be [bitnami/kafka/](https://hub.docker.com/r/bitnami/kafka/).\n\n# Kafka cluster\n\nLet's start with this `docker-compose.yml`:\n\n```yaml\nversion: '3.8'\n\nx-kafka-common: \u0026kafka-common\n  image: 'bitnami/kafka:latest'\n  ports:\n    - \"9092\"\n  networks:\n    - kafka\n  healthcheck:\n    test: \"bash -c 'printf \\\"\\\" \u003e /dev/tcp/127.0.0.1/9092; exit $$?;'\"\n    interval: 5s\n    timeout: 10s\n    retries: 3\n    start_period: 30s\n  restart: unless-stopped\n\nx-kafka-env-common: \u0026kafka-env-common\n  ALLOW_PLAINTEXT_LISTENER: 'yes'\n  KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: 'true'\n  KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 0@kafka-0:9093,1@kafka-1:9093\n  KAFKA_KRAFT_CLUSTER_ID: abcdefghijklmnopqrstuv\n  KAFKA_CFG_PROCESS_ROLES: controller,broker\n  KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER\n  KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093\n  EXTRA_ARGS: \"-Xms128m -Xmx256m\"\n\nservices:\n\n  kafka-0:\n    \u003c\u003c: *kafka-common\n    environment:\n      \u003c\u003c: *kafka-env-common\n      KAFKA_CFG_NODE_ID: 0\n    volumes:\n      - kafka_0_data:/bitnami/kafka\n\n  kafka-1:\n    \u003c\u003c: *kafka-common\n    environment:\n      \u003c\u003c: *kafka-env-common\n      KAFKA_CFG_NODE_ID: 1\n    volumes:\n      - kafka_1_data:/bitnami/kafka\n\nnetworks:\n  kafka:\n\nvolumes:\n  kafka_0_data:\n    driver: local\n  kafka_1_data:\n    driver: local\n```\n\n\nHere we define 2 services: `kafka-0` and `kafka-1`. \n\nWe don't need any encryption, so we use `ALLOW_PLAINTEXT_LISTENER=yes`.\nAlso we don't use zookeeper, instead we will use [KRaft protocol](https://developer.confluent.io/learn/kraft/).\n\nWe will add a healthcheck command, which will later help us to run dependent services. The reason why we do not \nuse `curl` is because the response will be empty, and it will make `curl` produce an error\n```\ncurl: (52) Empty reply from server\n```\nAs a result, node will be marked as unhealthy.\n\nOn each node we need to set its id with this variable: `KAFKA_CFG_NODE_ID`.\n\nBy setting `KAFKA_CFG_CONTROLLER_QUORUM_VOTERS` environment variable on both services, we make them to work in a cluster.\nAlso we will use random value for the cluster id with the variable `KAFKA_KRAFT_CLUSTER_ID`: just use any string.\n\nWe set also `KAFKA_CFG_PROCESS_ROLES`, `KAFKA_CFG_CONTROLLER_LISTENER_NAMES` and `KAFKA_CFG_LISTENERS` because\nthey do not have default values.\n\n\nAll other variables will use their default values, you can read more about them here: \nhttps://github.com/bitnami/containers/blob/main/bitnami/kafka/README.md#configuration.\n\n\n\nLet's run our cluster:\n\n```shell\ndocker compose up\n```\n\n# Validate cluster\n\nNow we need to create a new topic and put some messages into it.\nKafka comes with a set of scripts for managing it, let's use them:\n\n```shell\ndocker compose exec kafka-0 /opt/bitnami/kafka/bin/kafka-topics.sh --create \\\n    --bootstrap-server kafka-0:9092,kafka-1:9092 --replication-factor 1 --partitions 1 --topic test\n```\nWe tell the `kafka-topics.sh` that it should use our kafka instances as a bootstrap servers.\nAs a result you should see \n```\nCreated topic test.\n```\n\nLet's list all topics:\n```\n$ docker compose exec kafka-0 /opt/bitnami/kafka/bin/kafka-topics.sh --list --bootstrap-server kafka-0:9092,kafka-1:9092\ntest\n$ docker compose exec kafka-0 /opt/bitnami/kafka/bin/kafka-topics.sh --describe --topic test --bootstrap-server kafka-0:9092,kafka-1:9092\nTopic: test\tTopicId: 7FEciEQRRjqJ2zntIONOcQ\tPartitionCount: 1\tReplicationFactor: 1\tConfigs: segment.bytes=1073741824\n\tTopic: test\tPartition: 0\tLeader: 0\tReplicas: 0\tIsr: 0\n```\n\n\nLet's write some messages into this topic. Open new terminal (this will be producer terminal) and type:\n```\ndocker compose exec kafka-0 /opt/bitnami/kafka/bin/kafka-console-producer.sh --bootstrap-server kafka-0:9092,kafka-1:9092 --producer.config /opt/bitnami/kafka/config/producer.properties --topic test\n\u003emessage0\n```\n\nOpen another terminal (consumer terminal) and read the message with a consumer:\n```shell\ndocker compose exec kafka-0  /opt/bitnami/kafka/bin/kafka-console-consumer.sh \\\n  --bootstrap-server kafka-0:9092,kafka-1:9092 --consumer.config /opt/bitnami/kafka/config/consumer.properties --topic test --from-beginning\n```\nYou should see `message0`. Switch back to the producer terminal and type another message. You will see it immediately \nin consumer terminal.\n\n\nPress Control+C to close producer and consumer.\n\nAt this point we can start a new cluster, create topic, produce the messages and consume them.\n\n\n# Kafka UI\nNext we will add kafka-ui. This application will give us a nice UI to view our cluster.\n\nAdd the following to the `docker-compose.yml`:\n```yaml\n  kafka-ui:\n    container_name: kafka-ui\n    image: provectuslabs/kafka-ui:latest\n    volumes:\n      - ./kafka-ui/config.yml:/etc/kafkaui/dynamic_config.yaml\n    environment:\n      DYNAMIC_CONFIG_ENABLED: 'true'\n    depends_on:\n      - kafka-0\n      - kafka-1\n    networks:\n      - kafka\n    ports:\n      - '8080:8080'\n    healthcheck:\n      test: wget --no-verbose --tries=1 --spider localhost:8080 || exit 1\n      interval: 5s\n      timeout: 10s\n      retries: 3\n      start_period: 30s\n ```\n\nYou need to create a configuration file for the kafka ui:\n```shell\nmkdir -p kafka-ui\ntouch kafka-ui/config.yml\n```\nKafka ui can be configured with environment variables, but I prefer yaml, because variables will become very messy\nonce you will have more then 1 cluster.\n\nPut the following into `kafka-ui/config.yml`:\n```yaml\nauth:\n  type: LOGIN_FORM\n\nspring:\n  security:\n    user:\n      name: admin\n      password: admin\n\nkafka:\n  clusters:\n    - bootstrapServers: kafka-0:9092,kafka-1:9092\n      name: kafka\n```\n\nStop current docker compose process (control+c) and restart it:\n```shell\ndocker compose up\n```\n\n\nOpen kafka ui in the browser http://localhost:8080. Use `admin`:`admin` as a credentials (see `kafka-ui/config.yml`).\nYou will see our new cluster.\n\nVisit http://localhost:8080/ui/clusters/kafka/all-topics/test/messages?keySerde=String\u0026valueSerde=String\u0026limit=100 and \nmake sure you see all messages, that we produced earlier.\n\n\n\n# Metrics\n\n## Kafka exporter\n\nNow we need to expose kafka metrics to the prometheus and be able to see them in grafana.\n[Kafka exporter](https://github.com/danielqsj/kafka_exporter) will be used to actually export the metrics. \nLet's add new services to the `docker-compose.yml`:\n```yaml\n\n  prometheus:\n    image: prom/prometheus\n    container_name: prometheus\n    command:\n      - '--config.file=/etc/prometheus/prometheus.yml'\n    ports:\n      - 9090:9090\n    volumes:\n      - ./prometheus:/etc/prometheus\n      - prom_data:/prometheus\n    networks:\n      - kafka  \n    healthcheck:\n      test: wget --no-verbose --tries=1 --spider localhost:9090 || exit 1\n      interval: 5s\n      timeout: 10s\n      retries: 3\n      start_period: 5s\n      \n  kafka-exporter:\n    image: docker.io/bitnami/kafka-exporter:latest\n    depends_on:\n      kafka-0:\n        condition: service_healthy\n      kafka-1:\n        condition: service_healthy\n    networks:\n      - kafka\n    command: --kafka.server=kafka-0:9092 --kafka.server=kafka-1:9092\n    healthcheck:\n      test: \"bash -c 'printf \\\"\\\" \u003e /dev/tcp/127.0.0.1/9308; exit $$?;'\"\n      interval: 5s\n      timeout: 10s\n      retries: 3\n      start_period: 5s\n      \n        \n  grafana:\n    image: grafana/grafana\n    container_name: grafana\n    ports:\n      - 3000:3000\n    environment:\n      - GF_SECURITY_ADMIN_USER=admin\n      - GF_SECURITY_ADMIN_PASSWORD=grafana\n    volumes:\n      - ./grafana/provisioning:/etc/grafana/provisioning\n      - ./grafana/dashboards:/var/lib/grafana/dashboards\n    networks:\n      - kafka\n    healthcheck:\n      test: curl --fail localhost:3000\n      interval: 5s\n      timeout: 10s\n      retries: 3\n      start_period: 10s\n```\nand this in `volumes` section:\n```yaml\n  prom_data:\n    driver: local\n```\n\nKafka exporter depends on kafka instances, it will produce an error during start, if there are no running kafka brokers. \nThis is why we added the healthcheck for kafka.\n\n\n\n\n\n\n## JMX exporter\n\nThere is also one more way to collect metrics (they can be used together):\n[jmx exporter](https://github.com/prometheus/jmx_exporter). We will use it as a java agent.\n\nDownload the jar and the config:\n```shell\nmkdir -p jmx-exporter\ncurl https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.19.0/jmx_prometheus_javaagent-0.19.0.jar \\\n  -o jmx-exporter/jmx_prometheus_javaagent-0.19.0.jar\ncurl  https://raw.githubusercontent.com/prometheus/jmx_exporter/main/example_configs/kafka-2_0_0.yml \\\n  -o jmx-exporter/kafka-2_0_0.yml\n```\n\nMount it to kafka images:\n```\n  kafka-0:\n    \u003c\u003c: *kafka-common\n    environment:\n      \u003c\u003c: *kafka-env-common\n      KAFKA_CFG_NODE_ID: 0\n    volumes:\n      - kafka_0_data:/bitnami/kafka\n      - ./jmx-exporter:/opt/jmx-exporter\n\n  kafka-1:\n    \u003c\u003c: *kafka-common\n    environment:\n      \u003c\u003c: *kafka-env-common\n      KAFKA_CFG_NODE_ID: 1\n    volumes:\n      - kafka_1_data:/bitnami/kafka\n      - ./jmx-exporter:/opt/jmx-exporter\n```\n\n\nUpdate the `x-kafka-env-common` block in `docker-compose.yml` like this:\n```yaml\nx-kafka-env-common: \u0026kafka-env-common\n  ALLOW_PLAINTEXT_LISTENER: 'yes'\n  KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: 'true'\n  KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 0@kafka-0:9093,1@kafka-1:9093\n  KAFKA_KRAFT_CLUSTER_ID: abcdefghijklmnopqrstuv\n  KAFKA_CFG_PROCESS_ROLES: controller,broker\n  KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER\n  KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093\n  EXTRA_ARGS: \"-Xms128m -Xmx256m -javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent-0.19.0.jar=9404:/opt/jmx-exporter/kafka-2_0_0.yml\"\n```\n\nThe exporter will be available at port 9404:\n```shell\ndocker compose exec kafka-0 curl localhost:9404\n```\n\nWe also want Kafka UI to use the metrics. Add this to `kafka-ui/config.yml`:\n```yaml\nkafka:\n  clusters:\n    - bootstrapServers: kafka-0:9092,kafka-1:9092\n      name: kafka\n      metrics:\n        type: JMX\n        port: 9404\n```\n\n\n\n\n\n## Prometheus\nCreate configuration file for a prometheus:\n```shell\nmkdir -p prometheus\ntouch prometheus/prometheus.yml\n```\nPut the following content:\n```yaml\nglobal:\n  scrape_interval: 15s\n  scrape_timeout: 10s\n  evaluation_interval: 15s\nscrape_configs:\n- job_name: kafka-exporter\n  honor_timestamps: true\n  scrape_interval: 15s\n  scrape_timeout: 10s\n  metrics_path: /metrics\n  scheme: http\n  static_configs:\n  - targets:\n    - kafka-exporter:9308\n- job_name: jmx-exporter\n  honor_timestamps: true\n  scrape_interval: 15s\n  scrape_timeout: 10s\n  metrics_path: /metrics\n  scheme: http\n  static_configs:\n  - targets:\n    - kafka-0:9404\n    - kafka-1:9404\n```\nHere we ask prometheus to get metrics from kafka exporter from `/metrics` endpoint on port `9093` and from\njmx-exporter on port `9404`.\n\n\n\n\n## Grafana\n\nFinally, create the configuration for grafana:\n```shell\nmkdir -p grafana/provisioning/datasources\nmkdir -p grafana/provisioning/dashboards\ntouch grafana/provisioning/datasources/datasource.yml\ntouch grafana/provisioning/dashboards/dashboard.yml\ncurl https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/main/examples/metrics/grafana-dashboards/strimzi-kafka-exporter.json -o grafana/dashboards/strimzi-kafka-exporter.json\ncurl https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/main/examples/metrics/grafana-dashboards/strimzi-kafka.json -o grafana/dashboards/strimzi-kafka.json\n```\n\nWe tell the grafana to provision prometheus datasource. Also we download the dashboard for the kafka exporter and\njmx exporter  and tell grafana to use it.\n\nPut the following into `grafana/provisioning/datasources/datasource.yml`:\n\n```yaml\napiVersion: 1\n\ndeleteDatasources:\n  - name: Prometheus\n    orgId: 1\n\ndatasources:\n  - name: Prometheus\n    type: prometheus\n    access: proxy\n    orgId: 1\n    url: http://prometheus:9090\n    password:\n    user:\n    database:\n    basicAuth: false\n    basicAuthUser:\n    basicAuthPassword:\n    withCredentials:\n    isDefault: true\n    version: 1\n    editable: false\n```\n\nand the following into `grafana/provisioning/dashboards/dashboard.yml`:\n```yaml\napiVersion: 1\n\nproviders:\n  - name: 'dashboards-from-file'\n    orgId: 1\n    folder: ''\n    folderUid: ''\n    type: file\n    disableDeletion: false\n    updateIntervalSeconds: 10\n    allowUiUpdates: false\n    options:\n      path: /var/lib/grafana/dashboards\n      foldersFromFilesStructure: true\n```\n\n\nNow restart docker compose and visit http://localhost:3000/dashboards. Use `admin`:`grafana` as credentials.\nOpen dashboard with name `Strimzi Kafka Exporter`, you will see some details about the cluster.\n\n\n\n# Conclusion\nNow we can run grafana cluster, we can validate it by producing and consuming messages from CLI. \nAlso we can view the details of the cluster with UI and watch the cluster metrics in grafana.\n\n\n# Links\n- APACHE KAFKA: https://kafka.apache.org/\n- Bitnami kafka image: https://hub.docker.com/r/bitnami/kafka/\n- Grafana: https://grafana.com/\n- jmx_exporter: https://github.com/prometheus/jmx_exporter\n- Kafka Exporter: https://github.com/danielqsj/kafka_exporter\n- Kafka UI: https://github.com/provectus/kafka-ui\n- Prometheus: https://prometheus.io/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpapirosko%2Fkafka-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpapirosko%2Fkafka-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpapirosko%2Fkafka-demo/lists"}