https://github.com/riferrei/kafka-client-metrics-to-cloudwatch-with-kip-714

Are you tired of collecting Apache Kafka client metrics using JMX? Worse, you even can't do that? Here is a solution. With the KIP-714, you can collect client metrics from the Kafka broker using OpenTelemetry.
https://github.com/riferrei/kafka-client-metrics-to-cloudwatch-with-kip-714

amazon-cloudwatch apache-kafka apache-kafka-consumer apache-kafka-producer kip-714 metrics opentelemetry

Last synced: 4 months ago
JSON representation

Are you tired of collecting Apache Kafka client metrics using JMX? Worse, you even can't do that? Here is a solution. With the KIP-714, you can collect client metrics from the Kafka broker using OpenTelemetry.

• Host: GitHub
• URL: https://github.com/riferrei/kafka-client-metrics-to-cloudwatch-with-kip-714
• Owner: riferrei
• License: apache-2.0
• Created: 2024-06-20T21:20:56.000Z (8 months ago)
• Default Branch: main
• Last Pushed: 2024-08-07T13:56:01.000Z (6 months ago)
• Last Synced: 2024-09-27T06:43:02.216Z (4 months ago)
• Topics: amazon-cloudwatch, apache-kafka, apache-kafka-consumer, apache-kafka-producer, kip-714, metrics, opentelemetry
• Language: Java
• Homepage:
• Size: 6.4 MB
• Stars: 5
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Kafka Client Metrics to Amazon CloudWatch with KIP-714

This project contains an example that shows how to push metrics from your Apache Kafka clients to [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) using the [KIP-714: Client Metrics and Observability](https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability). To use this feature, you must use a Kafka cluster with the version 3.7.0 or higher. It also requires the Kraft mode enabled, which is the new mode to run Kafka brokers without requiring Zookeeper.

## Getting started

Everything in this project is configured for you just experience the KIP-714 with little hassle. If you want to understand what is going on behind the scenes, the blog post below explain everything in great details.

[![KIP-714: Keep your Kafka Clusters Close, and your Kafka Clients Closer](/images/blog-post.png 'KIP-714: Keep your Kafka Clusters Close, and your Kafka Clients Closer')](https://community.aws/content/2i3i0osdRihttgQ8AZii6F0Bv4F/kip-714-keep-your-kafka-clusters-close-and-your-kafka-clients-closer)

To get started, just execute the Docker Compose file. It will execute a Kafka broker and it will perform the required setup on it. It will also execute an OpenTelemetry collector. The OpenTelemetry collector will use your stored AWS credentials to connect with Amazon CloudWatch. If you have not configured your credentials using the AWS CLI, do it before proceeding.

```bash

docker compose up -d

```

💡 For the next steps, you must have a distribution of Apache Kafka installed in your machine.

Load a few records into the `load-test` topic. The command below loads `50K` records to trigger the producer metrics. Each record with a payload size of 1KB, sending 1K records every second.

```bash

kafka-producer-perf-test.sh --producer-props bootstrap.servers=localhost:9092 --throughput 1000 --num-records 50000 --record-size 1024 --topic load-test --print-metrics

```

You must also consume these records to trigger the consumer metrics. The command below loads the `50K` records from the `load-test` topic.

```bash

kafka-consumer-perf-test.sh --bootstrap-server localhost:9092 --messages 50000 --topic load-test --print-metrics

```

💡 Depending of the version of Apache Kafka you're using, you may see some warnings after executing this command. You can safely ignore them.

## Viewing the metrics at Amazon CloudWatch

Go to the AWS console and access Amazon CloudWatch. You should see a new namespace called `kafka-kip-714` with a load of metrics for you to play with.

![Multiple metric groups](/images/cloudwatch.png)

If you want to visualize the load from the `load-test` topic using gauges, use the graph source below on Amazon CloudWatch:

```json

{

    "metrics": [

        [ "kafka-kip-714", "org.apache.kafka.producer.topic.record.send.total", "topic", "load-test" ],

        [ "kafka-kip-714", "org.apache.kafka.consumer.fetch.manager.records.consumed.total", "topic", "load-test" ]

    ],

    "view": "gauge",

    "stacked": false,

    "region": "us-east-1",

    "yAxis": {

        "left": {

            "min": 0,

            "max": 50000

        }

    },

    "stat": "Sum",

    "period": 300,

    "liveData": true,

    "setPeriodToTimeRange": false,

    "sparkline": true,

    "trend": true

}

```

You should see the following result:

![Sample Kafka metrics](/images/client-metrics.png)

# License

This project is licensed under the [Apache 2.0 License](./LICENSE).