https://github.com/kairosdb/kafka-monitor

Service that reads and reports Kafka topic lag
https://github.com/kairosdb/kafka-monitor

kafka metrics

Last synced: about 1 month ago
JSON representation

Service that reads and reports Kafka topic lag

• Host: GitHub
• URL: https://github.com/kairosdb/kafka-monitor
• Owner: kairosdb
• Created: 2018-02-28T03:31:07.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2024-07-03T23:15:58.000Z (6 months ago)
• Last Synced: 2024-07-09T18:55:49.642Z (6 months ago)
• Topics: kafka, metrics
• Language: Java
• Homepage:
• Size: 15.7 MB
• Stars: 6
• Watchers: 4
• Forks: 4
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# kafka-topic-monitor

A stand alone service that reads kafka topic offsets and reports metrics using 

metrics4j to your favorite time-series database (see https://github.com/kairosdb/metrics4j for details)

You can run multiple monitors against a single Kafka cluster for redundancy and fail over.  The Kafka Topic Montior

will divide the work between each instance by using internal kafka topics.  This provides a reliable platform

for alerts to detect when consumers are having problems.

### Metrics that this service will report

For metric names and configuring the destination please see the metrics4j.conf file

  * Topic producer count - count of events for each topic being produced.

  * Topic/group consumer count - count of events consumed for each group/topic tuple.

  * Group/topic/partition lag - difference between the offset and the head of the topic for each group/topic/patition tuple.

  * Group/topic lag summary - same as above but a summary across all partitions (faster dashboard loading when you have a lot of partitions).

  * Group time to process - A rough calculation based on the consumer rate for the group and the amount of lag.

  * Offset age - time in milliseconds since the offset has been updated (data used to alert when a consumer has stopped functioning)

  * Stale offset age - Similar to offset age but is only reported when the partition also has a lag.

  

## Testing

This service uses Jooby as the service framework so the easiest way to run it is 

the following command line

>mvn jooby:run

## Deployment

The project is packaged with stork.  Calling `mvn package` will create a complete

install directory under target/stork

## Configuration

All configuration is done in application.conf see the comments in the file for details