Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/lechindianer/kafkaex_lag_exporter
Check Kafka consumer group lag with Kafka lag exporter
https://github.com/lechindianer/kafkaex_lag_exporter
docker elixir kafka prometheus
Last synced: 4 days ago
JSON representation
Check Kafka consumer group lag with Kafka lag exporter
- Host: GitHub
- URL: https://github.com/lechindianer/kafkaex_lag_exporter
- Owner: Lechindianer
- License: isc
- Created: 2024-04-14T19:55:54.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2024-06-01T10:11:54.000Z (6 months ago)
- Last Synced: 2024-06-01T11:32:14.974Z (6 months ago)
- Topics: docker, elixir, kafka, prometheus
- Language: Elixir
- Homepage:
- Size: 84 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# KafkaExLagExporter
This project will collect Kafka consumer lag and provide them via Prometheus.
## Metrics
[Prometheus](https://prometheus.io/) is a standard way to represent metrics in a modern cross-platform manner.
KafkaExLagExporter exposes several metrics as an HTTP endpoint that can be readily scraped by Prometheus.
**`kafka_consumergroup_group_topic_sum_lag`**
Labels: `cluster_name, group, topic, consumer_id, member_host`
The sum of the difference between the last produced offset and the last consumed offset of all partitions in this
topic for this group.
**`kafka_consumergroup_group_lag`**
Labels: `cluster_name, group, partition, topic, member_host, consumer_id`
The difference between the last produced offset and the last consumed offset for this partition in this topic
partition for this group.
## Start
```bash
docker run -ti --net="host" -e KAFKA_BROKERS=redpanda:29092 -p 4000:4000 lechindianer/kafkaex_lag_exporter:0.2.0
```
Now you can check the exposed metrics at [http://localhost:4000](http://localhost:4000).
## Configuration
KafkaExLagExporter uses 5 seconds as default interval to update the lags. If you want to configure it to use another
value set `KAFKA_EX_INTERVAL_MS`, i.e.
```bash
docker run -ti --net="host" -e KAFKA_BROKERS=redpanda:29092 -e KAFKA_EX_INTERVAL_MS=10000 -p 4000:4000 \
lechindianer/kafkaex_lag_exporter:0.2.0
```
## Developing
To start the project locally:
```bash
KAFKA_BROKERS="localhost:9092" iex -S mix
```
There is also a Docker compose file included which will start Kafka, serve Kowl (Web UI for Kafka) and start
KafkaexLagExporter:
```bash
docker compose up --build
```
Kowl is served at [http://localhost:8080](http://localhost:8080).
### Tests
```bash
MIX_ENV=test mix test --no-start
```
### Code style
Don't forget to check [credo](https://hexdocs.pm/credo/overview.html) for code violations:
```bash
mix credo
```
This project also leverages the use of typespecs in order to provide static code checking:
```bash
mix dialyzer
```
## Links
Source is on [Gitlab](https://gitlab.com/lechindianer/kafkaex-lag-exporter).
The initial project [Kafka Lag Exporter](https://github.com/seglo/kafka-lag-exporter) was a huge inspiration for me
creating my first real Elixir project. Thank you!