Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.
Awesome Lists | Featured Topics | Projects
https://github.com/christiangalsterer/kafkajs-prometheus-exporter

A prometheus exporter exposing metrics for KafkaJS
https://github.com/christiangalsterer/kafkajs-prometheus-exporter
grafana grafana-dashboard kafka kafkajs metrics monitoring node-js nodejs prometheus prometheus-exporter sre
Last synced: 2 months ago
JSON representation
A prometheus exporter exposing metrics for KafkaJS
Host: GitHub
URL: https://github.com/christiangalsterer/kafkajs-prometheus-exporter
Owner: christiangalsterer
License: mit
Created: 2023-08-11T19:07:57.000Z (over 1 year ago)
Default Branch: main
Last Pushed: 2024-04-17T01:34:56.000Z (8 months ago)
Last Synced: 2024-04-17T05:06:55.644Z (8 months ago)
Topics: grafana, grafana-dashboard, kafka, kafkajs, metrics, monitoring, node-js, nodejs, prometheus, prometheus-exporter, sre
Language: TypeScript
Homepage:
Size: 1.85 MB
Stars: 2
Watchers: 2
Forks: 2
Open Issues: 1
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project

README

        [![GitHub Actions CI Status](https://github.com/christiangalsterer/kafkajs-prometheus-exporter/actions/workflows/build.yaml/badge.svg)](https://github.com/christiangalsterer/kafkajs-prometheus-exporter/actions/workflows/build.yaml)

[![codecov](https://codecov.io/gh/christiangalsterer/kafkajs-prometheus-exporter/graph/badge.svg?token=Y91U2SLNHM)](https://codecov.io/gh/christiangalsterer/kafkajs-prometheus-exporter)

[![Coverage Status](https://coveralls.io/repos/github/christiangalsterer/kafkajs-prometheus-exporter/badge.svg?branch=main)](https://coveralls.io/github/christiangalsterer/kafkajs-prometheus-exporter?branch=main)

[![Known Vulnerabilities](https://snyk.io/test/github/christiangalsterer/kafkajs-prometheus-exporter/badge.svg)](https://github.com/christiangalsterer/kafkajs-prometheus-exporter/security/advisories)

[![npm downloads](https://img.shields.io/npm/dt/@christiangalsterer/kafkajs-prometheus-exporter.svg)](https://www.npmjs.com/package/@christiangalsterer/kafkajs-prometheus-exporter)

[![npm version](https://img.shields.io/npm/v/@christiangalsterer/kafkajs-prometheus-exporter.svg)](https://www.npmjs.com/package/@christiangalsterer/kafkajs-prometheus-exporter?activeTab=versions)

[![npm license](https://img.shields.io/npm/l/@christiangalsterer/kafkajs-prometheus-exporter.svg)](https://www.npmjs.com/package/@christiangalsterer/kafkajs-prometheus-exporter)

[![semver](https://img.shields.io/badge/semver-2.0.0-green)](https://semver.org)

[![Conventional Commits](https://img.shields.io/badge/Conventional%20Commits-1.0.0-yellow.svg)](https://conventionalcommits.org)

[![renovate](https://img.shields.io/badge/renovate-enabled-brightgreen.svg)](https://developer.mend.io/github/christiangalsterer/kafkajs-prometheus-exporter)

[![code style: prettier](https://img.shields.io/badge/code_style-prettier-ff69b4.svg?style=flat-square)](https://github.com/prettier/prettier)

![github stars](https://img.shields.io/github/stars/christiangalsterer/kafkajs-prometheus-exporter.svg)

# Prometheus Exporter for KafkaJS

A prometheus exporter exposing metrics for [KafkaJS](https://kafka.js.org/).

Metrics names follow the same naming convention used by [Kafka JMX MBeans](https://kafka.apache.org/documentation/#monitoring) and [micrometer](https://github.com/micrometer-metrics/micrometer). This allows to use the same metrics in dashboards and alerts across different technology stacks, e.g. when you use Spring Boot and Node.js in different applications.

## Available Metrics

The exporter provides the following metrics.

### Producer Metrics

|Metric Name|Description|Labels|Since|

|---|---|---|---|

|kafka_producer_connection_count|The current number of active connections established with a broker||0.8.0|

|kafka_producer_connection_creation_total|The total number of connections established with a broker||0.8.0|

|kafka_producer_connection_close_total|The total number of connections closed with a broker||0.8.0|

|kafka_producer_request_duration_seconds_bucket|The time taken for processing a producer request.|


_broker_: The broker

|3.1.0|

|kafka_producer_request_duration_seconds_sum|The cumulated time for processing producer requests.|

_broker_: The broker

|3.1.0|

|kafka_producer_request_duration_seconds_count|The total number of producer requests.|

_broker_: The broker

|3.1.0|

|kafka_producer_request_total|The total number of requests sent.|

_client_id_: An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name. The client_id is shared across multiple instances of the same application.

_broker_: The broker

|0.8.0|

|kafka_producer_request_size_total|The size of any request sent.|

_client_id_: An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name. The client_id is shared across multiple instances of the same application.

_broker_: The broker

|0.9.0|

|kafka_producer_request_queue_size|Size of the request queue.|

_client_id_: An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name. The client_id is shared across multiple instances of the same application.

_broker_: The broker

|0.8.0|

### Consumer Metrics

|Metric Name|Description|Labels|Since|

|---|---|---|---|

|kafka_consumer_connection_count|The current number of active connections established with a broker||0.8.0|

|kafka_consumer_connection_creation_total|The total number of connections established with a broker||0.8.0|

|kafka_consumer_connection_close_total|The total number of connections closed with a broker||0.8.0|

|kafka_consumer_connection_crashed_total|The total number of crashed connections with a broker|


_group_id_: The id of the consumer group.

_error_: The error which caused the connection crash.

_restart_: Determines if the connection was automatically restarted.

|0.8.0|

|kafka_consumer_heartbeat_total|The total number of heartbeats with a broker|

_group_id_: The id of the consumer group.

_member_id_: The member of the consumer group.

|0.8.0|

|kafka_consumer_request_duration_seconds_bucket|The time taken for processing a consumer request.|

_broker_: The broker

|3.1.0|

|kafka_consumer_request_duration_seconds_sum|The cumulated time for processing consumer requests.|

_broker_: The broker

|3.1.0|

|kafka_consumer_request_duration_seconds_count|The total number of consumer requests.|

_broker_: The broker

|3.1.0|

|kafka_consumer_request_total|The total number of requests sent.|

_broker_: The broker

|0.8.0|

|kafka_consumer_request_queue_size|Size of the request queue.|

_broker_: The broker

|0.8.0|

|kafka_consumer_request_size_total|The size of any request sent.|

_broker_: The broker

|0.9.0|

|kafka_consumer_fetch_duration_seconds_bucket|The time taken for processing a fetch request.||3.1.0|

|kafka_consumer_fetch_duration_seconds_sum|The cumulated time for processing fetch requests.||3.1.0|

|kafka_consumer_fetch_duration_seconds_count|The total number of fetch requests.||3.1.0|

|**Deprecated** kafka_consumer_fetch_latency|The time taken for processing a fetch request.||0.8.0|

|kafka_consumer_fetch_total|The total number of fetch requests.||0.8.0|

|kafka_consumer_batch_size_total|The number of bytes received per partition per request|

_topic_: The topic from which the messages are consumed.

_partition_: The partition of the topic from which the messages are consumed.

|0.9.0|

|kafka_consumer_batch_duration_seconds_bucket|The time taken for processing a batch.|

_topic_: The topic from which the messages are consumed.

_partition_: The partition of the topic from which the messages are consumed.

|3.1.0|

|kafka_consumer_batch_duration_seconds_sum|The cumulated time for processing batches.|

_topic_: The topic from which the messages are consumed.

_partition_: The partition of the topic from which the messages are consumed.

|3.1.0|

|kafka_consumer_batch_duration_seconds_count|The total number of batches.|

_topic_: The topic from which the messages are consumed.

_partition_: The partition of the topic from which the messages are consumed.

|3.1.0|

|**Deprecated**  kafka_consumer_batch_latency|The time taken for processing a batch.|

_topic_: The topic from which the messages are consumed.

_partition_: The partition of the topic from which the messages are consumed.

|0.8.0|

### Admin Metrics

|Metric Name|Description|Labels|Since|

|---|---|---|---|

|kafka_admin_connection_count|The current number of active connections established with a broker||1.0.0|

|kafka_admin_connection_creation_total|The total number of connections established with a broker||1.0.0|

|kafka_admin_connection_close_total|The total number of connections closed with a broker||1.0.0|

|kafka_admin_request_duration_seconds_bucket|The time taken for processing an admin request.|


_broker_: The broker

|3.1.0|

|kafka_admin_request_duration_seconds_sum|The cumulated time for processing admin requests.|

_broker_: The broker

|3.1.0|

|kafka_admin_request_duration_seconds_count|The total number of admin requests.|

_broker_: The broker

|3.1.0|

|kafka_admin_request_total|The total number of requests sent.|

_broker_: The broker

|1.0.0|

|kafka_admin_request_size_total|The size of any request sent.|

_broker_: The broker

|1.0.0|

|kafka_admin_request_queue_size|Size of the request queue.|

_broker_: The broker

|1.0.0|

## Client Id

As documented in the [KafkaJS documentation](https://kafka.js.org/docs/configuration#client-id) the client-id is a logical identifier for an application which is shared across multiple instances of the same application. Until v2.0.0 of the exporter the client_id was a dedicated parameter for _monitorKafkaJSProducer_, _monitorKafkaJSConsumer_ and _monitorKafkaJSAdmin_ respectively, but was removed starting with v2.0.0, see [changelog](/CHANGELOG.md) for details.

Until KafkaJS instrumentation events]() don't provide the client_id for all events it is **strongly** recommended to add the client_id as a default label as in the following example.

```ts

    kafkaExporter.monitorKafkaJSProducer(producer, register, { defaultLabels: {client_id:'nodejs-example-app'} })

```

## Example Output

Here an example output in the prometheus format of the provided metrics.

```sh

# HELP kafka_consumer_connection_count The current number of active connections established with a broker

# TYPE kafka_consumer_connection_count gauge

kafka_consumer_connection_count{client_id="nodejs-example-app",alice="bob"} 2

# HELP kafka_consumer_connection_creation_total The total number of connections established with a broker

# TYPE kafka_consumer_connection_creation_total counter

kafka_consumer_connection_creation_total{client_id="nodejs-example-app",alice="bob"} 2

# HELP kafka_consumer_connection_close_total The total number of connections closed with a broker

# TYPE kafka_consumer_connection_close_total counter

# HELP kafka_consumer_connection_crashed_total The total number of crashed connections with a broker

# TYPE kafka_consumer_connection_crashed_total counter

# HELP kafka_consumer_heartbeat_total The total number of heartbeats with a broker

# TYPE kafka_consumer_heartbeat_total counter

kafka_consumer_heartbeat_total{client_id="nodejs-example-app",group_id="myGroupId",member_id="nodejs-example-app-e8962841-acad-4c88-a076-f4f4fa9f27bd",alice="bob"} 7

# HELP kafka_consumer_request_total The total number of requests sent.

# TYPE kafka_consumer_request_total counter

kafka_consumer_request_total{client_id="nodejs-example-app",broker="broker1:9094",alice="bob"} 37

kafka_consumer_request_total{client_id="nodejs-example-app",broker="broker2:9094",alice="bob"} 186

kafka_consumer_request_total{client_id="nodejs-example-app",broker="broker3:9094",alice="bob"} 34

# HELP kafka_consumer_request_size_total The size of any request sent.

# TYPE kafka_consumer_request_size_total counter

kafka_consumer_request_size_total{client_id="nodejs-example-app",broker="broker1:9094",alice="bob"} 458

kafka_consumer_request_size_total{client_id="nodejs-example-app",broker="broker2:9094",alice="bob"} 40

kafka_consumer_request_size_total{client_id="nodejs-example-app",broker="broker3:9094",alice="bob"} 458

# HELP kafka_consumer_request_queue_size Size of the request queue.

# TYPE kafka_consumer_request_queue_size gauge

# HELP kafka_consumer_fetch_duration_seconds The time taken for a fetch request.

# TYPE kafka_consumer_fetch_duration_seconds histogram

kafka_consumer_fetch_duration_seconds_bucket{le="0.001",client_id="nodejs-example-app",alice="bob"} 0

kafka_consumer_fetch_duration_seconds_bucket{le="0.005",client_id="nodejs-example-app",alice="bob"} 0

kafka_consumer_fetch_duration_seconds_bucket{le="0.01",client_id="nodejs-example-app",alice="bob"} 0

kafka_consumer_fetch_duration_seconds_bucket{le="0.02",client_id="nodejs-example-app",alice="bob"} 0

kafka_consumer_fetch_duration_seconds_bucket{le="0.03",client_id="nodejs-example-app",alice="bob"} 0

kafka_consumer_fetch_duration_seconds_bucket{le="0.04",client_id="nodejs-example-app",alice="bob"} 0

kafka_consumer_fetch_duration_seconds_bucket{le="0.05",client_id="nodejs-example-app",alice="bob"} 24

kafka_consumer_fetch_duration_seconds_bucket{le="0.1",client_id="nodejs-example-app",alice="bob"} 79

kafka_consumer_fetch_duration_seconds_bucket{le="0.2",client_id="nodejs-example-app",alice="bob"} 82

kafka_consumer_fetch_duration_seconds_bucket{le="0.5",client_id="nodejs-example-app",alice="bob"} 84

kafka_consumer_fetch_duration_seconds_bucket{le="1",client_id="nodejs-example-app",alice="bob"} 86

kafka_consumer_fetch_duration_seconds_bucket{le="2",client_id="nodejs-example-app",alice="bob"} 86

kafka_consumer_fetch_duration_seconds_bucket{le="5",client_id="nodejs-example-app",alice="bob"} 86

kafka_consumer_fetch_duration_seconds_bucket{le="10",client_id="nodejs-example-app",alice="bob"} 86

kafka_consumer_fetch_duration_seconds_bucket{le="+Inf",client_id="nodejs-example-app",alice="bob"} 86

kafka_consumer_fetch_duration_seconds_sum{client_id="nodejs-example-app",alice="bob"} 6.728

kafka_consumer_fetch_duration_seconds_count{client_id="nodejs-example-app",alice="bob"} 86

# HELP kafka_consumer_fetch_total The total number of fetch requests.

# TYPE kafka_consumer_fetch_total counter

kafka_consumer_fetch_total{client_id="nodejs-example-app",alice="bob"} 86

# HELP kafka_consumer_batch_size_total The number of bytes received per partition per request

# TYPE kafka_consumer_batch_size_total counter

kafka_consumer_batch_size_total{client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 1

kafka_consumer_batch_size_total{client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 1

kafka_consumer_batch_size_total{client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 1

kafka_consumer_batch_size_total{client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 1

kafka_consumer_batch_size_total{client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 1

# HELP kafka_consumer_batch_duration_seconds The time taken for processing a batch.

# TYPE kafka_consumer_batch_duration_seconds histogram

kafka_consumer_batch_duration_seconds_bucket{le="0.001",client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 21

kafka_consumer_batch_duration_seconds_bucket{le="0.005",client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="0.01",client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="0.02",client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="0.03",client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="0.04",client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="0.05",client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="0.1",client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.2",client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.5",client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="1",client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="2",client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="5",client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="10",client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="+Inf",client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 28

kafka_consumer_batch_duration_seconds_sum{client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 0.12200000000000003

kafka_consumer_batch_duration_seconds_count{client_id="nodejs-example-app",topic="topic1",partition="2",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.001",client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 26

kafka_consumer_batch_duration_seconds_bucket{le="0.005",client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="0.01",client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.02",client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.03",client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.04",client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.05",client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.1",client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.2",client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.5",client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="1",client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="2",client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="5",client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="10",client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="+Inf",client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 28

kafka_consumer_batch_duration_seconds_sum{client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 0.024000000000000014

kafka_consumer_batch_duration_seconds_count{client_id="nodejs-example-app",topic="topic1",partition="1",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.001",client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 25

kafka_consumer_batch_duration_seconds_bucket{le="0.005",client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.01",client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.02",client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.03",client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.04",client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.05",client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.1",client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.2",client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.5",client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="1",client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="2",client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="5",client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="10",client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="+Inf",client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 28

kafka_consumer_batch_duration_seconds_sum{client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 0.025000000000000012

kafka_consumer_batch_duration_seconds_count{client_id="nodejs-example-app",topic="topic1",partition="4",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.001",client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 21

kafka_consumer_batch_duration_seconds_bucket{le="0.005",client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="0.01",client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="0.02",client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="0.03",client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="0.04",client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="0.05",client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="0.1",client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="0.2",client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="0.5",client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="1",client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="2",client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="5",client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="10",client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="+Inf",client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 27

kafka_consumer_batch_duration_seconds_sum{client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 0.039000000000000014

kafka_consumer_batch_duration_seconds_count{client_id="nodejs-example-app",topic="topic1",partition="3",alice="bob"} 27

kafka_consumer_batch_duration_seconds_bucket{le="0.001",client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 26

kafka_consumer_batch_duration_seconds_bucket{le="0.005",client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.01",client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.02",client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.03",client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.04",client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.05",client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.1",client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.2",client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="0.5",client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="1",client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="2",client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="5",client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="10",client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 28

kafka_consumer_batch_duration_seconds_bucket{le="+Inf",client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 28

kafka_consumer_batch_duration_seconds_sum{client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 0.023000000000000013

kafka_consumer_batch_duration_seconds_count{client_id="nodejs-example-app",topic="topic1",partition="0",alice="bob"} 28

# HELP kafka_producer_connection_count The current number of active connections established with a broker

# TYPE kafka_producer_connection_count gauge

kafka_producer_connection_count{client_id="nodejs-example-app",foo="bar"} 342

# HELP kafka_producer_connection_creation_total The total number of connections established with a broker

# TYPE kafka_producer_connection_creation_total counter

kafka_producer_connection_creation_total{client_id="nodejs-example-app",foo="bar"} 342

# HELP kafka_producer_connection_close_total The total number of connections closed with a broker

# TYPE kafka_producer_connection_close_total counter

# HELP kafka_producer_request_total The total number of requests sent.

# TYPE kafka_producer_request_total counter

kafka_producer_request_total{client_id="nodejs-example-app",broker="broker2:9094",foo="bar"} 72

kafka_producer_request_total{client_id="nodejs-example-app",broker="broker3:9094",foo="bar"} 139

kafka_producer_request_total{client_id="nodejs-example-app",broker="broker1:9094",foo="bar"} 139

# HELP kafka_producer_request_size_total The size of any request sent.

# TYPE kafka_producer_request_size_total counter

kafka_producer_request_size_total{client_id="nodejs-example-app",broker="broker2:9094",foo="bar"} 649

kafka_producer_request_size_total{client_id="nodejs-example-app",broker="broker3:9094",foo="bar"} 457

kafka_producer_request_size_total{client_id="nodejs-example-app",broker="broker1:9094",foo="bar"} 585

# HELP kafka_producer_request_queue_size Size of the request queue.

# TYPE kafka_producer_request_queue_size gauge

# HELP kafka_admin_connection_count The current number of active connections established with a broker

# TYPE kafka_admin_connection_count gauge

kafka_admin_connection_count{client_id="nodejs-example-app",foo="bar"} 0

# HELP kafka_admin_connection_creation_total The total number of connections established with a broker

# TYPE kafka_admin_connection_creation_total counter

kafka_admin_connection_creation_total{client_id="nodejs-example-app",foo="bar"} 11

# HELP kafka_admin_connection_close_total The total number of connections closed with a broker

# TYPE kafka_admin_connection_close_total counter

kafka_admin_connection_close_total{client_id="nodejs-example-app",foo="bar"} 11

# HELP kafka_admin_request_total The total number of requests sent.

# TYPE kafka_admin_request_total counter

kafka_admin_request_total{client_id="nodejs-example-app",broker="broker2:9094",foo="bar"} 32

kafka_admin_request_total{client_id="nodejs-example-app",broker="broker3:9094",foo="bar"} 4

kafka_admin_request_total{client_id="nodejs-example-app",broker="broker1:9094",foo="bar"} 8

# HELP kafka_admin_request_size_total The size of any request sent.

# TYPE kafka_admin_request_size_total counter

kafka_admin_request_size_total{client_id="nodejs-example-app",broker="broker2:9094",foo="bar"} 5499

kafka_admin_request_size_total{client_id="nodejs-example-app",broker="broker3:9094",foo="bar"} 687

kafka_admin_request_size_total{client_id="nodejs-example-app",broker="broker1:9094",foo="bar"} 1376

# HELP kafka_admin_request_queue_size Size of the request queue.

# TYPE kafka_admin_request_queue_size gauge

```

# Usage

## Add Dependency

Add the following dependency to your project to download the package from [npm](https://www.npmjs.com/package/@christiangalsterer/kafkajs-prometheus-exporter).

```sh

npm i @christiangalsterer/kafkajs-prometheus-exporter

```

## TypeScript

The following example illustrates how to use the exporter to enable monitoring for the KafkaJS.

```ts

import { Kafka } from "kafkajs";

import { Registry, collectDefaultMetrics } from "prom-client";

import { monitorKafkaJSProducer, monitorKafkaJSConsumer, monitorKafkaJSAdmin } from "@christiangalsterer/kafkajs-prometheus-exporter";

...

// set up the KafkaJS client

const clientId = 'myClientId'

const kafka = new Kafka({

    clientId: clientId,

    brokers: ['localhost:9094'],

})

const producer = kafka.producer()

const consumer = kafka.consumer({ groupId: 'myGroupId' })

const admin = kafka.admin()

// set up the prometheus client

const register = new Registry();

collectDefaultMetrics({ register })

// monitor KafkaJS producer

monitorKafkaJSProducer(producer, register, { defaultLabels: {client_id: clientId} })

// monitor KafkaJS consumer

monitorKafkaJSConsumer(consumer, register, { defaultLabels: {client_id: clientId} })

// monitor KafkaJS admin

kafkaExporter.monitorKafkaJSAdmin(admin, register, { defaultLabels: {client_id: clientId} })

...

// connect to Kafka *after* calling monitorKafkaJSProducer() / monitorKafkaJSConsumer / monitorKafkaJSAdmin

await producer.connect()

await consumer.connect()

await admin.connect()

```

## JavaScript

The following example illustrates how to use the exporter to enable monitoring for KafkaJS.

```js

const Kafka = require('kafkajs')

const promClient = require( 'prom-client');

const kafkaExporter = require('@christiangalsterer/kafkajs-prometheus-exporter')

...

// set up the KafkaJS client

const clientId = 'myClientId'

const kafka = new Kafka({

    clientId: clientId,

    brokers: ['localhost:9094'],

})

const producer = kafka.producer()

const consumer = kafka.consumer({ groupId: 'myGroupId' })

const admin = kafka.admin()

// set up the prometheus client

const collectDefaultMetrics = promClient.collectDefaultMetrics;

const Registry = promClient.Registry;

const register = new Registry();

collectDefaultMetrics({ register });

// monitor KafkaJS producer

kafkaExporter.monitorKafkaJSProducer(producer, register, { defaultLabels: {client_id: clientId} })

// monitor KafkaJS consumer

kafkaExporter.monitorKafkaJSConsumer(consumer, register, { defaultLabels: {client_id: clientId} })

// monitor KafkaJS admin

kafkaExporter.monitorKafkaJSAdmin(admin, register, { defaultLabels: {client_id: clientId} })

...

// connect to Kafka *after* calling monitorKafkaJSProducer() / monitorKafkaJSConsumer / monitorKafkaJSAdmin

await producer.connect()

await consumer.connect()

await admin.connect()

```

# Configuration

The exporter can be configured via properties specified on the optional parameter of type

_KafkaJSProducerExporterOptions_, _KafkaJSConsumerExporterOptions_ and _KafkaJSAdminExporterOptions_ respectively.

## KafkaJSProducerExporterOptions

|property|Description|Example|Since |

|---|---|---|---|

| producerRequestDurationHistogramBuckets | Buckets for the kafka_producer_request_duration_seconds metric in seconds. Default buckets are [0.001, 0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10] | [0.001, 0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10]| 3.1.0|

| defaultLabels | Default labels added to each metrics. | {'foo':'bar', 'alice': 3} | 0.8.0 |

## KafkaJSConsumerExporterOptions

|property|Description|Example|Since |

|---|---|---|---|

| consumerRequestDurationHistogramBuckets | Buckets for the kafka_consumer_request_duration_seconds metric in seconds. Default buckets are [0.001, 0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10] | [0.001, 0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10]| 3.1.0|

| consumerBatchDurationHistogramBuckets | Buckets for the kafka_consumer_batch_duration_seconds metric in seconds. Default buckets are [0.001, 0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10] | [0.001, 0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10]| 0.8.0|

| **Deprecated** consumerBatchLatencyHistogramBuckets | Buckets for the kafka_consumer_batch_latency metric in seconds. Default buckets are [0.001, 0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10] | [0.001, 0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10]| 0.8.0|

| consumerFetchDurationHistogramBuckets | Buckets for the kafka_consumer_fetch_duration_seconds metric in seconds. Default buckets are [0.001, 0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10] | [0.001, 0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10]| 0.8.0|

| **Deprecated** consumerFetchLatencyHistogramBuckets | Buckets for the kafka_consumer_fetch_latency metric in seconds. Default buckets are [0.001, 0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10] | [0.001, 0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10]| 0.8.0|

| defaultLabels | Default labels added to each metrics. | {'foo':'bar', 'alice': 3} | 0.8.0|

## KafkaJSAdminExporterOptions

|property|Description|Example|Since |

|---|---|---|---|

| adminRequestDurationHistogramBuckets | Buckets for the kafka_admin_request_duration_seconds metric in seconds. Default buckets are [0.001, 0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10] | [0.001, 0.005, 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10]| 3.1.0|

| defaultLabels | Default labels added to each metrics. | {'foo':'bar', 'alice': 3} | 1.0.0 |

# Grafana Dashboard

An example dashboard for Grafana is available [here](/docs/grafana/dashboard.json) displaying the provided metrics by the exporter.

Here an example for KafkaJS producer metrics:

![Grafana:KafkaJS Producer Metrics](/docs/images/grafana_kafkajs_producer_1.png "Grafana: KafkaJS Producer Metrics")

Here an example for KafkaJS consumer metrics:

![Grafana:KafkaJS Consumer Metrics](/docs/images/grafana_kafkajs_consumer_1.png "Grafana: KafkaJS Consumers Metrics")

Here an example for KafkaJS admin metrics:

![Grafana:KafkaJS Admin Metrics](/docs/images/grafana_kafkajs_admin_1.png "Grafana: KafkaJS Admin Metrics")

# Changelog

The changes to project can be found in the [changelog](/CHANGELOG.md).

# Compatibility

The following table list the compatibility of exporter versions with different KafkaJS and prom-client versions.

|Exporter Version|KafkaJS Version|prom-client version|

|---|---|---|

|^1.0.0|^2.2.4|^14.2.0|

|^2.0.0|^2.2.4|^14.2.0|

|^3.0.0|^2.2.4|^15.0.0|

|^3.1.0|^2.2.4|^15.0.0|

# Contributions

Contributions are highly welcome. If you want to contribute to this project please follow the steps described in the [contribution guidelines](/CONTRIBUTING.md).

# Projects Using The Exporter

If you want to support this project, please add a link to your project and/or company when you use this exporter.

# Related Projects

If you are looking for a way to monitor your MongoDB Driver for Node.js you may have a look at .

If you are looking for a way to monitor node-postgres you may have a look at .