https://github.com/bsm/rumour
Kafka Consumer Monitoring and Reporting
https://github.com/bsm/rumour
Last synced: about 1 year ago
JSON representation
Kafka Consumer Monitoring and Reporting
- Host: GitHub
- URL: https://github.com/bsm/rumour
- Owner: bsm
- License: apache-2.0
- Created: 2018-12-05T09:44:15.000Z (over 7 years ago)
- Default Branch: main
- Last Pushed: 2022-02-01T08:51:03.000Z (over 4 years ago)
- Last Synced: 2025-04-13T05:54:45.110Z (about 1 year ago)
- Language: Go
- Size: 40 KB
- Stars: 0
- Watchers: 2
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Rumour
[](https://godoc.org/github.com/bsm/rumour)
[](https://travis-ci.org/bsm/rumour)
[](https://goreportcard.com/report/github.com/bsm/rumour)
Rumour is a leaner, meaner and more easily configurable version of LinkedIn's [Burrow](https://github.com/linkedin/Burrow). It is a small binary which can perform continuous monitoring tasks of [Apache Kafka](https://kafka.apache.org/) consumer states, offsets and lags.
## Setup
Download the latest release from [GitHub](https://github.com/bsm/rumour/releases) or run it directly via Docker:
```shell
docker run --rm \
-e RUMOUR_CLUSTERS=default \
-e RUMOUR_DEFAULT_BROKERS=kafka:9092 \
blacksquaremedia/rumour:latest
```
## Configuration
All configuration is done via ENV variables. The main configuration parameters are:
- `RUMOUR_CLUSTERS` - a comma-separated list of cluster names to monitor. Default: `default`
- `RUMOUR_HTTP_ADDR` - the address to listen on. Default: `:8080`.
- `RUMOUR_LOG_LEVEL` - the log level. Default: `info`.
- `RUMOUR_LOG_JSON` - use JSON format. Default: `false`.
- `RUMOUR_LOG_TAGS` - additional logging tags as comma-separated map
`key1:value,key1:value`. Default: _none_.
Additonal configuration can be specified for each of the named clusters using the `RUMOUR_{cluster}_` prefix.
- `RUMOUR_{cluster}_BROKERS` - a comma-separated list of broker addresses.
- `RUMOUR_{cluster}_META_REFRESH` - metadata refresh interval. Default: 180s.
- `RUMOUR_{cluster}_OFFSET_REFRESH` - offset refresh interval. Default: 30s.
Example:
```shell
RUMOUR_CLUSTERS=main,prio \
RUMOUR_MAIN_BROKERS=10.0.0.1:9092,10.0.0.2:9092,10.0.0.3:9092 \
RUMOUR_PRIO_BROKERS=10.0.0.1:9192,10.0.0.2:9192,10.0.0.3:9192 \
RUMOUR_PRIO_META_REFRESH=120s \
./rumour
```
## Integrations
- [datadog](./integrations/datadog/) - a Datadog check to pull metrics out of Rumour and push them to [Datadog](https://www.datadoghq.com/).
## API
Rumour exposes metrics via a HTTP API for data collectors. It is loosely based on [Burrow's](https://github.com/linkedin/Burrow/wiki/HTTP-Endpoint) HTTP endpoints.
### Error Responses
For bad requests, the API will return an appropriate HTTP status code and a JSON body containing:
```json
{
"error": true,
"message": "Full error message"
}
```
### Endpoints
#### Health check:
```
GET /healthz
```
#### List clusters:
```
GET /v1/clusters
```
```json
{
"clusters": ["main", "prio"]
}
```
#### Show cluster details:
```
GET /v1/clusters/NAME
```
```json
{
"cluster": "main",
"brokers": ["10.0.0.1:9092", "10.0.0.2:9092", "10.0.0.3:9092"],
"topics": ["my-topic"],
"consumers": ["consumer-x", "consumer-y"]
}
```
#### Show cluster topics:
```
GET /v1/clusters/NAME/topics
```
```json
{
"cluster": "main",
"topics": ["my-topic"]
}
```
#### Show cluster consumers:
```
GET /v1/clusters/NAME/consumers
```
```json
{
"cluster": "main",
"consumers": ["consumer-x", "consumer-y"]
}
```
#### Show topic:
```
GET /v1/clusters/NAME/topics/TOPIC
```
```json
{
"cluster": "main",
"topic": "my-topic",
"offsets": [1041, 1042, 1043, 1044]
}
```
#### Show consumer:
```
GET /v1/clusters/NAME/consumers/GROUP
```
```json
{
"cluster": "main",
"consumer": "consumer-x",
"topics": [
{
"topic": "my-topic",
"timestamp": 1515151515,
"offsets": [
{ "offset": 1037, "lag": 4 },
{ "offset": 1041, "lag": 1 },
{ "offset": 1029, "lag": 14 },
{ "offset": 1044, "lag": 0 }
]
}
]
}
```