Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/msfidelis/kafka-stress

CLI Tool to Stress Apache Kafka Clusters
https://github.com/msfidelis/kafka-stress

Last synced: 12 days ago
JSON representation

CLI Tool to Stress Apache Kafka Clusters

Host: GitHub
URL: https://github.com/msfidelis/kafka-stress
Owner: msfidelis
License: mit
Created: 2021-06-22T00:47:18.000Z (over 3 years ago)
Default Branch: main
Last Pushed: 2024-06-07T13:24:08.000Z (5 months ago)
Last Synced: 2024-10-16T20:06:17.132Z (27 days ago)
Language: Go
Size: 267 KB
Stars: 39
Watchers: 3
Forks: 4
Open Issues: 1
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

# Kafka Stress - Stress Test Tool for Kafka Clusters, Producers and Consumers Tunning

# Introduction

> Kafka Stress is a simple CLI Tool to produce and consume events on Kafka topics in high throughput to tests and validate brokers, topics, partitions and consumers performances.

# Installation

### Docker

```bash
docker pull fidelissauro/kafka-stress:latest
```

```bash
docker run --network host -it fidelissauro/kafka-stress:latest --bootstrap-servers 0.0.0.0:9092 --topic kafka-stress --test-mode consumer --consumers 6
```

### MacOS amd64

```bash
wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.9/kafka-stress_0.0.9_darwin_amd64 -O kafka-stress
mv kafka-stress /usr/local/bin
chmod +x /usr/local/bin/kafka-stress
```

### MacOS arm64

```bash
wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.9/kafka-stress_0.0.9_darwin_arm64 -O kafka-stress
mv kafka-stress /usr/local/bin
chmod +x /usr/local/bin/kafka-stress
```

### Linux amd64

```bash
wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.9/kafka-stress_0.0.9_linux_amd64 -O kafka-stress
mv kafka-stress /usr/local/bin
chmod +x /usr/local/bin/kafka-stress
```

### Linux arm64

```bash
wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.9/kafka-stress_0.0.9_linux_arm64 -O kafka-stress
mv kafka-stress /usr/local/bin
chmod +x /usr/local/bin/kafka-stress
```

### Freebsd amd64

```bash
wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.9/kafka-stress_0.0.9_freebsd_amd64 -O kafka-stress
mv kafka-stress /usr/local/bin
chmod +x /usr/local/bin/kafka-stress
```

### Freebsd arm64

```bash
wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.9/kafka-stress_0.0.9_freebsd_arm64 -O kafka-stress
mv kafka-stress /usr/local/bin
chmod +x /usr/local/bin/kafka-stress
```

# v0 Usage

```bash
Usage of kafka-stress:
-ack int
Required ACKs to produce messages (default 1)
-balancer string
Balance algorithm for producer mode; Ex: hash,murmur2,crc32 (default "hash")
-batch-size int
Batch size for producer mode
-bootstrap-servers string
Kafka Bootstrap Servers Broker Lists (default "0.0.0.0:9092")
-consumer-group string
Consumer group name (default "kafka-stress")
-consumers int
Number of consumers will be used in topic (default 1)
-create-topic
Auto Create Topic?
-events int
Numer of events will be created in topic (default 10000)
-format string
Events Format; ex string,json,avro (default "string")
-schema string
Schema
-schema-registry string
Schema Registry URL (default "0.0.0.0:8081")
-size int
Message size in bytes (default 62)
-ssl-enabled
SSL Mode
-test-mode string
Test Type; Ex producer;consumer. Default: producer (default "producer")
-topic string
Kafka Stress Topics (default "kafka-stress")
-verbose
Verbose Mode; It Prints Events consumed
-zookeeper-servers string
Zookeeper Connection String (default "0.0.0.0:2181")
```

## Producer

```bash
kafka-stress --bootstrap-servers localhost:29092 --events 30000 --topic kafka-stress
```

```bash
kafka-stress --bootstrap-servers localhost:29092 --events 10000 --topic kafka-stress

Sent 10000 messages to topic kafka-stress with 0 errors
Tests finished in 1.232463918s. Producer mean time 8113.83/s
```

### Customize ACK's

Use `--ack` parameter to customize ack's quorum

```bash
kafka-stress --bootstrap-servers 0.0.0.0:9092 --events 20000 --topic tunning-3 --test-mode producer --ack 1
```

### Producer Format

You can produce random data in `string` and `json` format using `--format` flag

```bash
kafka-stress --bootstrap-servers 0.0.0.0:9092 --events 20000 --topic tunning-3 --test-mode producer --ack 1 --format json
```

### Producer Balance Algorithms

You can specify some algorithms to balance producer messages like `hash`, `murmur2` and `crc21` using `--balance` flag

```bash
kafka-stress --bootstrap-servers 0.0.0.0:9092 --events 20000 --topic tunning-3 --test-mode producer --balance murmur2 --format json
```

## Consumer

kafka-stress --bootstrap-servers 0.0.0.0:9092 --topic kafka-stress --test-mode consumer --consumers 6
```

```bash
kafka-stress --bootstrap-servers 0.0.0.0:9092 --events 10000 --topic kafka-stress --test-mode consumer --consumers 6

...
[Consumer 6] Message from consumer group kafka-stress at topic/partition/offset kafka-stress/1/0: 65b27cc0-1053-4fbc-b7a9-a40972fcaca6 = a124e95d-9226-4eb9-9169-411318bb6e4e
[Consumer 5] Message from consumer group kafka-stress at topic/partition/offset kafka-stress/2/0: 35fc905f-27f3-4b9f-81db-af89c1fa4c28 = 94f06092-589c-4030-8e3f-66a5a980123b
...
```

### Customize consumer-group name

Use `--consumer-group` to change customer group name used by workers.

```bash
kafka-stress --bootstrap-servers 0.0.0.0:9092 --topic kafka-stress --test-mode consumer --consumer-group custom-consumer-group
```

### Verbose mode

Use `--verbose` to print keys and events

```bash
kafka-stress --bootstrap-servers 0.0.0.0:9092 --topic kafka-stress --test-mode consumer --verbose
```

## Authentication methods

### SSL

Use `--ssl-enabled` to enable ssl authentication.

```bash
kafka-stress --bootstrap-servers 0.0.0.0:9092 --topic kafka-stress --test-mode consumer --ssl-enabled
```

## Roadmap

Improve reports output
* Add retry mechanism
* Add message size options
* Add test header
* Add SCRAM authentication
* Add SASL authetication
* Add TLS authetication
* Add IAM authentication for Amazon MSK
* Add Unit Tests
* Add time based tests
* Add Schema Registry / AVRO, JSON, PROTO Support