https://github.com/scribd/kafka-player

A simple scala app that plays messages from a file onto a Kafka topic.
https://github.com/scribd/kafka-player

cplat kafka

Last synced: about 2 months ago
JSON representation

A simple scala app that plays messages from a file onto a Kafka topic.

• Host: GitHub
• URL: https://github.com/scribd/kafka-player
• Owner: scribd
• License: mit
• Created: 2020-03-20T17:55:48.000Z (almost 5 years ago)
• Default Branch: master
• Last Pushed: 2021-01-04T17:47:25.000Z (about 4 years ago)
• Last Synced: 2023-04-10T07:46:18.649Z (almost 2 years ago)
• Topics: cplat, kafka
• Language: Scala
• Size: 355 KB
• Stars: 6
• Watchers: 8
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

        
# kafka-player

Plays a file onto a Kafka topic - one line == one message.

### Build

Build the project by running `sbt assembly`. This produces an uber jar at `target/scala-2.12/kafka-player.jar`.

### Invocation

For running against a local Kafka, invoke the player with a command like below:

```

java -jar $jar_file \

  --message-file $message_file \

  --num-messages $num_messages \

  --messages-per-second $messages_per_second \

  --topic $topic \

  --broker-string $broker_string

```

If running against a TLS authenticated Kafka cluster, include additional parameters to specify the keystore location and keystore password:

```

  --keystore-location $keystore_location

  --keystore-passphrase $keystore_passphrase

```

#### Parameters

* `--message-file` - A file containing the lines to play onto Kafka. The player will play one message per line.

* `--num-messages` - The number of messages to play. The default is 100,000 if not provided.

* `--messages-per-second` - The number of messages to play per second. The default is 2 per second if not provided. Float values are supported (e.g. 0.5 to produce one message every two seconds).

* `--topic` - The topic to play the messages onto.

* `--broker-string` - The bootstrap broker string of the target cluster. The default is localhost:9092 if not provided.

* `--keystore-location` - The location of the keystore to use for TLS authentication. Only set this if TLS authentication is desired.

* `--keystore-passphrase` - The passphrase for the keystore. Only set this if TLS authentication is desired and a `keystore-location` is also provided.

### Local Kafka

A docker-compose file is included for local development for spinning up a single broker wurstmeister/kafka cluster (https://hub.docker.com/r/wurstmeister/kafka/) exposed to localhost.

Topics required for specific scenarios may be added to the `KAFKA_CREATE_TOPICS` environment variable in `docker-compose.yml`.

The required format is described in https://github.com/wurstmeister/kafka-docker#automatically-create-topics.

### Example

The `example` folder of this repository contains a data file called `example.json` and a launch script called `play-example.sh` to play the data file onto a local Kafka. Run the example with the following steps:

```

# start the kafka docker containers in the background - this will also create a topic called `example` on the cluster.

docker-compose up -d

# play the example data file onto the local kafka

./example/play-example.sh

```

Once the player starts, run `kafkacat` in another terminal to verify your messages are coming in.

```

kafkacat -C -b localhost:9092 -t example

```

The example plays 10,000 messages at a rate of one message every two seconds.

The `example.json` data file is an abridged and slightly transformed (into a line delimited format) version of American movies from https://github.com/jdorfman/awesome-json-datasets#movies.