https://github.com/dalelane/sample-kafka-java-apps
https://github.com/dalelane/sample-kafka-java-apps
Last synced: about 2 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/dalelane/sample-kafka-java-apps
- Owner: dalelane
- License: apache-2.0
- Created: 2024-07-06T10:58:19.000Z (10 months ago)
- Default Branch: master
- Last Pushed: 2024-08-21T09:48:28.000Z (9 months ago)
- Last Synced: 2025-01-23T22:16:53.769Z (4 months ago)
- Language: Java
- Size: 37.1 KB
- Stars: 0
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Simple Kafka client apps
Sample Java applications for sending, receiving, and processing messages on Kafka topics. These require Java and Maven to build and run, and a Kafka cluster to connect to.
Four variations are included:
- [Text](#text)
- [JSON](#json)
- [Avro](#avro)
- [Bytes](#bytes)
## Text
### Sending
- [`TextProducer`](./src/main/java/com/ibm/eventautomation/demos/producers/TextProducer.java)
This sends the contents of each txt file in the [`testdata/text`](./testdata/text/) folder to a Kafka topic - each file contents a separate Kafka message.
It exits once it has sent all of the files in the test data folder.
**To modify what this sends**, create text files in the test data folder. Ensure that your files have a `.txt` file extension.
**To modify the Kafka topic it sends to**, modify the [`producer.properties`](./testdata/producer.properties) properties file.
### Receiving
- [`TextConsumer`](./src/main/java/com/ibm/eventautomation/demos/consumers/TextConsumer.java)
This receives messages from a Kafka topic and prints each to stdout.
It will keep doing this until the app is killed.
**To modify the Kafka topic it receives from**, modify the [`consumer.properties`](./testdata/consumer.properties) properties file.
### Processing
- [`TextProcessor`](./src/main/java/com/ibm/eventautomation/demos/streamprocessors/TextProcessor.java)
This consumes text messages from an input Kafka topic, and produces messages with an upper-cased version of the text to an output topic.
It will keep doing this until the app is killed.
**To modify the Kafka topics it uses**, modify the [`streams.properties`](/testdata/streams.properties) properties file.
**To modify the processing it performs**, modify [the Java source code](./src/main/java/com/ibm/eventautomation/demos/streamprocessors/TextProcessor.java#L49).
### Scripts
- **To compile**: [`./scripts/compile.sh`](./scripts/compile.sh)
- **To run**:
- [`./scripts/produce-text.sh`](./scripts/produce-text.sh)
- [`./scripts/consume-text.sh`](./scripts/consume-text.sh)
- [`./scripts/process-text.sh`](./scripts/process-text.sh)
## JSON
### Sending
- [`JsonProducer`](./src/main/java/com/ibm/eventautomation/demos/producers/JsonProducer.java)
This sends the contents of each json file in the [`testdata/json`](./testdata/json/) folder to a Kafka topic - each file contents a separate Kafka message.
It exits once it has sent all of the files in the test data folder.
**To modify what this sends**, create JSON files in the test data folder. Ensure that your files hava a `.json` file extension, and contain a valid JSON object.
**To modify the Kafka topic it sends to**, modify the [`producer.properties`](./testdata/producer.properties) properties file.
### Receiving
- [`JsonConsumer`](./src/main/java/com/ibm/eventautomation/demos/consumers/JsonConsumer.java)
This receives messages from a Kafka topic and prints each to stdout.
It will keep doing this until the app is killed.
**To modify the Kafka topic it receives from**, modify the [`consumer.properties`](./testdata/consumer.properties) properties file.
### Processing
- [`JsonProcessor`](./src/main/java/com/ibm/eventautomation/demos/streamprocessors/JsonProcessor.java)
This consumes messages from an input Kafka topic, and produces messages that match a filter to an output Kafka topic.
It will keep doing this until the app is killed.
**To modify the Kafka topics it uses**, modify the [`streams.properties`](/testdata/streams.properties) properties file.
**To modify the data it can process**, modify [the Java source code](./src/main/java/com/ibm/eventautomation/demos/streamprocessors/JsonProcessor.java#L53-L56).
### Scripts
- **To compile**: [`./scripts/compile.sh`](./scripts/compile.sh)
- **To run**:
- [`./scripts/produce-json.sh`](./scripts/produce-json.sh)
- [`./scripts/consume-json.sh`](./scripts/consume-json.sh)
- [`./scripts/process-json.sh`](./scripts/process-json.sh)
## Avro
### Sending
- [`AvroProducer`](./src/main/java/com/ibm/eventautomation/demos/producers/AvroProducer.java)
This sends hard-coded messages to a Kafka topic, using the Avro schema [`testdata/avro/schema.avsc`](./testdata/avro/schema.avsc).
It exits once it has sent all of the messages in the app.
**To modify what this sends**, modify [the Java source code](./src/main/java/com/ibm/eventautomation/demos/producers/AvroProducer.java#L60-L87). To change the message format, you will also need to modify the [the Avro schema](./testdata/avro/schema.avsc) to match.
**To modify the Kafka topic it sends to**, modify the [`producer.properties`](./testdata/producer.properties) properties file.
### Receiving
- [`AvroConsumer`](./src/main/java/com/ibm/eventautomation/demos/consumers/AvroConsumer.java)
This receives messages from a Kafka topic, deserializes them using the Avro schema [`testdata/avro/schema.avsc`](./testdata/avro/schema.avsc), and prints properties from each to stdout.
It will keep doing this until the app is killed.
**To modify what it prints**, modify [the Java source code](./src/main/java/com/ibm/eventautomation/demos/consumers/AvroConsumer.java#L60-L64).
**To modify the Kafka topic it receives from**, modify the [`consumer.properties`](./testdata/consumer.properties) properties file.
### Scripts
- **To compile**: [`./scripts/compile.sh`](./scripts/compile.sh)
- **To run**:
- [`./scripts/produce-avro.sh`](./scripts/produce-avro.sh)
- [`./scripts/consume-avro.sh`](./scripts/consume-avro.sh)
## Bytes
### Sending
- [`RandomBytesProducer`](./src/main/java/com/ibm/eventautomation/demos/producers/RandomBytesProducer.java)
This sends [10,000](./src/main/java/com/ibm/eventautomation/demos/producers/RandomBytesProducer.java#L48) messages, each containing a randomly-generated [128](./src/main/java/com/ibm/eventautomation/demos/utils/Utils.java#L67)-byte array, to a Kafka topic.
It exits once all messages have been successfully sent.
**To modify the Kafka topic it sends to**, modify the [`producer.properties`](./testdata/producer.properties) properties file.
### Receiving
- [`BytesConsumer`](./src/main/java/com/ibm/eventautomation/demos/consumers/BytesConsumer.java)
This receives messages from a Kafka topic and prints each to stdout in a hex format.
It will keep doing this until the app is killed.
**To modify the Kafka topic it receives from**, modify the [`consumer.properties`](./testdata/consumer.properties) properties file.
### Scripts
- **To compile**: [`./scripts/compile.sh`](./scripts/compile.sh)
- **To run**:
- [`./scripts/produce-random.sh`](./scripts/produce-random.sh)
- [`./scripts/consume-bytes.sh`](./scripts/consume-bytes.sh)