https://github.com/isopropylcyanide/springbatch-kafkadb
A small demo that leverages Spring batch's capabilities to do job processing and Apache Kafka's stream processing. A simple CSV file gets used up in a batch job which then writes it to a Kafka queue (and H2 database) for further processing.
https://github.com/isopropylcyanide/springbatch-kafkadb
apache-kafka h2-database kafka kafka-consumer kafka-producer spring-batch spring-boot
Last synced: 3 months ago
JSON representation
A small demo that leverages Spring batch's capabilities to do job processing and Apache Kafka's stream processing. A simple CSV file gets used up in a batch job which then writes it to a Kafka queue (and H2 database) for further processing.
- Host: GitHub
- URL: https://github.com/isopropylcyanide/springbatch-kafkadb
- Owner: isopropylcyanide
- Created: 2018-08-07T11:16:48.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2018-09-06T12:07:39.000Z (about 7 years ago)
- Last Synced: 2025-05-03T07:37:12.499Z (6 months ago)
- Topics: apache-kafka, h2-database, kafka, kafka-consumer, kafka-producer, spring-batch, spring-boot
- Language: Java
- Homepage:
- Size: 27.3 KB
- Stars: 14
- Watchers: 2
- Forks: 14
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## SpringBatch-KafkaDB Demo ##
---
### What is it? ###
A small demo that leverages Spring batch's capabilities to do job processing and Apache Kafka's stream processing.
A simple CSV file is used in a batch job which then writes it to a Kafka Producer for further processing.
A Kafka consumer can then verify by consuming the messages from the correct topic.
---
### Spring Batch pipeline ###
Below mentioned pipeline has been followed through out the codebase. The implementation is trivial once you modularize the responsibilities of each relevant class.
---
### Prerequisites ###
* Spring Boot + Batch + JPA
* Apache Kafka
* Apache Zookeeper
---
### Expectation ###
Batch systems offer tremendous advantages as compared to interactive systems.
* Repeated jobs are done fast in batch systems without user interaction.
* You don’t need special hardware and system support to input data in batch systems.
* Best for large organizations but small organizations can also benefit from it.
---
Expectation is to convert the following flat file into something meaningful when run as a batch process.
* Such as a Kafka stream like this *
* Or to a datastore like this *
---
### Setting up Apache Kafka ###
```
# Start Zookeeper instance
$ zookeeper-server-start.bat ..\..\config\zookeeper.properties
# Start Kafka server
$ kafka-server-start.bat ..\..\config\server.properties
# Create a topic
$ kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic CSV_TOPIC_K
```
Make sure following is appended to **config\server.properties**
```
port = 9092
advertised.host.name = localhost
```
---
### What are the list of branches ###
Branch | Description
------------ | -------------
master | Base branch that reads from CSV and processes them to a topic in a Kafka producer
batch-db-upload | Similar to master except that it deserializes the CSV to a H2 Database instead of Kafka