Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/james-kennet/kafka-project
Spring Boot, Kafka, Zookeeper, Kafdrop(kafka viewer), Kafka producer, Kafka consumer, Spring Data/JPA, MySQL, Microservices, Rest API
https://github.com/james-kennet/kafka-project
kafdrop kafka kafka-consumer kafka-producer kafka-viewer microservices mysql rest-api springboot2 zookeeper
Last synced: 6 days ago
JSON representation
Spring Boot, Kafka, Zookeeper, Kafdrop(kafka viewer), Kafka producer, Kafka consumer, Spring Data/JPA, MySQL, Microservices, Rest API
- Host: GitHub
- URL: https://github.com/james-kennet/kafka-project
- Owner: james-kennet
- License: gpl-3.0
- Created: 2024-07-25T16:34:37.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2024-08-18T10:27:43.000Z (3 months ago)
- Last Synced: 2024-08-18T11:39:08.631Z (3 months ago)
- Topics: kafdrop, kafka, kafka-consumer, kafka-producer, kafka-viewer, microservices, mysql, rest-api, springboot2, zookeeper
- Language: Java
- Homepage:
- Size: 8.97 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Spring Boot, Kafka, Zookeeper, Kafdrop, Spring Data/JPA, MySQL, Rest API
## Sample kafka-project
- Single topic per individual platform.
- Kafka as a traditional message broker.
- Single topic to all platforms.
- Different clientType value attribute. To be processed separately using recordFilterStrategy.
### Use Cases(Address creation)
* Improve the speed of synchronizing customer information in a typical Insurance domain.
* In the Insurance domain, multiple platforms are able to update customer information, like addresses:
* Website application
* Salesforce
* Life400 - mainframe
* Use Kafka to receive and process all ADDRESS creation from multiple platforms and process to the downstream services.
* Introduced address_transaction table which will be referred by the consumers.
### Use Case #1
* Kafka as traditional message broker.
* One topic per platform
* ADDRESS_CREATE_SALESFORCE
* ADDRESS_CREATE_WEBSITE
* ADDRESS_CREATE_LIFE400
* One producer endpoint per platform/topic.
* One consumer listener per platform/topic.
#### Sample High-level architecture diagram
### Use Case #2
* Kafka single topic with consumer using recordFilterStrategy.
* One topic for all platforms with different clientType.
* ADDRESS_CREATE_ALL_PLATFORM
* One producer endpoint to all platforms.
* One consumer listener per platform clientType by using recordFilterStrategy.
* containerFactory = "allPlatformSalesforceFactory"
* containerFactory = "allPlatformWebsiteFactory"
* containerFactory = "allPlatformLife400Factory"
#### Sample High-level architecture diagram
* 
### Prerequisites on running this project:
- Install Java 11(Needed by Kafkdrop - kafka message viewer).
- Install Docker Desktop.
- JMeter - for testing.
### Tech stack and services in kafka-project.
- kafka
- zookeeper
- Kafkdrop - kafka message viewer
- kafka-producer service
- kafka-consumer service
- address-service - services to process consume messages
- MySql - to save all transactions
### Steps in running kafka-project.
- Checkout the project in Github.
- Go the project directory.
```bash
cd //kafka-project
```
- Install kafka-project services.
```bash
mvn clean install
```
- Start kafka-project services.
```bash
docker-compose up
```
- View Kafdrop on the browser using the below URL.
```bash
http://localhost:9000
```
### Testing kafka-project locally.
- Copy the JMeter script to JMeter bin directory.
```bash
JMeter script - kafka-project/_external_files/kafka-producer-v1.jmx
JMeter bin - /bin
cp
```
- Open JMeter interface. No need to run in non-GUI mode.
```bash
cd
./jmeter.sh
```
- Execute JMeter script.
- 
### View Kafka messages using Kafdrop.
- 
- 
- 
- 
### View the address records.
- Setup MySQL client.
- 
- Verify the data.
- 