https://github.com/el-moudni-hicham/springcloud-kafka
This repository contains code demonstrating the integration of Spring Cloud services with Apache Kafka for building scalable and resilient applications with event-driven architecture.
https://github.com/el-moudni-hicham/springcloud-kafka
consumer kafka kafka-streams producer spring-boot spring-cloud
Last synced: 8 months ago
JSON representation
This repository contains code demonstrating the integration of Spring Cloud services with Apache Kafka for building scalable and resilient applications with event-driven architecture.
- Host: GitHub
- URL: https://github.com/el-moudni-hicham/springcloud-kafka
- Owner: el-moudni-hicham
- Created: 2023-10-02T18:17:12.000Z (about 2 years ago)
- Default Branch: master
- Last Pushed: 2023-10-03T19:28:18.000Z (about 2 years ago)
- Last Synced: 2025-01-13T10:09:28.383Z (9 months ago)
- Topics: consumer, kafka, kafka-streams, producer, spring-boot, spring-cloud
- Language: Java
- Homepage:
- Size: 79.1 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Event Driven Distributed Processing
- KAFKA, KAFKA STREAMS
- Spring Cloud Stream Functions
```
This project leverages Event-Driven Distributed Processing by employing Apache Kafka and Kafka Streams
for real-time data ingestion, processing, and analysis.
Additionally, it utilizes Spring Cloud Stream Functions
```
# Table of Contents
- [Prerequisites](#prerequisites)
- [Starting KAFKA Server](#starting-kafka-server)
- [Start zookeeper](#start-zookeeper)
- [Start KAFKA server](#start-kafka-server)
- [Services Types](#services-types)
- [KAFKA Console](#kafka-console)
- [Docker Compose](#docker-compose)
- [Rest Controller with Stream Bridge](#rest-controller-with-stream-bridge)
- [Cunsomer](#cunsomer)
- [Supplier](#supplier)
- [Function](#function)
- [KAFKA Stream](#kafka-stream)
## Prerequisites
Before running this application, you need to have the following software installed on your system :
```java
- Java Development Kit (JDK) version 11 or later
- Kafka
```
## Project Structure
D:.
│
├───src
│ ├───main
│ │ ├───java
│ │ │ └───ma
│ │ │ └───enset
│ │ │ └───springcloudkafkastreams
│ │ │ │ SpringcloudKafkaStreamsApplication.java
│ │ │ │
│ │ │ ├───entites
│ │ │ │ PageEvent.java
│ │ │ │
│ │ │ ├───service
│ │ │ │ PageEventService.java
│ │ │ │
│ │ │ └───web
│ │ │ PageEventRestController.java
│ │ │
│ │ └───resources
│ │ │ application.properties
│ │ │
│ │ ├───static
│ │ └───templates
│
│ docker-compose.yml
│ pom.xml
## Starting KAFKA Server
### Start zookeeper
`start bin\windows\zookeeper-server-start.bat config/zookeeper.properties`
### Start KAFKA server
`start bin\windows\kafka-server-start.bat config/server.properties`
## Services Types
### Kafka Console
> Subscribing to a topic to consume messages.
`start kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic T1`
> Producing messages to the topic
`start kafka-console-producer.bat --broker-list localhost:9092 --topic T1`
### Docker compose
`docker-compose up -d`
`docker exec --interactive --tty broker kafka-console-consumer --bootstrap-server broker:9092 --topic T2`
`docker exec --interactive --tty broker kafka-console-producer --bootstrap-server broker:9092 --topic T2`
### Rest Controller with Stream Bridge
```java
@GetMapping("/publish/{topic}/{name}")
public PageEvent publish(@PathVariable String topic, @PathVariable String name){
PageEvent pageEvent = new PageEvent(
name,
Math.random()>0.5?"U1":"U2",
new Date(),
(long) new Random().nextInt(9000));
streamBridge.send(topic, pageEvent);
return pageEvent;
}
```
### Cunsomer
```java
@Bean
public Consumer pageEventConsumer(){
return (input)->{
System.out.println(input.toString());
};
}
```
### Supplier
```java
@Bean
public Supplier pageEventSupplier(){
return ()-> new PageEvent(
Math.random()>0.5?"P1":"P2",
Math.random()>0.5?"U1":"U2",
new Date(),
(long) new Random().nextInt(9000)
);
}
```
### Function
```java
@Bean
public Function pageEventFunction(){
return (input)->{
input.setName(input.getName()+"after");
input.setUser("User");
return input;
};
}
```
### KAFKA Stream
* Data Analytics Real Time Stream Processing with Kaflka Streams
```java
@Bean
public Function, KStream> kStreamFunction(){
return (input)->{
return input
.filter((k,v) -> v.getDuration()>100)
.map((k,v) -> new KeyValue<>(v.getName(), 0L))
.groupBy((k,v) -> k, Grouped.with(Serdes.String(), Serdes.Long()))
.windowedBy(TimeWindows.of(Duration.ofMillis(5000)))
.count(Materialized.as("count-pages"))
.toStream()
.map((k,v) -> new KeyValue<>(""+ k.window().startTime() + " -> " + k.window().endTime() + " , page : " + k.key() , v));
};
}
```
* Web application that displays the results of Stream Data Analytics in real time
