https://github.com/anusikh/100k-data-streaming
flink-kafka-postgres
https://github.com/anusikh/100k-data-streaming
Last synced: about 1 month ago
JSON representation
flink-kafka-postgres
- Host: GitHub
- URL: https://github.com/anusikh/100k-data-streaming
- Owner: anusikh
- Created: 2024-02-07T19:36:58.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-02-10T09:12:26.000Z (over 2 years ago)
- Last Synced: 2025-03-01T11:27:02.626Z (over 1 year ago)
- Language: Shell
- Size: 250 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Flink + Kafka
### about
- generate-data service creates 100k rows in postgres
- kafka-producer service reads the data and sends it to kafka
- flink-processor runs at the same time with a tumbling window of 5 mins and processes the sum of the costs based on customer_name's
- if a tumbling window has a window interval of 10 seconds, every incoming event from a data stream for 10 seconds will fall into the same window
- then we compare the results from flink by running a sql query and it's accurate
.
### setup
- create jars in flink-processor folder
```
mvn clean install
```
- to start containers
```
docker-compose up -d
```
- kafka tricks:
```sh
kafka-topics.sh --bootstrap-server kafka:9092 --create --topic datastream
```
```sh
# to check list of topics
kafka-topics.sh --bootstrap-server kafka:9092 --list
```
- in code, for local testing, replace all the container name in url's with localhost. To make it work in docker, we need to make sure we have container names instead