https://github.com/yuhexiong/kafka-data-pipeline-structured-flink-java
Data pipeline from Kafka to Kafka, Doris and Doris to Kafka using Flink Java.
https://github.com/yuhexiong/kafka-data-pipeline-structured-flink-java
apache-doris apache-flink apache-kafka doris flink flink-stream-processing kafka
Last synced: 9 months ago
JSON representation
Data pipeline from Kafka to Kafka, Doris and Doris to Kafka using Flink Java.
- Host: GitHub
- URL: https://github.com/yuhexiong/kafka-data-pipeline-structured-flink-java
- Owner: yuhexiong
- Created: 2024-06-21T08:59:51.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-12-03T01:09:31.000Z (about 1 year ago)
- Last Synced: 2025-04-11T00:03:44.135Z (9 months ago)
- Topics: apache-doris, apache-flink, apache-kafka, doris, flink, flink-stream-processing, kafka
- Language: Java
- Homepage:
- Size: 76.2 KB
- Stars: 3
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README-CH.md
Awesome Lists containing this project
README
# Kafka Data Pipeline Flink
使用 Flink 編寫的資料管道，用於將 Kafka 資料傳輸到 Kafka、Doris，也支援合併兩個資料來源。
## Overview
- 平台：JDK 11
- 建構工具：Apache Maven v3.9.6
- 資料處理框架：Flink v1.18.1
## Run
使用 IntelliJ IDEA
## Entry
### 1. KafkaToKafka
將 Kafka (localhost:9092) 中 `topic-source` 的所有產品訊息傳輸到 Kafka 中的 `topic-sink`。
- Kafka 主題 `topic-source` 訊息範例
```json
{
"id": "12345",
"name": "Wireless Mouse",
"category": "Electronics",
"manufacturer": "TechCorp",
"description": "A sleek, ergonomic wireless mouse with advanced optical tracking.",
"cost": 29.99
}
```
### 2. KafkaToDoris
將 Kafka (localhost:9092) 中 `topic-product` 的所有訊息轉換後插入 Doris (localhost:9030) 資料庫 `database.product` 中。
- Kafka 主題 `topic-product` 訊息範例
```json
{
"id": "12345",
"name": "Wireless Mouse",
"category": "Electronics",
"manufacturer": "TechCorp",
"description": "A sleek, ergonomic wireless mouse with advanced optical tracking.",
"cost": 29.99
}
```
- Doris 資料表 `database.product`
```
| id | name | category | manufacturer | description | cost |
|---------|-----------------|-------------|---------------|-----------------------------------|-------|
| 12345 | Wireless Mouse | Electronics | TechCorp | A sleek, ergonomic wireless mouse | 29.99 |
```
### 3. TwoKafkaToDoris
將 Kafka (localhost:9092) 中 `topic-product` 與 `topic-sale` 的訊息結合後，傳輸到 Doris (localhost:9030) 資料庫 `database.sale_report` 中。
- Kafka 主題 `topic-product` 訊息範例
```json
{
"id": "12345",
"name": "Wireless Mouse",
"category": "Electronics",
"manufacturer": "TechCorp",
"description": "A sleek, ergonomic wireless mouse with advanced optical tracking.",
"cost": 29.99
}
```
- Kafka 主題 `topic-sale` 訊息範例
```json
{
"id": "A98765",
"productId": "12345",
"unit": 3,
"unitPrice": 49.99,
"totalPrice": 149.97,
"saleDate": "2024-11-28"
}
```
- Doris 資料表 `database.sale_report`
```
| sale_id | product_id | unit | unit_price | total_price | sale_date | product_name | product_unit_cost | profit |
|-----------|------------|------|------------|-------------|----------------------|---------------|-------------------|--------|
| A98765 | 12345 | 3 | 49.99 | 149.97 | 2024-11-28T08:00:00 | Wireless Mouse| 29.99 | 60.00 |
```