https://github.com/ververica/lab-paimon-analytics

Set of Flink jobs to ingest data into Paimon tables, then join it and visualize
https://github.com/ververica/lab-paimon-analytics

Last synced: 14 days ago
JSON representation

Set of Flink jobs to ingest data into Paimon tables, then join it and visualize

• Host: GitHub
• URL: https://github.com/ververica/lab-paimon-analytics
• Owner: ververica
• License: apache-2.0
• Created: 2023-12-06T15:12:53.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2024-01-17T12:52:36.000Z (about 1 year ago)
• Last Synced: 2024-11-12T21:17:32.288Z (2 months ago)
• Language: Jupyter Notebook
• Size: 24.4 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Build real-time view in Data Lake

This repository contains Flink jobs to process data with a help of Apache Paimon.

Pre-requisites:

1. MySQL server 8.x

2. [Ammonite](https://ammonite.io/)

3. Apache Flink cluster 1.17 or higher

## Ingestion jobs

1. Create MySQL Database and apply `mysql.sql` DDL script to it.

2. Create Flink SQL jobs using `flink-sql/ingestion.sql` code.

3. Run Flink CDC job using Ammonite: `amm runCdcAction.sc`

## Aggregation jobs

Create Flink SQL job to aggregate data using `flink-sql/aggregation.sql`.

## Visualize data view

Run Paimon table stream-reader to print current data continuously in the console using Ammonite: `amm readStream.sc`.

## Sync via Merge action

In order to sync country_sales table with customers run the Paimon Merge action via Ammonite: `amm mergeDiff.sc`.