https://github.com/nkonev/flink-pipe

A [single|uber|fat] jar standalone Apache Flink connected to PostgreSQL via Ververica CDC connector. Usage Flink SQL to replicate data from PostgreSQL to Elasticsearch/something else.
https://github.com/nkonev/flink-pipe

cdc debezium debezium-connector debezium-embedded elastic elasticsearch flink flink-cdc flink-sql flink-stream-processing flink-streaming postgres postgresql replication ververica-connector

Last synced: 26 days ago
JSON representation

A [single|uber|fat] jar standalone Apache Flink connected to PostgreSQL via Ververica CDC connector. Usage Flink SQL to replicate data from PostgreSQL to Elasticsearch/something else.

• Host: GitHub
• URL: https://github.com/nkonev/flink-pipe
• Owner: nkonev
• Created: 2024-01-31T19:06:03.000Z (9 months ago)
• Default Branch: master
• Last Pushed: 2024-02-11T22:52:50.000Z (9 months ago)
• Last Synced: 2024-04-16T18:24:32.109Z (7 months ago)
• Topics: cdc, debezium, debezium-connector, debezium-embedded, elastic, elasticsearch, flink, flink-cdc, flink-sql, flink-stream-processing, flink-streaming, postgres, postgresql, replication, ververica-connector
• Language: Kotlin
• Homepage:
• Size: 77.1 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: readme.md

Awesome Lists containing this project

README

        
# What is it ?

It is Apache Flink with [Ververica Postgres CDC](https://github.com/ververica/flink-cdc-connectors) built as one fat|uber jar to be launched via `java -jar` as a standalone application.

The purpose is - declaratively (via Flink SQL) replicate data from PostgreSQL to any other datastores.

A docker image of PostgreSQL 16 configured by Debuzium  included to be run via `docker-compose up -d`.

Sample data is placed to the predefined by Postgres Docker `docker-entrypoint-initdb.d` directory.

Concrete example to replicate data to Elasticsearch 7 derived from [example](https://www.ververica.com/blog/how-to-guide-build-streaming-etl-for-mysql-and-postgres-based-on-flink-cdc) is in branch `elastic`.

# Build

```

./gradlew clean shadowJar

```

# Run

```

java -jar /home/nkonev/javaWorkspace/flink-pipe/build/libs/flink-pipe-0.1.0-all.jar

```

# Links

* [Streaming SQL with Apache Flink: A Gentle Introduction](https://blog.rockthejvm.com/flink-sql-introduction/)

* [How-to guide: Build Streaming ETL for MySQL and Postgres based on Flink CDC](https://www.ververica.com/blog/how-to-guide-build-streaming-etl-for-mysql-and-postgres-based-on-flink-cdc)

* [Streaming ETL for MySQL and Postgres with Flink CDC](https://ververica.github.io/flink-cdc-connectors/release-3.0/content/quickstart/mysql-postgres-tutorial.html)

* [Потоковый захват изменений из PostgreSQL/MySQL с помощью Apache Flink](https://habr.com/ru/companies/neoflex/articles/567930/)

* [Как использовать Spring в качестве фреймворка для Flink-приложений](https://habr.com/ru/companies/ru_mts/articles/775970/)

* [debezium/container-images](https://github.com/debezium/container-images/tree/main/examples/postgres)