https://github.com/msergo/flink-landed-planes

A Flink job that reads OpenSky flight states from RabbitMQ and writes to a DB when a plane is landed
https://github.com/msergo/flink-landed-planes

Last synced: about 1 month ago
JSON representation

A Flink job that reads OpenSky flight states from RabbitMQ and writes to a DB when a plane is landed

• Host: GitHub
• URL: https://github.com/msergo/flink-landed-planes
• Owner: msergo
• Created: 2024-11-14T20:07:34.000Z (about 1 month ago)
• Default Branch: master
• Last Pushed: 2024-11-14T20:17:12.000Z (about 1 month ago)
• Last Synced: 2024-11-14T21:23:18.374Z (about 1 month ago)
• Language: Java
• Size: 13.7 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# A Flink job that reads OpenSky flight states from RabbitMQ and writes to a DB when a plane is landed.

This project is an experiment to collect states when a specific moment happens, in this case, when a plane is landed. So

transition from "IN AIR" to "LANDED" is the moment we are interested in.

Source of data is RabbitMQ, where OpenSky vectors are fed every minute.

Stream data is enriched with airport information (many thanks

to [ip2location project](https://github.com/ip2location/ip2location-iata-icao/tree/master))

## Flink features used

* Custom RMQSource

* RichAsyncFunction

* Custom Watermark strategy

* State backend: RocksDB

Postgres is used as a database to store the data, but can be easily changed to any other database.

All the params are submitted as arguments

```shell

--RABBITMQ_HOST rabbitmq-service --RABBITMQ_USER admin --RABBITMQ_PASSWORD admin --DB_URL  "jdbc:postgresql://postgres-url/flink-db?currentSchema=public" --DB_DRIVER org.postgresql.Driver --DB_USER flink --DB_PASSWORD 12345 

```

## How to run

I used a Flink cluster running on Kubernetes, but it can run locally as well, including writes to a SQLIte database.