https://github.com/ahmetfurkandemir/airflow-spark-kafka-example

Airflow, Spark and Kafka example
https://github.com/ahmetfurkandemir/airflow-spark-kafka-example

airflow airflow-dags airflow-docker docker kafka kafka-producer kafka-ui python3 spark spark-jobs

Last synced: 2 months ago
JSON representation

Airflow, Spark and Kafka example

• Host: GitHub
• URL: https://github.com/ahmetfurkandemir/airflow-spark-kafka-example
• Owner: AhmetFurkanDEMIR
• Created: 2023-11-21T19:38:28.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2023-11-21T19:45:15.000Z (about 1 year ago)
• Last Synced: 2024-05-07T18:16:59.975Z (9 months ago)
• Topics: airflow, airflow-dags, airflow-docker, docker, kafka, kafka-producer, kafka-ui, python3, spark, spark-jobs
• Language: Dockerfile
• Homepage:
• Size: 532 KB
• Stars: 4
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
![](https://img.shields.io/badge/Airflow-017CEE?style=for-the-badge&logo=Apache%20Airflow&logoColor=white) ![](https://img.shields.io/badge/Apache_Spark-FFFFFF?style=for-the-badge&logo=apachespark&logoColor=#E35A16) ![](https://img.shields.io/badge/Apache_Kafka-231F20?style=for-the-badge&logo=apache-kafka&logoColor=white) ![](https://img.shields.io/badge/Docker-2CA5E0?style=for-the-badge&logo=docker&logoColor=white) ![](https://img.shields.io/badge/Python-FFD43B?style=for-the-badge&logo=python&logoColor=blue)

## Airflow, Spark and Kafka example

![](/images/airflow_dag.png)

![](/images/airflow_tree.png)

![](/images/spark_job.png)

![](/images/kafka_topic_message.png)

![](/images/kafka_topic_consumer.png)