https://github.com/jaki729/simple_kafkaairflowstream
Sample Data Pipeline
https://github.com/jaki729/simple_kafkaairflowstream
airflow docker kafka python3 sql
Last synced: 15 days ago
JSON representation
Sample Data Pipeline
- Host: GitHub
- URL: https://github.com/jaki729/simple_kafkaairflowstream
- Owner: jaki729
- Created: 2025-01-07T15:51:00.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-01-08T06:22:26.000Z (over 1 year ago)
- Last Synced: 2025-08-13T19:40:49.531Z (11 months ago)
- Topics: airflow, docker, kafka, python3, sql
- Language: Python
- Homepage:
- Size: 49.8 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Simple_KafkaAirflowStream
KafkaAirflowStream is a project designed to integrate Kafka for real-time data streaming and Apache Airflow for orchestrating complex data pipelines. The system captures and processes streams of data, managing their flow from data generation, through Kafka's producer and broker architecture, to consumption and processing. Apache Airflow handles the scheduling and management of these processes, ensuring reliable and efficient operation. The data is stored in a MySQL database for further analysis. This project aims to create a robust, scalable, and Dockerized solution for handling and orchestrating real-time data streams with a focus on flexibility and automation.