https://github.com/prekshivyas/datastreamingetl
Utilizing my background and love for Apache Airflow and Data to build a real-time data streaming pipeline
https://github.com/prekshivyas/datastreamingetl
apache-airflow apache-kafka apache-spark apache-zookeeper cassandra data-engineering data-ingestion data-pipeline data-processing data-visualization docker docker-compose
Last synced: 4 months ago
JSON representation
Utilizing my background and love for Apache Airflow and Data to build a real-time data streaming pipeline
- Host: GitHub
- URL: https://github.com/prekshivyas/datastreamingetl
- Owner: prekshivyas
- Created: 2024-06-21T02:52:15.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-06-21T09:35:39.000Z (almost 2 years ago)
- Last Synced: 2025-02-13T08:16:47.260Z (over 1 year ago)
- Topics: apache-airflow, apache-kafka, apache-spark, apache-zookeeper, cassandra, data-engineering, data-ingestion, data-pipeline, data-processing, data-visualization, docker, docker-compose
- Language: Python
- Homepage:
- Size: 722 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# DataStreamingETL
Utilizing my background and love for Apache Airflow and data to build a real-time data streaming pipeline, covering each phase from data ingestion to processing and finally storage.
The system is built using Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra — all neatly containerized using Docker for an end to end project!
## Architecture
- The project is designed with the following components:
- Data Source: randomuser.me API to generate random user data for the pipeline.
- Apache Airflow: For orchestrating the pipeline and storing fetched data in a PostgreSQL database.
- Apache Kafka and Zookeeper: For streaming data from PostgreSQL to the processing engine.
- Control Center and Schema Registry: For monitoring and schema management of Kafka streams. Control Center listens for events on the Schema Registry to visualize data directly on Kafka that is managed by Zookeeper.
- Apache Spark: For data processing with master and worker nodes.
- Cassandra: Where the processed data will be stored.
## Steps to run the project:
1. Clone the repository:
```
git clone https://github.com/prekshivyas/DataStreamingETL.git
```
2. Navigate to the project directory:
```
cd DataStreamingETL
```
Run Docker Compose to spin up the services:
```
docker-compose up
```