https://github.com/aditya1191/cloud-data-engineering

ETL Python files for loading data to cloud
https://github.com/aditya1191/cloud-data-engineering

airflow aws aws-s3 cassandra datalake-etl lambda postgresql redshift snowflake

Last synced: 5 days ago
JSON representation

ETL Python files for loading data to cloud

• Host: GitHub
• URL: https://github.com/aditya1191/cloud-data-engineering
• Owner: aditya1191
• Created: 2024-11-07T23:07:30.000Z (3 months ago)
• Default Branch: main
• Last Pushed: 2025-01-30T02:35:48.000Z (5 days ago)
• Last Synced: 2025-01-30T03:23:52.762Z (5 days ago)
• Topics: airflow, aws, aws-s3, cassandra, datalake-etl, lambda, postgresql, redshift, snowflake
• Language: Python
• Homepage:
• Size: 1.56 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Data Engineering Projects

## Project 1: Data Modeling with Postgres

In this project, I apply Data Modeling with Postgres and build an ETL pipeline using Python. A startup wants to analyze the data they've been collecting on songs and user activity on their new music streaming app. Currently, they are collecting data in json format and the analytics team is particularly interested in understanding what songs users are listening to.

## Project 2: Data Modeling with Cassandra

In this project, I apply Data Modeling with Cassandra and build an ETL pipeline using Python. I will build a Data Model around our queries that I want to get answers for. 

For my use case I want below answers: 

- Get details of a song that was heard on the music app history during a particular session. 

- Get songs played by a user during particular session on music app. 

- Get all users from the music app history who listened to a particular song.

## Project 3: Data Warehouse

In this project, I apply the Data Warehouse architectures I learnt and built a Data Warehouse on AWS cloud. I build an ETL pipeline to extract and transform data stored in json format in s3 buckets and move the data to Warehouse hosted on Amazon Redshift. 

## Project 4: Data Lake

In this project, I will build a Data Lake on AWS cloud using Spark and AWS EMR cluster. The data lake will serve as a Single Source of Truth for the Analytics Platform. I will write spark jobs to perform ELT operations that picks data from landing zone on S3 and transform and stores data on the S3 processed zone.

## Project 5: Data Pipelines with Airflow

In this project, I will orchestrate our Data Pipeline workflow using an open-source Apache project called Apache Airflow. I will schedule our ETL jobs in Airflow, create project related custom plugins and operators and automate the pipeline execution. 

## Project 6: Api Data to Postgres

In this project, I built an etl pipeline to fetch data from yelp API and insert it into the Postgres Database. This project is a very basic example of fetching real time data from an open source API.