https://github.com/pylena/sparkifydatawarehouse

ETL pipeline that extracts data from S3, stages them in Redshift, and transforms data into a set of dimensional tables for analytics
https://github.com/pylena/sparkifydatawarehouse

aws-redshift aws-s3 etl sql

Last synced: 5 months ago
JSON representation

ETL pipeline that extracts data from S3, stages them in Redshift, and transforms data into a set of dimensional tables for analytics

• Host: GitHub
• URL: https://github.com/pylena/sparkifydatawarehouse
• Owner: pylena
• Created: 2025-06-14T23:08:00.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2025-06-16T13:48:28.000Z (about 1 year ago)
• Last Synced: 2025-06-23T14:48:00.331Z (about 1 year ago)
• Topics: aws-redshift, aws-s3, etl, sql
• Language: Python
• Homepage:
• Size: 15.6 KB
• Stars: 0
• Watchers: 0
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Sparkify Data Warehouse 

The goal of this project is to build an ETL pipeline that extracts data from S3, stages it in Amazon Redshift, and transforms it into a set of dimensional tables optimized for analytical queries. Sparkify, a music streaming startup, needs to analyze user listening behavior and understand their music preferences.

## Dataset

* Log Data: JSON logs of user activity (e.g., song plays).

* Song Data: JSON metadata about songs and artists.

## Project Architecture



## Sparkfy Star Schema Database



## How to Run the Project

- Step 1: Configure AWS Resources

  * Launch a Redshift cluster and IAM role with S3 read access.

  * Update the dwh.cfg file with your resources Info.

- Step 2: Set Up the Tables

  * Run : python create_tables.py

- Step 3: Run the ETL Pipeline

  * Run: python etl.py

### Repository File Structure

| File               | Description                                                                    |

| ------------------ | ------------------------------------------------------------------------------ |

| `create_tables.py` | Connects to Redshift and creates all necessary tables.                         |

| `etl.py`           | Runs the ETL pipeline: loads staging tables and inserts into analytics tables. |

| `sql_queries.py`   | Contains all SQL queries.                      |

| `dwh.cfg`          | Configuration file with AWS credentials, Redshift, and S3 paths.               |