https://github.com/eljandoubi/aws-data-warehouse
Build an ETL pipeline for a database hosted on AWS Redshift.
https://github.com/eljandoubi/aws-data-warehouse
aws-redshift etl-pipeline postgresql
Last synced: about 2 months ago
JSON representation
Build an ETL pipeline for a database hosted on AWS Redshift.
- Host: GitHub
- URL: https://github.com/eljandoubi/aws-data-warehouse
- Owner: eljandoubi
- License: mit
- Created: 2024-03-11T11:10:27.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-03-11T11:44:21.000Z (about 1 year ago)
- Last Synced: 2025-01-24T22:35:38.461Z (3 months ago)
- Topics: aws-redshift, etl-pipeline, postgresql
- Language: Python
- Homepage:
- Size: 11.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Data-Warehouse
## Overview
This project involves building an ETL pipeline for a music streaming startup. The datasets provided include song data and log data, both stored in Amazon S3 buckets. The song dataset is a subset of the Million Song Dataset, containing JSON files with metadata about songs and their artists. The log dataset consists of JSON files generated by an event simulator, simulating app activity logs from the music streaming app.
## Dataset Description
### Song Dataset
The song dataset contains JSON files partitioned by the first three letters of each song's track ID. Each file provides metadata about a song and its corresponding artist.
### Log Dataset
The log dataset comprises JSON files partitioned by year and month, simulating app activity logs from the music streaming app. These logs include user interactions such as song plays.
## Schema
The project employs a star schema with one fact table and multiple dimension tables:
### Fact Table
- `songplays`: Records in log data associated with song plays (i.e., records with the page NextSong). Columns include `songplay_id`, `start_time`, `user_id`, `level`, `song_id`, `artist_id`, `session_id`, `location`, and `user_agent`.
### Dimension Tables
1. `users`: Users in the app. Columns include `user_id`, `first_name`, `last_name`, `gender`, and `level`.
2. `songs`: Songs in the music database. Columns include `song_id`, `title`, `artist_id`, `year`, and `duration`.
3. `artists`: Artists in the music database. Columns include `artist_id`, `name`, `location`, `latitude`, and `longitude`.
4. `time`: Timestamps of records in songplays broken down into specific units. Columns include `start_time`, `hour`, `day`, `week`, `month`, `year`, and `weekday`.
## Project Structure
### Code Files
1. `create_table.py`: Creates fact and dimension tables as well as staging tables for the star schema in Redshift.
2. `etl.py`: Loads data from S3 into staging tables on Redshift and then processes that data into analytics tables on Redshift.
3. `sql_queries.py`: Defines SQL statements imported into other files.
4. `test.ipynb`: Creates a Redshift cluster, IAM role, and verifies the result after running `etl.py`.
5. `README.md`: Documentation providing insights into the project process and decisions for the ETL pipeline.
## Project Steps
### Table Schemas Design
1. Design schemas for fact and dimension tables.
2. Write SQL CREATE statements for each table in `sql_queries.py`.
3. Complete logic in `create_tables.py` to connect to the database and create tables.
4. Write SQL DROP statements in `create_tables.py` to drop tables if they already exist, enabling resetting of the database for pipeline testing.
5. Launch a Redshift cluster and create an IAM role with S3 read access.
6. Add Redshift database and IAM role info to `dwh.cfg`.
7. Test by running `create_tables.py` and checking table schemas in the Redshift database.
### ETL Pipeline Implementation
1. Implement logic in `etl.py` to load data from S3 to staging tables on Redshift.
2. Implement logic in `etl.py` to load data from staging tables to analytics tables on Redshift.
3. Test by running `etl.py` after running `create_tables.py` and running analytic queries on Redshift database to compare results with expected results.
4. Delete the Redshift cluster when finished.
## Running Scripts
1. Set environment variables `KEY` and `SECRET`.
2. Choose `DB/DB_PASSWORD` in `dhw.cfg`.
3. Create IAM role, Redshift cluster, connect to S3 bucket, and configure TCP connectivity.
4. Drop and recreate tables:
```bash
$ python create_tables.py
```
5. Run the ETL pipeline:
```bash
$ python etl.py
```
6. Delete IAM role and Redshift cluster.