{"id":15157879,"url":"https://github.com/maxinexiong/cloud-data-warehousing-with-aws-redshift","last_synced_at":"2026-02-27T17:07:35.243Z","repository":{"id":256213496,"uuid":"854597581","full_name":"MaxineXiong/Cloud-Data-Warehousing-with-AWS-Redshift","owner":"MaxineXiong","description":"This project builds a cloud-based ETL pipeline for Sparkify to move data to a cloud data warehouse. It extracts song and user activity data from AWS S3, stages it in Redshift, and transforms it into a star-schema data model with fact and dimension tables, enabling efficient querying to answer business questions.","archived":false,"fork":false,"pushed_at":"2024-09-09T15:52:49.000Z","size":21,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-31T02:04:38.028Z","etag":null,"topics":["aws-boto3","aws-redshift","aws-s3","cloud-data-warehouse","data-engineering","data-warehouse","data-warehousing","dimensional-model","dimensional-modeling","etl","etl-pipeline","extract-transform-load","infrastructure-as-code","postgresql","postgresql-database","redshift-cluster"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MaxineXiong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-09T13:07:49.000Z","updated_at":"2024-12-31T07:24:00.000Z","dependencies_parsed_at":"2024-09-09T17:05:26.471Z","dependency_job_id":null,"html_url":"https://github.com/MaxineXiong/Cloud-Data-Warehousing-with-AWS-Redshift","commit_stats":null,"previous_names":["maxinexiong/cloud-data-warehousing-with-aws-redshift"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxineXiong%2FCloud-Data-Warehousing-with-AWS-Redshift","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxineXiong%2FCloud-Data-Warehousing-with-AWS-Redshift/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxineXiong%2FCloud-Data-Warehousing-with-AWS-Redshift/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxineXiong%2FCloud-Data-Warehousing-with-AWS-Redshift/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MaxineXiong","download_url":"https://codeload.github.com/MaxineXiong/Cloud-Data-Warehousing-with-AWS-Redshift/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237982335,"owners_count":19397246,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-boto3","aws-redshift","aws-s3","cloud-data-warehouse","data-engineering","data-warehouse","data-warehousing","dimensional-model","dimensional-modeling","etl","etl-pipeline","extract-transform-load","infrastructure-as-code","postgresql","postgresql-database","redshift-cluster"],"created_at":"2024-09-26T20:20:18.041Z","updated_at":"2026-02-27T17:07:35.208Z","avatar_url":"https://github.com/MaxineXiong.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cloud Data Warehousing with AWS Redshift\n\n[![GitHub](https://badgen.net/badge/icon/GitHub?icon=github\u0026color=black\u0026label)](https://github.com/MaxineXiong)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Made with Python](https://img.shields.io/badge/Python-\u003e=3.6-blue?logo=python\u0026logoColor=white)](https://www.python.org)\n[![Amazon Redshift](https://img.shields.io/badge/Amazon_Redshift-8C4FFF?logo=Amazon+Redshift\u0026logoColor=white)](https://aws.amazon.com/redshift/)\n[![Amazon S3](https://img.shields.io/badge/Amazon_S3-569A31?logo=Amazon+S3\u0026logoColor=white)](https://aws.amazon.com/s3/)\n\n\u003cbr/\u003e\n\n## Project Description\n\nThis project is aimed at helping *Sparkify*, a music streaming startup, move its data and processes to the cloud. The goal is to build an **ETL pipeline** that extracts data from AWS S3, loads it into staging tables in a database hosted on Amazon Redshift, and transforms it into a star-schema-based data model to support analytics. This project handles large volumes of song metadata and user activity logs stored in S3, bringing them into a Redshift cluster for analysis by *Sparkify*'s data team. The final output is a dimensional model with fact and dimension tables in a star schema in Redshift that allows for efficient querying to answer business questions such as popular songs, user listening patterns, and peak activity times.\n\nThe ETL pipeline involves the following steps:\n\n1. **Extracting** song metadata and user activity logs from **S3**.\n2. **Loading** the data into **staging tables** in a **Redshift cluster**.\n3. **Transforming** the staging data into a **fact and dimension tables** following a star schema.\n\nThe image below demonstrates the ETL process of moving data from S3 to Redshift:\n\n![overall-process](https://github.com/user-attachments/assets/5c89b88f-d416-4af9-b85f-ed160c4efecc)\n\nThe project implements a scalable cloud solution for *Sparkify*'s analytics team to gain insights from their user and song data.\n\n\u003cbr/\u003e\n\n## Project Data\n\nThe project relies on two datasets stored in AWS S3:\n\n- **Song Data**: Metadata about songs and artists, stored in JSON format in the path `s3://udacity-dend/song_data`.\n- **Log Data**: User activity logs generated by the *Sparkify* app, stored in JSON format in the path `s3://udacity-dend/log_data`.\n\nAdditionally, the JSON metadata file `s3://udacity-dend/log_json_path.json` specifies how the log data is structured, enabling proper parsing during data loading into staging tables.\n\nThe song dataset consists of JSON files partitioned by the first three letters of each song’s track ID. For example, here are file paths to two files in the song dataset:\n\n```\nsong_data/A/B/C/TRABCEI128F424C983.json\nsong_data/A/A/B/TRAABJL12903CDCF1A.json\n```\n\nBelow is is an example of what the single song file, TRAABJL12903CDCF1A.json, looks like:\n\n```\n{\"num_songs\": 1, \"artist_id\": \"ARJIE2Y1187B994AB7\", \"artist_latitude\": null, \"artist_longitude\": null, \"artist_location\": \"\", \"artist_name\": \"Line Renaud\", \"song_id\": \"SOUPIRU12A6D4FA1E1\", \"title\": \"Der Kleine Dompfaff\", \"duration\": 152.92036, \"year\": 0}\n```\n\nThe log dataset comprises log files in JSON format that are partitioned by year and month.  For example, here are file paths to two files in the dataset.\n\n```\nlog_data/2018/11/2018-11-12-events.json\nlog_data/2018/11/2018-11-13-events.json\n```\n\nAnd this image shows what the data in the log file, 2018-11-12-events.json, looks like: \n\n![log-data](https://github.com/user-attachments/assets/e127ca71-f26a-4d47-a055-731ce268c524)\n\nThese datasets are processed and transformed into a **star-schema data model** in Redshift, as shown in the Entity Relationship Diagram (ERD) below, consisting of fact and dimension tables to facilitate analysis.\n\n![ERD](https://github.com/user-attachments/assets/e590483b-9d90-40d9-8a7b-0baafabe4565)\n\n\u003cbr/\u003e\n\n## Repository Structure\n\nThe repository is structured as follows:\n\n```\nSparkify-ETL-Pipeline/\n├── 0_launch_Redshift_cluster.ipynb     \n├── 1_create_tables.py                  \n├── 2_etl.py                            \n├── 3_test_dimensional_model.ipynb      \n├── sql_queries.py                      \n├── dwh.cfg                             \n├── .gitignore                          \n├── README.md                           \n└── LICENSE\n```\n\n- **0_launch_Redshift_cluster.ipynb**: Jupyter notebook that sets up and configures an Amazon Redshift cluster used in the ETL process.\n- **1_create_tables.py**: Python script responsible for creating the staging, fact, and dimension tables in the Redshift database.\n- **2_etl.py**: Python script that extracts data from S3, loads it into staging tables on Redshift, and then transforms it into the target fact and dimension tables.\n- **3_test_dimensional_model.ipynb**: Jupyter notebook used for testing and verifying the data loading process, validating the schema, and running analytic queries.\n- **sql_queries.py**: This file contains all the SQL queries required for creating tables and performing the ETL operations.\n- **dwh.cfg**: Configuration file that stores Redshift cluster, database, and AWS credentials.\n- **.gitignore**: Specifies files and directories for Git to ignore, helping to manage sensitive data and unnecessary files.\n- **README.md**: Provides an overview and instructions for this repository.\n- **LICENSE**: The license file for the project.\n\n\u003cbr/\u003e\n\n## Usage\n\n1. **Launch Redshift Cluster**: First, configure and launch the Redshift cluster using the `0_launch_Redshift_cluster.ipynb` notebook. This step sets up the target database on Redshift.\n2. **Create Tables**: Run `1_create_tables.py` to create the staging, fact, and dimension tables in Redshift. This script can be rerun to reset the database if needed.\n3. **Run ETL Pipeline**: Execute `2_etl.py` to load data from S3 into the staging tables in Redshift using the `COPY` command, and then insert the data into the fact and dimension tables using the staging tables.\n4. **Test Dimensional Model**: Use `3_test_dimensional_model.ipynb` to validate the schema, check row counts, and run analytic queries to ensure that the model is ready for analytical workloads.\n5. **Tear Down Cluster**: After completing the project, return to the final step in `1_create_tables.py` to delete the Redshift cluster and clean up associated resources.\n\n\u003cbr/\u003e\n\n## Contribution\n\nContributions to this project are welcome. If you'd like to improve the ETL pipeline or add additional functionalities, please fork the repository, create a new branch, and submit a pull request. Ensure that your code follows best practices and is well documented.\n\n\u003cbr/\u003e\n\n## License\n\nThis project is licensed under the [MIT License](https://choosealicense.com/licenses/mit/). Feel free to use, modify, and distribute the application in accordance with the terms of the license.\n\n\u003cbr/\u003e\n\n## Acknowledgement\n\nSpecial thanks to [Udacity](https://www.udacity.com/) for providing the datasets and project specifications. The song and log data used in this project come from the [Million Song Dataset](http://millionsongdataset.com/) and [event simulator](https://github.com/Interana/eventsim) logs provided by Udacity.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxinexiong%2Fcloud-data-warehousing-with-aws-redshift","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaxinexiong%2Fcloud-data-warehousing-with-aws-redshift","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxinexiong%2Fcloud-data-warehousing-with-aws-redshift/lists"}