{"id":18266101,"url":"https://github.com/alexandrabaturina/redshift-data-warehouse","last_synced_at":"2026-07-21T21:34:28.958Z","repository":{"id":119258792,"uuid":"529340787","full_name":"alexandrabaturina/redshift-data-warehouse","owner":"alexandrabaturina","description":"Python ETL pipeline to load data from Amazon S3 to Redshift analytics tables","archived":false,"fork":false,"pushed_at":"2022-08-29T23:56:04.000Z","size":27,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-08T07:00:21.275Z","etag":null,"topics":["amazon-s3","python-etl-pipeline","redshift-cluster"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alexandrabaturina.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-26T17:02:53.000Z","updated_at":"2022-08-29T19:15:08.000Z","dependencies_parsed_at":null,"dependency_job_id":"0cc9b675-cc12-4135-a826-fa9852a5e4a4","html_url":"https://github.com/alexandrabaturina/redshift-data-warehouse","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/alexandrabaturina/redshift-data-warehouse","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexandrabaturina%2Fredshift-data-warehouse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexandrabaturina%2Fredshift-data-warehouse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexandrabaturina%2Fredshift-data-warehouse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexandrabaturina%2Fredshift-data-warehouse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alexandrabaturina","download_url":"https://codeload.github.com/alexandrabaturina/redshift-data-warehouse/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexandrabaturina%2Fredshift-data-warehouse/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35738448,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-21T02:00:06.728Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-s3","python-etl-pipeline","redshift-cluster"],"created_at":"2024-11-05T11:21:46.176Z","updated_at":"2026-07-21T21:34:28.942Z","avatar_url":"https://github.com/alexandrabaturina.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Engineering ND #3: Data Warehouse\r\n## Overview\r\n**Data Warehouse** is the third project of Udacity [Data Engineering Nanodegree](https://d20vrrgs8k4bvw.cloudfront.net/documents/en-US/Data+Engineering+Nanodegree+Program+Syllabus.pdf). The goal of the project is to work with data warehouses and [AWS](https://aws.amazon.com/). It requires to build an [ETL](https://en.wikipedia.org/wiki/Extract,_transform,_load) pipeline that:\r\n* Extracts music streaming data from [Amazon S3](https://aws.amazon.com/s3/) buckets\r\n* Stages extracted data in [Redshift](https://aws.amazon.com/redshift/)\r\n* Transforms data into a set of dimensional tables for analytics team\r\n## Repo Contents\r\nThe repo contains the following files:\r\n* ```create_tables.py```: drops existing tables and creates new ones.\r\n* ```etl.py```: loads data from S3 to Redshift staging tables, and inserts extracted data into analytics tables.\r\n* ```sql_queries.py```: contains all SQL queries.\r\n## Database\r\n### Database Purpose\r\nA music streaming startup, *Sparkify*, wants to move their processes and data onto the cloud. They need a set of dimensional tables for their analytics team to find insights in what songs their users are listening to. \r\n### Project Datasets\r\n*Sparkify* data resides in S3, in a directory of JSON logs on user activity on the app, as well as a directory with JSON metadata on the songs in their app.\r\n#### Song Dataset ```s3://udacity-dend/song_data```\r\nThe first dataset is a subset of real data from the [Million Song Dataset](http://millionsongdataset.com/). Each file is in JSON format and contains metadata about a song and the artist of that song. \r\n\r\nBelow is an example of what a single song file ```data/song_data/A/A/B/TRAABJV128F1460C49.json``` looks like.\r\n```\r\n{\r\n  \"num_songs\": 1, \r\n  \"artist_id\": \"ARIK43K1187B9AE54C\", \r\n  \"artist_latitude\": null, \r\n  \"artist_longitude\": null, \r\n  \"artist_location\": \"Beverly Hills, CA\", \r\n  \"artist_name\": \"Lionel Richie\", \r\n  \"song_id\": \"SOBONFF12A6D4F84D8\", \r\n  \"title\": \"Tonight Will Be Alright\", \r\n  \"duration\": 307.3824, \r\n  \"year\": 1986\r\n}\r\n```\r\n#### Log Dataset ```s3://udacity-dend/log_data```\r\nLog data json path: ```s3://udacity-dend/log_json_path.json```.\r\n\r\nThis dataset consists of log files in JSON format generated by [eventsim](https://github.com/Interana/eventsim) event simulator based on the songs in the song dataset. These simulate activity logs from a music streaming app based on specified configurations.\r\n\r\nBelow is an example of what a single line of a single file ```data/log_data/2018/11/2018-11-09-events.json``` looks like.\r\n```\r\n{\r\n  \"artist\":\"Beastie Boys\",\r\n  \"auth\":\"Logged In\",\r\n  \"firstName\":\"Harper\",\r\n  \"gender\":\"M\",\r\n  \"itemInSession\":2,\r\n  \"lastName\":\"Barrett\",\r\n  \"length\":161.56689,\r\n  \"level\":\"paid\",\r\n  \"location\":\"New York-Newark-Jersey City, NY-NJ-PA\",\r\n  \"method\":\"PUT\",\r\n  \"page\":\"NextSong\",\r\n  \"registration\":1540685364796.0,\r\n  \"sessionId\":275,\r\n  \"song\":\"Lighten Up\",\r\n  \"status\":200,\r\n  \"ts\":1541722186796,\r\n  \"userAgent\":\"\\\"Mozilla\\/5.0 (Windows NT 6.3; WOW64) AppleWebKit\\/537.36 (KHTML, like Gecko) Chrome\\/36.0.1985.143 Safari\\/537.36\\\"\",\r\n  \"userId\":\"42\"\r\n}\r\n```\r\n## Database Design\r\n### Step 1: Staging Tables\r\nOn step 1, data is loaded from S3 into two staging tables.\r\n#### ```staging_events``` table\r\nThe ```staging_event``` table has the following fields:\r\n* artist: VARCHAR\r\n* auth: VARCHAR\r\n* first_name: VARCHAR\r\n* gender: CHAR(1)\r\n* item_in_session: INT\r\n* last_name: VARCHAR\r\n* length: FLOAT\r\n* level: VARCHAR\r\n* location: VARCHAR\r\n* method: VARCHAR\r\n* page: VARCHAR\r\n* registration: BIGINT\r\n* session_id: INT\r\n* song: VARCHAR\r\n* status: INT\r\n* ts: TIMESTAMP\r\n* user_agent: VARCHAR\r\n* user_id: INT\r\n#### ```staging_songs``` table\r\nThe ```staging_songs``` table has the following fields:\r\n* num_songs: INT\r\n* artist_id: CHAR(18)\r\n* artist_latitude: FLOAT\r\n* artist_longitude: FLOAT\r\n* artist_location: VARCHAR\r\n* artist_name: VARCHAR\r\n* song_id: CHAR(18)\r\n* title: VARCHAR\r\n* duration: FLOAT\r\n* year: INT\r\n### Step 2: Analytics Tables\r\nThe database for analytics team contains the following tables:\r\n* Fact table\r\n    * **songplays**: records in log data associated with song plays, i.e. records with page NextSong\r\n* Dimension tables\r\n    * **users**: users in the app\r\n    * **songs**: songs in music database\r\n    * **artists**: artists in music database\r\n    * **time**: timestamps of records in songplays broken down into specific units\r\n    \r\nThe database schema is shown below. Primary and foreign keys are marked as ```PK``` and ```FK```, respectively.\r\n![image](https://user-images.githubusercontent.com/53233637/187318753-c965452d-65ac-47b6-bcba-c34c623f989e.png)\r\n\r\n## Getting Started\r\n### Prerequisites\r\nTo run ETL, it's required to launch a Redshift cluster and create an IAM role that has read access to S3. The data should be saved in ```dwh.cfg``` file. Below is an example of what ```dwh.cfg``` looks like.\r\n```\r\n[CLUSTER]\r\nHOST=YOUR_HOST\r\nDB_NAME=YOUR_DATABASE_NAME\r\nDB_USER=YOUR_DATABASE_USER\r\nDB_PASSWORD=YOUR_DATABASE_PASSWORK\r\nDB_PORT=YOUR_DATABASE_PORT\r\n\r\n[IAM_ROLE]\r\nARN=YOUR_IAM_ROLE\r\n\r\n[S3]\r\nLOG_DATA='s3://udacity-dend/log_data'\r\nLOG_JSONPATH='s3://udacity-dend/log_json_path.json'\r\nSONG_DATA='s3://udacity-dend/song_data'\r\n```\r\n### Running ETL Locally\r\nTo run ETL locally,\r\n1. Clone this repo.\r\n2. ```cd``` into project directory.\r\n3. Put ```dwh.cfg``` into project directory.\r\n4. Run ```create_tables.py``` to reset tables.\r\n```\r\nroot@1a2dc16602ee:/home/workspace# python create_tables.py \r\nTables are dropped.\r\nTables are created.\r\n```\r\n\u003e Remember to run create_tables.py every time before running etl.py to reset tables.\r\n5. Run ```etl.py```. Query execution progress is displayed in terminal.\r\n```\r\nroot@1a2dc16602ee:/home/workspace# python etl.py\r\nLoading staging tables...\r\nQuery 1 of 2 executed.\r\nQuery 2 of 2 executed.\r\nInsert data into analytics tables...\r\nQuery 1 of 5 executed.\r\nQuery 2 of 5 executed.\r\nQuery 3 of 5 executed.\r\nQuery 4 of 5 executed.\r\nQuery 5 of 5 executed.\r\n```\r\n## Authors\r\n[Alexandra Baturina](https://www.linkedin.com/in/alexandrabaturina/)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexandrabaturina%2Fredshift-data-warehouse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falexandrabaturina%2Fredshift-data-warehouse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexandrabaturina%2Fredshift-data-warehouse/lists"}