{"id":19318511,"url":"https://github.com/federicoserini/dend-project-3-data-warehouse-aws","last_synced_at":"2025-06-12T17:08:27.582Z","repository":{"id":189106854,"uuid":"183090693","full_name":"FedericoSerini/DEND-Project-3-Data-Warehouse-AWS","owner":"FedericoSerini","description":"Project 3 - Data Engineering Nanodegree","archived":false,"fork":false,"pushed_at":"2019-04-26T15:54:36.000Z","size":64,"stargazers_count":4,"open_issues_count":0,"forks_count":10,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-22T17:41:13.332Z","etag":null,"topics":["aws","aws-redshift","aws-s3","data-engineering","udacity-nanodegree"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FedericoSerini.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-04-23T20:25:54.000Z","updated_at":"2021-12-07T16:00:45.000Z","dependencies_parsed_at":"2023-08-18T10:10:00.455Z","dependency_job_id":null,"html_url":"https://github.com/FedericoSerini/DEND-Project-3-Data-Warehouse-AWS","commit_stats":null,"previous_names":["federicoserini/dend-project-3-data-warehouse-aws"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/FedericoSerini/DEND-Project-3-Data-Warehouse-AWS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FedericoSerini%2FDEND-Project-3-Data-Warehouse-AWS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FedericoSerini%2FDEND-Project-3-Data-Warehouse-AWS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FedericoSerini%2FDEND-Project-3-Data-Warehouse-AWS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FedericoSerini%2FDEND-Project-3-Data-Warehouse-AWS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FedericoSerini","download_url":"https://codeload.github.com/FedericoSerini/DEND-Project-3-Data-Warehouse-AWS/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FedericoSerini%2FDEND-Project-3-Data-Warehouse-AWS/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259509428,"owners_count":22868834,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-redshift","aws-s3","data-engineering","udacity-nanodegree"],"created_at":"2024-11-10T01:18:54.420Z","updated_at":"2025-06-12T17:08:27.555Z","avatar_url":"https://github.com/FedericoSerini.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Project 3: Song Play Analysis With S3 and Redshift\r\n[![Project passed](https://img.shields.io/badge/project-passed-success.svg)](https://img.shields.io/badge/project-passed-success.svg)\r\n\r\n## Summary\r\n* [Preamble](#Preamble)\r\n* [Schema definition](#Schema-definition)\r\n* [ETL process](#ETL-process)\r\n* [How to run](#How-to-run)\r\n* [Project structure](#Project-structure)\r\n--------------------------------------------\r\n\r\n#### Preamble\r\n\r\nIn this project we are going to use two Amazon Web Services,\r\n[S3](https://aws.amazon.com/en/s3/) (Data storage) \u003cbr\u003e\r\nand [Redshift](https://aws.amazon.com/en/redshift/) (Data warehouse with ``columnar storage``)\r\n\r\nData sources are provided by two public ``S3 buckets``. One bucket contains \u003cbr\u003e\r\ninfo about songs and artists, the second has info concerning actions done \u003cbr\u003e by users (which song are listening, etc.. ). The objects  contained in both buckets \u003cbr\u003e are JSON files. The song bucket has all\r\nthe files under the same directory but \u003cbr\u003e the event ones don't,\r\nso we need a descriptor file (also a JSON) in order to extract \u003cbr\u003e data from the folders by path. We used a descriptor file because we don't have a common prefix on folders\r\n\r\nThe Redshift service is where data will be ingested and transformed, \u003cbr\u003e\r\nin fact though COPY command we will access to the JSON files inside \u003cbr\u003e\r\nthe buckets and copy their content on our staging tables\r\n\r\n--------------------------------------------\r\n\r\n#### Schema definition\r\nThis is the schema of the database\r\n\r\nHow to read the schema:\r\n* Blank bullets are used to identify the fields that can be null \u003cbr\u003e\r\n* Black bullets are used to identify the fields that can not be null \u003cbr\u003e\r\n* If the field is underlined means that is a primary key \u003cbr\u003e\r\n\r\n![schema](./img/SongPlayAnalysis.png)\r\n\r\nTo represent this context a ``Star schema`` has been used \u003cbr\u003e\r\n\r\nThe songplays table is the core of this schema, is it our fact table and \u003cbr\u003e\r\nit contains foreign keys to four tables;\r\n* start_time REFERENCES time(start_time)\r\n* user_id REFERENCES time(start_time)\r\n* song_id REFERENCES songs(song_id)\r\n* artist_id REFERENCES artists(artist_id)\r\n\r\nThere are also two staging tables; One for song dataset and one for \u003cbr\u003e\r\nand one for event dataset\r\n\r\n\r\n--------------------------------------------\r\n\r\n#### ETL process\r\n\r\nIn this project most of ETL is done with SQL (Python used just as bridge), transformation and data normalization is done by Query, check out the ``sql_queries`` python module\r\n\r\n--------------------------------------------\r\n\r\n#### How to run\r\nAlthough the data-sources are provided by two [``S3 buckets``](https://aws.amazon.com/en/s3/) the only thing you need for running the example is an [``AWS Redshift Cluster``](https://aws.amazon.com/en/redshift/) up and running\r\n\r\nAnd of course [Python](https://www.python.org/downloads/) \u003cbr\u003e\r\n\r\n\u003cb\u003e Notes: \u003c/b\u003e\r\n* In this example a Redshift ``dc2.large``  cluster with \u003cb\u003e 4 nodes \u003c/b\u003e has been created, with a cost of ``USD 0.25/h (on-demand option)`` per cluster\r\n* In this example we will use [``IAM role ``](https://docs.aws.amazon.com/en_us/IAM/latest/UserGuide/id_roles.html) authorization mechanism, the only policy attached to this IAM will be am [``AmazonS3ReadOnlyAccess``](https://aws.amazon.com/en/blogs/security/organize-your-permissions-by-using-separate-managed-policies/)\r\n\r\n\r\nAfter opening terminal session, set your filesystem on project root folder \u003cbr\u003e\r\nand  insert these commands in order to run the demo: \u003cbr\u003e\u003cbr\u003e\r\n\u003cI\u003e This will create our tables, this must be runned first \u003c/I\u003e \u003cbr\u003e\r\n`` python create_tables.py`` \u003cbr\u003e\r\n\r\n\u003cI\u003e And this will execute our ETL process \u003c/I\u003e \u003cbr\u003e\r\n`` python etl.py`` \u003cbr\u003e\r\n\r\n--------------------------------------------\r\n\r\n#### Project structure\r\nThis is the project structure, if the bullet contains ``/`` \u003cbr\u003e\r\nmeans that the resource is a folder:\r\n\r\n* \u003cb\u003e /img \u003c/b\u003e - Simply a folder with images that are used in this ``md``\r\n* \u003cb\u003e create_tables.py \u003c/b\u003e - This script will drop old tables (if exist) ad re-create new tables\r\n* \u003cb\u003e etl.py \u003c/b\u003e - This script executes the queries that extract JSON data\r\nfrom the S3 bucket and ingest them to Redshift\r\n* \u003cb\u003e sql_queries.py \u003c/b\u003e - This file contains variables with SQL statement in String formats,  partitioned by CREATE, DROP, COPY and INSERT statements\r\n* \u003cb\u003e dhw.cfg \u003c/b\u003e - Configuration file used that contains info about Redshift, IAM and S3\r\n\r\n--------------------------------------------\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffedericoserini%2Fdend-project-3-data-warehouse-aws","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffedericoserini%2Fdend-project-3-data-warehouse-aws","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffedericoserini%2Fdend-project-3-data-warehouse-aws/lists"}