{"id":18811674,"url":"https://github.com/gares95/datalake-spark","last_synced_at":"2026-01-11T13:30:19.546Z","repository":{"id":218120525,"uuid":"302151024","full_name":"Gares95/DataLake-Spark","owner":"Gares95","description":"This repository consist of a project to build an ETL pipeline for a data lake hosted on S3 using Spark. This project is based on Udacity's template.","archived":false,"fork":false,"pushed_at":"2020-10-14T15:30:36.000Z","size":45,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-12-30T00:13:15.480Z","etag":null,"topics":["data-lake","datalake","udacity","udacity-data-engineer-nanodegree"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Gares95.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-10-07T20:20:54.000Z","updated_at":"2020-10-14T15:30:39.000Z","dependencies_parsed_at":"2024-01-19T21:31:17.567Z","dependency_job_id":"ee14e76a-f5d6-4959-8b34-4f9ac09cd326","html_url":"https://github.com/Gares95/DataLake-Spark","commit_stats":null,"previous_names":["gares95/datalake-spark"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gares95%2FDataLake-Spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gares95%2FDataLake-Spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gares95%2FDataLake-Spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gares95%2FDataLake-Spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Gares95","download_url":"https://codeload.github.com/Gares95/DataLake-Spark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239748248,"owners_count":19690232,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-lake","datalake","udacity","udacity-data-engineer-nanodegree"],"created_at":"2024-11-07T23:27:16.054Z","updated_at":"2026-01-11T13:30:19.511Z","avatar_url":"https://github.com/Gares95.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Lake with Spark\n***\nThis project will include the files neccessary to build an ETL pipeline for a *Data Lake* hosted on S3. The objective of this project is to succesfully extract data from an S3 bucket and to process it with Spark to load them back into another S3 bucket as a set of dimensional tables. This Spark process will be deployed on a cluster using *AWS*.\n\nThis repository simulates the creation of an ETL pipeline for a music streaming startup whose data resides in S3 and want to move their data warehouse to a data lake.\n\nThe data currently is in an S3 bucket in directories which contains their log data and song data in JSON files. The objective of the main program of this repository is to process the data and create a star schema optimized for queries for the song play analysis.\n\n# Data Files\n***\nThe datasets used for this project that reside in S3 are:\n- Song data: s3://udacity-dend/song_data\n- Log data: s3://udacity-dend/log_data\n\n### Song Dataset\nThe first dataset is a subset of real data from the Million Song Dataset. Each file is in JSON format and contains metadata about a song and the artist of that song. The files are partitioned by the first three letters of each song's track ID. Here is an example of a filepath: _\"song_data/A/B/C/TRABCEI128F424C983.json\"_\nAnd here is an example of one of the json files: _{\"num_songs\": 1, \"artist_id\": \"ARJIE2Y1187B994AB7\", \"artist_latitude\": null, \"artist_longitude\": null, \"artist_location\": \"\", \"artist_name\": \"Line Renaud\", \"song_id\": \"SOUPIRU12A6D4FA1E1\", \"title\": \"Der Kleine Dompfaff\", \"duration\": 152.92036, \"year\": 0}_\n\n### Log Dataset\nThe second dataset consists of log files in JSON format generated by this _event simulator_ based on the songs in the dataset above. These simulate activity logs from a music streaming app based on specified configurations.\nHere is an example of a filepath: _\"log_data/2018/11/2018-11-12-events.json\"_\nAnd here is an example of a json file for these events: _{\"artist\": \"None\", \"auth\": \"Logged In\", \"gender\": \"F\", \"itemInSession\": 0, \"lastName\": \"Williams\", \"length\": \"227.15873\", \"level\": \"free\", \"location\": \"Klamath Falls OR\", \"method\": \"GET\", \"page\": \"Home\", \"registration\": \"1.541078e-12\", \"sessionId\": \"438\", \"Song\": \"None\", \"status\": \"200\", \"ts\": \"15465488945234\", \"userAgent\": \"Mozilla/5.0(WindowsNT,6.1;WOW641)\", \"userId\": \"53\"}_\n\n### The star schema tables\nThe star schema that is going to be created using this program will have the next structure:\n\n- _Fact table_:\n1. songplays [songplay_id, start_time, user_id, level, song_id, artist_id, session_id, location, user_agent]\n\n- _Dimension tables_:\n2. users [user_id, first_name, last_name, gender, level]\n3. songs [song_id, title, artist_id, year, duration]\n4. artist [artist_id, name, location, lattitude, longitude]\n5. time [start_time, hour, day, week, month, year, weekday]\n\n![alt text](https://raw.githubusercontent.com/Gares95/DataLake-Spark/master/Star%20Schema.PNG)\n\n# Program files\n***\n## df.cfg\n\nThis file contains the AWS credentials to access the S3 buckets. \nHere you will have to introduce your AWS key and secret access key:\n\n[AWS]\n\nAWS_ACCESS_KEY_ID=\u003cyour AWS access key\u003e\n    \nAWS_SECRET_ACCESS_KEY=\u003cyour AWS secret access key\u003e\n\n## etl.py\n\nWith this file we will process all files from the S3 buckets and create the star schema (with all the tables mentioned above) and will introduce them into a new S3 which will act as our Data Lake. \n\n## README.md\n\nThis file provides the descrpition of the program and process of the etl.\n\n### Credits\nUdacity provided the template and the guidelines to start this project.\nThe completion of this was made by Guillermo Garcia and the review of the program and the verification that the project followed the proper procedures was also made by my mentor from udacity.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgares95%2Fdatalake-spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgares95%2Fdatalake-spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgares95%2Fdatalake-spark/lists"}