{"id":18574220,"url":"https://github.com/vermicida/data-lake","last_synced_at":"2026-02-18T05:34:22.212Z","repository":{"id":94734094,"uuid":"201413118","full_name":"vermicida/data-lake","owner":"vermicida","description":"Data Lake, the code corresponding the project #4 of the Udacity's Data Engineer Nanodegree Program","archived":false,"fork":false,"pushed_at":"2024-10-15T18:44:59.000Z","size":893,"stargazers_count":1,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-19T21:38:52.854Z","etag":null,"topics":["aws-s3","data-engineering","data-lake","etl-pipeline","python","spark"],"latest_commit_sha":null,"homepage":"https://eu.udacity.com/course/data-engineer-nanodegree--nd027","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vermicida.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-09T07:15:15.000Z","updated_at":"2020-05-26T17:00:35.000Z","dependencies_parsed_at":"2024-12-26T14:44:36.656Z","dependency_job_id":"87907ebf-c768-4d41-9a61-58e7e7924061","html_url":"https://github.com/vermicida/data-lake","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vermicida/data-lake","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vermicida%2Fdata-lake","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vermicida%2Fdata-lake/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vermicida%2Fdata-lake/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vermicida%2Fdata-lake/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vermicida","download_url":"https://codeload.github.com/vermicida/data-lake/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vermicida%2Fdata-lake/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29569856,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-18T04:18:28.490Z","status":"ssl_error","status_checked_at":"2026-02-18T04:13:49.018Z","response_time":162,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-s3","data-engineering","data-lake","etl-pipeline","python","spark"],"created_at":"2024-11-06T23:14:25.078Z","updated_at":"2026-02-18T05:34:17.203Z","avatar_url":"https://github.com/vermicida.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Project: Data Lake\n\nA music streaming startup, Sparkify, has grown their user base and song database even more and want to move their data warehouse to a data lake. Their data resides in S3, in a directory of JSON logs on user activity on the app, as well as a directory with JSON metadata on the songs in their app.\n\nAs their data engineer, you are tasked with building an ETL pipeline that extracts their data from S3, processes them using Spark, and loads the data back into S3 as a set of dimensional tables. This will allow their analytics team to continue finding insights in what songs their users are listening to.\n\nYou'll be able to test your database and ETL pipeline by running queries given to you by the analytics team from Sparkify and compare your results with their expected results.\n\n## Table of contents\n\n- [Structure](#structure)\n- [Requirements](#requirements)\n  - [Preparing AWS](#preparing-aws)\n    - [Creating a bucket](#creating-a-bucket)\n    - [Creating an administrator user](#creating-an-administrator)\n  - [Cloning the repository](#cloning-the-repository)\n  - [Creating the virtual environment](#creating-the-virtual-environment)\n- [How to use](#how-to-use)\n  - [Running the ETL](#running-the-etl)\n\n---\n\n## Structure\u003ca name=\"structure\"\u003e\u003c/a\u003e\n\nThis tree shows the repository structure. Only the project's main files are described.\n\n```\n.\n├── images\n│   ├── iam-create-user-01.png\n│   ├── iam-create-user-02.png\n│   ├── iam-create-user-03.png\n│   ├── iam-create-user-04.png\n│   ├── iam-create-user-05.png\n│   ├── s3-artists-parquet.png\n│   ├── s3-create-bucket-01.png\n│   └── s3-create-bucket-02.png\n│   └── s3-create-bucket-03.png\n├── src\n│   ├── pipelines\n│   │   ├── __init__.py\n│   │   ├── artists.py             # Artists table pipeline\n│   │   ├── base.py                # Pipeline common code\n│   │   ├── cache.py               # Dataframes cache\n│   │   ├── songplays.py           # Songplays table pipeline\n│   │   ├── songs.py               # Songs table pipeline\n│   │   ├── time.py                # Time table pipeline\n│   │   └── users.py               # Users table pipeline\n│   ├── config.py                  # Application config manager\n│   ├── etl.py                     # ETL pipeline script\n│   └── sparkify.cfg               # Application config file\n├── .editorconfig\n├── .gitignore\n├── README.md\n└── requirements.txt\n```\n\n---\n\n## Requirements\u003ca name=\"requirements\"\u003e\u003c/a\u003e\n\nIt is assumed that the tools below are properly installed locally:\n\n- [Python](https://www.python.org/) is a programming language that lets you work quickly and integrate systems more effectively. Versión 3.x is required.\n- [pip](https://pip.pypa.io/en/stable/) is the package installer for Python.\n- [Virtualenv](https://virtualenv.pypa.io/en/latest/) is a tool to create isolated Python environments.\n- [Amazon Web Services](https://aws.amazon.com/) is the world's most comprehensive and broadly adopted cloud platform, offering over 165 fully featured services from data centers globally.\n\n### Preparing AWS\u003ca name=\"preparing-aws\"\u003e\u003c/a\u003e\n\nBefore running the pipelines that will load the source data into our data lake, we must tune up some things in our AWS account.\n\n#### Creating a bucket\u003ca name=\"creating-a-bucket\"\u003e\u003c/a\u003e\n\nWe are using [Amazon S3](https://docs.aws.amazon.com/s3/index.html) as data lake, so we must first create the bucket in which the data will be stored.\n\nOpen the AWS console, log into your account and go to [S3](https://s3.console.aws.amazon.com/s3/home). Choose _Buckets_ from the left side menu, and then click the button _Create bucket_.\n\n\u003cimg src=\"images/s3-create-bucket-01.png\" width=\"707\" alt=\"Create bucket\"\u003e\n\nThen choose a name for the bucket; remember to check the naming rules [here](https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html#bucketnamingrules). For the region we will use _US West (Oregon)_, the same as the source data bucket.\n\n\u003cimg src=\"images/s3-create-bucket-02.png\" width=\"460\" alt=\"Bucket name and region\"\u003e\n\nFor other options and permission we will keep the default values. Just follow the steps and confirm the bucket creation.\n\nOnce the bucket is created, select it from the main list. A popup box will appear from the right, showing some interesting properties regarding the selected bucket. Click the button _Copy the Bucket ARN_, because we will use it in a couple of minutes.\n\n\u003cimg src=\"images/s3-create-bucket-03.png\" width=\"773\" alt=\"Bucket ARN\"\u003e\n\n#### Creating an administrator user\u003ca name=\"creating-an-administrator\"\u003e\u003c/a\u003e\n\nNow go to [IAM](https://console.aws.amazon.com/iam/home#/users). Choose _User_ from the left side menu, and then click the button _Add user_.\n\n\u003cimg src=\"images/iam-create-user-01.png\" width=\"527\" alt=\"Add user\"\u003e\n\nFill in the _User name_ box, and also mark the _Programmatic access_ checkbox.\n\n\u003cimg src=\"images/iam-create-user-02.png\" width=\"646\" alt=\"User details\"\u003e\n\nDo not add any permission to the user; just go to the end of the wizard, when the credentials are shown.\n\n\u003cimg src=\"images/iam-create-user-03.png\" width=\"684\" alt=\"User credentials\"\u003e\n\nCopy the _Access key ID_ and the _Secret access key_ directly from the site and paste them in the application configuration file `src/sparkify.cfg`: do it in the proper configuration keys:\n\n```ini\n...\nACCESS_KEY_ID = the-access-key-id\nSECRET_ACCESS_KEY = the-secret-access-key\n...\n```\n\n**Important:** never share this credentials to anyone, your AWS account can be compromised.\n\nIt's time to add some permissions to the user. Go to the _Users_ menu and navigate to the user _Summary_. Here you can select the tab _Permissions_ and click the button _Add inline policy_.\n\n\u003cimg src=\"images/iam-create-user-04.png\" width=\"611\" alt=\"User summary\"\u003e\n\nThe inline policy can be created easily using a visual editor, but we will write the JSON document corresponding the policy instead. Select the tab _JSON_.\n\n\u003cimg src=\"images/iam-create-user-05.png\" width=\"636\" alt=\"Inline policy\"\u003e\n\nAnd now copy and paste the JSON below into the policy editor.\n\n```json\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:ListBucket\"\n            ],\n            \"Resource\": [\n                \"arn:aws:s3:::udacity-dend\"\n            ]\n        },\n        {\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:GetObject\"\n            ],\n            \"Resource\": [\n                \"arn:aws:s3:::udacity-dend/log-data/*\",\n                \"arn:aws:s3:::udacity-dend/song-data/*\"\n            ]\n        },\n        {\n            \"Effect\": \"Allow\",\n            \"Action\": \"s3:ListBucket\",\n            \"Resource\": [\n                \"data-lake-bucket-arn\"\n            ]\n        },\n        {\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:GetObject\",\n                \"s3:DeleteObject\",\n                \"s3:PutObject\"\n            ],\n            \"Resource\": [\n                \"data-lake-bucket-arn/*\"\n            ]\n        }\n    ]\n}\n```\n\nThe permissions are listed in the `Statement` collection:\n\n- The user can list the source bucket\n- The user can get all the objects stored in the `log-data` and `song-data` folders\n- The user can list the data lake bucket\n- The user can get, create and delete all the objects stored in the data lake bucket\n\nIn the last two permissions, make the `Resource` to point to the data lake: replace the text `data-lake-bucket-arn` with tha actual bucket ARN that we get [before](#creating-a-bucket). Once the policy is ready, click the button _Review policy_ and then _Save changes_.\n\n\n### Cloning the repository\u003ca name=\"cloning-the-repository\"\u003e\u003c/a\u003e\n\nThe first step is to clone this repository. Just type the following command in your Terminal:\n\n```bash\n# Clone the repository...\ngit clone https://github.com/vermicida/data-lake.git\n\n# ...and move to its directory\ncd data-lake\n```\n\n### Creating the virtual environment\u003ca name=\"creating-the-virtual-environment\"\u003e\u003c/a\u003e\n\nOnce the project is cloned, you must create a new virtual environment for it. You may have multiple Python versions installed, so it's important to specify the correct one with the `--python` argument. The code on this repository has been successfully tested over Python 3.6, but it must work on any Python 3 version; set the version that applies to your case in the following commands:\n\n```bash\n# Create the virtual environment...\nvirtualenv --python python3.6 project-env\n\n# ...and activate it\nsource project-env/bin/activate\n```\n\nIt's time to install the project's dependencies. You will use `pip` to do this job:\n\n```bash\npip install -r requirements.txt\n```\n\nIt will be ready in a minute.\n\n---\n\n## How to use\u003ca name=\"how-to-use\"\u003e\u003c/a\u003e\n\nHere are listed the steps to follow in order to make the ETL pipeline work.\n\n### Running the ETL\u003ca name=\"running-the-etl\"\u003e\u003c/a\u003e\n\nThis step will test your patience: it can takes a few hours to complete. It performs the actions described below:\n\n1. Create a Spark session.\n2. Read the _song data_, stored in S3, into a dataframe. **Note:** I decided to take just a slice of this data because of the slowness of the writting process that comes later. Instead of get all the JSON files, it just take those from partition `A/A/A`.\n3. Transform the song data to create the `songs` dimension table and store it back in S3 in Apache Parquet format.\n4. Transform the song data to create the `artists` dimension table and store it back in S3 in Apache Parquet format.\n5. Read the _log data_, stored in S3, into a dataframe.\n6. Transform the log data to create the `users` dimension table and store it back in S3 in Apache Parquet format.\n7. Transform the log data to create the `time` dimension table and store it back in S3 in Apache Parquet format.\n8. Transform the song and log data to create the `songplays` fact table and store it back in S3 in Apache Parquet format.\n\nRun this command:\n\n```bash\n# Move to the src directory...\ncd src\n\n# ...and run the ETL\npython etl.py\n```\n\nNow better go to eat something, take a nap, or watch the last episode of Money Heist.\n\nWhen the ETL process ends, go to your data lake bucket and verify that it worked like expected. The image below shows a sample of the `artists` table in Apache Parquet format.\n\n\u003cimg src=\"images/s3-artists-parquet.png\" width=\"724\" alt=\"User summary\"\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvermicida%2Fdata-lake","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvermicida%2Fdata-lake","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvermicida%2Fdata-lake/lists"}