https://github.com/ovsundal/sparkify-data-lake
https://github.com/ovsundal/sparkify-data-lake
data-lake parquet spark
Last synced: 3 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/ovsundal/sparkify-data-lake
- Owner: ovsundal
- Created: 2019-06-04T06:38:14.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2019-07-08T14:02:38.000Z (about 7 years ago)
- Last Synced: 2025-10-08T02:59:56.309Z (10 months ago)
- Topics: data-lake, parquet, spark
- Language: Python
- Size: 407 KB
- Stars: 1
- Watchers: 0
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Sparkify-data-lake
This is project 4 of Udacitys Data Engineering Nanodegree. In this project Spark is used to pull data from a s3 bucket.
Temporary tables are then created, and data is uploaded to another bucket written in .parquet format.
#### Starting the program
1. Add global aws config values in dl.cfg
2. Execute "etl.py".