https://github.com/fabioba/sparkify-data-pipeline
This project is the final project of the the Data Pipeline - Udacity module.
https://github.com/fabioba/sparkify-data-pipeline
airflow aws docker-compose
Last synced: 2 months ago
JSON representation
This project is the final project of the the Data Pipeline - Udacity module.
- Host: GitHub
- URL: https://github.com/fabioba/sparkify-data-pipeline
- Owner: fabioba
- Created: 2022-11-17T15:01:37.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2022-11-22T10:18:50.000Z (over 3 years ago)
- Last Synced: 2025-03-23T03:41:29.893Z (over 1 year ago)
- Topics: airflow, aws, docker-compose
- Language: Python
- Homepage:
- Size: 240 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# SPARKIFY-DATA-PIPELINE
## Business Context
A music streaming company, `Sparkify`, has decided that it is time to introduce more automation and monitoring to their data warehouse ETL pipelines and come to the conclusion that the best tool to achieve this is Apache Airflow.
## Prerequisites
* Connect Airflow and AWS
* Connect Airflow to the AWS Redshift Cluster
## Source Data
The source data resides in S3 and needs to be processed in Sparkify's data warehouse in Amazon Redshift. The source datasets consist of JSON logs that tell about user activity in the application and JSON metadata about the songs the users listen to.
* Log data: s3://udacity-dend/log_data
* Song data: s3://udacity-dend/song_data
## Workflow
