https://github.com/subkanthi/s3-data-lake-example

https://github.com/subkanthi/s3-data-lake-example

data-lake data-lake-store

Last synced: over 1 year ago
JSON representation

• Host: GitHub
• URL: https://github.com/subkanthi/s3-data-lake-example
• Owner: subkanthi
• Created: 2019-06-30T02:15:11.000Z (about 7 years ago)
• Default Branch: master
• Last Pushed: 2019-07-24T03:59:07.000Z (about 7 years ago)
• Last Synced: 2025-02-04T12:48:07.237Z (over 1 year ago)
• Topics: data-lake, data-lake-store
• Language: Python
• Size: 18.6 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# s3-data-lake-example

Creating a S3 Data lake with pyspark ETL.

First step involves using pandas to only extract the columns that 

are required and to create files in data lake using parquet format.

Queries to be supported

The Lines where the expected vs actual arrival time is long.