https://github.com/waridrox/etl-databricks

ETL pipeline using low level factory design to draw insights on apple products.
https://github.com/waridrox/etl-databricks

Last synced: 11 months ago
JSON representation

ETL pipeline using low level factory design to draw insights on apple products.

• Host: GitHub
• URL: https://github.com/waridrox/etl-databricks
• Owner: waridrox
• Created: 2024-12-25T06:53:17.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2024-12-25T07:14:24.000Z (over 1 year ago)
• Last Synced: 2025-05-19T15:12:01.350Z (about 1 year ago)
• Language: Python
• Size: 2.04 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Scalable ETL Pipelines for Apple Products Analysis with PySpark

- Created ETL pipelines for different use cases to analyze Apple products information using PySpark with sources such as **CSV**, **Parquet**, and **Delta Tables**.

- Implemented the **Factory Pattern** for designing reader classes to handle multiple data sources effectively.

- Applied PySpark's **DataFrame API** and **Spark SQL** for business transformation logic.

- Demonstrated data loading strategies for both **DataLake** and **Data LakeHouse** architectures.

- Explored and implemented PySpark concepts like:

  - **Broadcast joins**

  - **Partitioning and bucketing**

  - **Window functions** like `LAG` and `LEAD`

  - **Delta Table operations**

![ETL architecture](./architecture.png)

## Technologies Used

- **Apache Spark** (PySpark)

- **Databricks** for pipeline creation and storage (Community version)

- **Data sources**: CSV, Parquet, Delta Table

- **Storage**: DataLake and Data LakeHouse (Databricks)