https://github.com/majdi-akrmi/elt-ipl

This is an End-to-End Data Engineering Project that using the IPL Dataset.
https://github.com/majdi-akrmi/elt-ipl

apache-spark databricks-notebooks pyspark snowflake snowsql

Last synced: 3 months ago
JSON representation

This is an End-to-End Data Engineering Project that using the IPL Dataset.

• Host: GitHub
• URL: https://github.com/majdi-akrmi/elt-ipl
• Owner: Majdi-Akrmi
• Created: 2024-04-28T11:31:28.000Z (9 months ago)
• Default Branch: main
• Last Pushed: 2024-04-28T12:20:48.000Z (9 months ago)
• Last Synced: 2024-09-28T07:02:47.129Z (3 months ago)
• Topics: apache-spark, databricks-notebooks, pyspark, snowflake, snowsql
• Language: Jupyter Notebook
• Homepage:
• Size: 1.67 MB
• Stars: 7
• Watchers: 1
• Forks: 5
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        

# IPL Data Engineering

This is an End-to-End Data Engineering Project that using the IPL Dataset.

## Pipeline Architecture

[![IPL.png](https://i.postimg.cc/9fpNxFRk/IPL.png)](https://postimg.cc/FfYZzvxb)

## Screenshots

### Snowflake Environment

[![Screenshot-from-2024-04-27-14-09-35.png](https://i.postimg.cc/kM03RGxN/Screenshot-from-2024-04-27-14-09-35.png)](https://postimg.cc/9zYNvWgz)

### Databricks Environment

[![Screenshot-from-2024-04-28-13-01-33.png](https://i.postimg.cc/ZKSshd8Q/Screenshot-from-2024-04-28-13-01-33.png)](https://postimg.cc/Z0wx80DP)

### Data Visualization

[![Screenshot-from-2024-04-28-13-02-10.png](https://i.postimg.cc/yxvZ5tXs/Screenshot-from-2024-04-28-13-02-10.png)](https://postimg.cc/644q4Hmm)

[![Screenshot-from-2024-04-28-13-02-26.png](https://i.postimg.cc/8ktf18MV/Screenshot-from-2024-04-28-13-02-26.png)](https://postimg.cc/s1GgcNTn)

## Tech Stack

**Dataset:** IPL-dataset from @data.world

**Extract & Load Data:** SnowSQL & Snowflake DB 

**Transform Data:** Pyspark & Databricks