https://github.com/mohammedowaiskh/retail-data-warehouse

An end-to-end Azure Data Engineering pipeline for Retail Analytics, built using ADF, Databricks, Synapse, and Power BI.
https://github.com/mohammedowaiskh/retail-data-warehouse

azure data-engineering data-warehousing etl-pipeline python

Last synced: about 1 month ago
JSON representation

An end-to-end Azure Data Engineering pipeline for Retail Analytics, built using ADF, Databricks, Synapse, and Power BI.

• Host: GitHub
• URL: https://github.com/mohammedowaiskh/retail-data-warehouse
• Owner: mohammedOwaiskh
• License: mit
• Created: 2025-08-31T19:00:00.000Z (3 months ago)
• Default Branch: master
• Last Pushed: 2025-09-01T18:28:08.000Z (3 months ago)
• Last Synced: 2025-09-01T20:33:32.047Z (3 months ago)
• Topics: azure, data-engineering, data-warehousing, etl-pipeline, python
• Homepage:
• Size: 99.6 KB
• Stars: 0
• Watchers: 0
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Retail Data Warehouse on Azure

An end-to-end Azure Data Engineering pipeline for Retail Analytics, built using ADF, Databricks, Synapse, and Power BI.

---

## Project Overview

This project demonstrates the design and implementation of a modern Data Lakehouse pipeline using Azure Data Engineering services. The architecture follows the Medallion (Bronze-Silver-Gold) pattern for data ingestion, transformation, and consumption, enabling scalable and efficient analytics.

The goal of this project is to build a cloud-based data platform that ingests raw data from multiple sources, processes it into structured and curated formats, and makes it available for advanced analytics and reporting.

This end-to-end pipeline showcases:

- Data Ingestion from diverse sources (databases, logs, files).

- Data Storage in Azure Data Lake (Bronze, Silver, Gold zones).

- Orchestration with Azure Data Factory (ADF).

- Data Transformation using Azure Databricks.

- Consumption Layer for analytics with Power BI dashboards.

---

## 🏗️ Architecture

### High Level Architecture

![Data Warehouse Architecture](snapshots/high-level-architecture.png)

---

## Prerequisites and Setup

Steps to setup this project on the Azure Cloud Portal and be found in the [setup.md](/docs/setup.md).