https://github.com/k178412/sql-data-warehouse-project

A hands-on data warehouse project using SQL Server, covering ETL processes, and data modeling.
https://github.com/k178412/sql-data-warehouse-project

bronze-layer data-analysis data-analytics data-cleaning data-engineering data-warehouse database datalake dataset datawarehouse etl etl-pipeline etl-process gold-layer silver-layer sql sql-query sql-server sqlserver

Last synced: about 2 months ago
JSON representation

A hands-on data warehouse project using SQL Server, covering ETL processes, and data modeling.

• Host: GitHub
• URL: https://github.com/k178412/sql-data-warehouse-project
• Owner: k178412
• License: mit
• Created: 2025-05-11T12:31:21.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2025-06-28T06:12:56.000Z (12 months ago)
• Last Synced: 2025-06-28T07:25:05.495Z (12 months ago)
• Topics: bronze-layer, data-analysis, data-analytics, data-cleaning, data-engineering, data-warehouse, database, datalake, dataset, datawarehouse, etl, etl-pipeline, etl-process, gold-layer, silver-layer, sql, sql-query, sql-server, sqlserver
• Language: TSQL
• Homepage:
• Size: 1.75 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# SQL Data Warehouse Project

A hands-on data warehouse project using SQL Server, covering ETL processes, and data modeling.

---

## 🏗️ Data Architecture

This project follows the Medallion Data Architecture, breaking the data pipeline into three layers (Bronze, Silver, and Gold layers) to ensure clarity, maintainability, and scalability.

![data architecture](https://github.com/k178412/sql-data-warehouse-project/blob/main/docs/data%20architecture.png)

1. **Bronze Layer** - Stores raw data exactly as received from source systems.

2. **Silver Layer** - Cleans and transforms data for consistency, applying standardization and normalization.

3. **Gold Layer** - Contains business-ready data, optimized for reporting and insights.

---

## 🔍 Project Overview

This project shows the full data warehouse lifecycle, from source data ingestion to business-ready data models.  

Key components include:

1. 🧱 **Data Architecture** - Designing a structured data warehouse using Medallion Architecture.

2. 🔄 **ETL Pipelines** - Extracting, transforming, and loading data using SQL scripts.

3. 🧮 **Data Modeling** - Creating fact and dimension tables for optimized querying and analytics.

---

## 📂 Project Files

1. [**Datasets/**](https://github.com/k178412/sql-data-warehouse-project/tree/main/datasets) - Source CRM and ERP data stored as CSV files, used for ingestion into the warehouse.

2. [**Docs/**](https://github.com/k178412/sql-data-warehouse-project/tree/main/docs) - Diagrams (created in Draw.io) for architecture, data flow, and data modeling.

3. [**Scripts/**](https://github.com/k178412/sql-data-warehouse-project/tree/main/scripts) - SQL scripts for database setup, table creation, ETL processes, and transformations.

---

## 📊 Diagrams

1. [**Data Architecture**](https://github.com/k178412/sql-data-warehouse-project/blob/main/docs/data%20architecture.png) - Defines the structural flow of the data warehouse.

2. [**Data Flow**](https://github.com/k178412/sql-data-warehouse-project/blob/main/docs/data%20flow.png) - Illustrates the journey from raw to refined data.

   ![](https://github.com/k178412/sql-data-warehouse-project/blob/main/docs/data%20flow.png)

4. [**Data Integration**](https://github.com/k178412/sql-data-warehouse-project/blob/main/docs/data%20integration.png) - Highlights how different source systems connect.

5. [**Data Model**](https://github.com/k178412/sql-data-warehouse-project/blob/main/docs/data%20model.png) - Represents logical schema for fact and dimension tables.

6. [**ETL Pipeline**](https://github.com/k178412/sql-data-warehouse-project/blob/main/docs/etl.png) - Shows different extraction, transformation, and loading processes.

---

## 🛠️ Tools & Technologies

1. **SQL Server** - Core database platform for data storage and transformation.

2. **Notion** - For planning and tracking project progress.

3. **Draw.io** - Used to design diagrams and workflows.

4. **Git** - To manage version control and repository tracking.

---

## 📌 Project Tracking

You can view the detailed plan, and progress here:

**Notion Project Link**: [Data Warehouse Project](https://glimmer-ricotta-2b2.notion.site/Data-Warehouse-Project-1ee49f37ed3480e1812dc7b441fb2b11?pvs=4)

---

## 🔒 License

This project is licensed under MIT License.

---

## 🤝 Contributing

Contributions, issues, and feature requests are welcome!

---

⭐️ If you find this project useful, please consider giving it a star!