https://github.com/rumana-amin/sql-data-warehouse-and-analytics-project
Building a data warehouse using with SQL Server, including ETL processes, data modeling, and analytics
https://github.com/rumana-amin/sql-data-warehouse-and-analytics-project
data-cleaning data-visualization data-warehouse etl etl-pipeline sql-queries sql-server
Last synced: 11 months ago
JSON representation
Building a data warehouse using with SQL Server, including ETL processes, data modeling, and analytics
- Host: GitHub
- URL: https://github.com/rumana-amin/sql-data-warehouse-and-analytics-project
- Owner: rumana-amin
- Created: 2025-01-29T15:35:31.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2025-03-01T04:54:51.000Z (11 months ago)
- Last Synced: 2025-03-01T05:25:52.891Z (11 months ago)
- Topics: data-cleaning, data-visualization, data-warehouse, etl, etl-pipeline, sql-queries, sql-server
- Language: TSQL
- Homepage:
- Size: 424 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# sql-data-warehouse-project
Welcome to the Data Warehouse and Analytics Project repository! 🚀
This guided project demonstrates a data warehousing and analytics solution, from building a data warehouse to generating actionable insights.
# 🏗️Data Architecture
The data architecture for this project follows Medallion Architecture Bronze, Silver, and Gold layers:
**1.Bronze Layer:** Stores raw data as-is from the source systems. Data is ingested from CSV Files into SQL Server Database.
**2.Silver Layer:** This layer includes data cleansing, standardization, and normalization processes to prepare data for analysis.
**3.Gold Layer:** Houses business-ready data modeled into a star schema required for reporting and analytics.
# 📖 Project Overview
This project involves:
**Data Architecture:** Designing a Modern Data Warehouse Using Medallion Architecture Bronze, Silver, and Gold layers.
**ETL Pipelines:** Extracting, transforming, and loading data from source systems into the warehouse.
**Data Modeling:** Developing fact and dimension tables optimized for analytical queries.
**Analytics & Reporting:** Creating SQL-based reports and dashboards for actionable insights.
# 🚀 Project Requirements
## Building the Data Warehouse (Data Engineering)
### Objective
Develop a modern data warehouse using SQL Server to consolidate sales data, enabling analytical reporting and informed decision-making.
### Specifications
- **Data Sources:** Import data from two source systems (ERP and CRM) provided as CSV files.
- **Data Quality:** Cleanse and resolve data quality issues prior to analysis.
- **Integration:** Combine both sources into a single, user-friendly data model designed for analytical queries.
- **Scope:** Focus on the latest dataset only; historization of data is not required.
- **Documentation:** Provide clear documentation of the data model to support both business stakeholders and analytics teams.
# BI: Analytics & Reporting (Data Analysis)
## Objective
Develop SQL-based analytics to deliver detailed insights into:
- **Customer Behavior**
- **Product Performance**
- **Sales Trends**
These insights empower stakeholders with key business metrics, enabling strategic decision-making.
For more details, refer to docs/requirements.md.
# 📂 Repository Structure
```
sql-data-warehouse-and-analytics-project/
│
├── Datasets/ # Raw datasets used for the project (ERP and CRM data in csv)
│
├── Docs/ # Project documentation and architecture details
│ ├── etl.drawio # Draw.io file shows all different techniquies and methods of ETL
│ ├── 1_Data_Architecture.png # PNG file shows the project's architecture
| ├── 2_Integration_Model.png
│ ├── Data Catalog.md # Catalog of datasets, including field descriptions and metadata
│ ├── 3_Data_Flow_Diagram.png # PNG file for the data flow diagram
│ ├── 4_Data_mart.png # PNG file for data models (star schema)
│
├── Scripts/ # SQL scripts for ETL and transformations
│ ├── bronze/ # Scripts for extracting and loading raw data
│ ├── silver/ # Scripts for cleaning and transforming data
│ ├── gold/ # Scripts for creating analytical models
│
├── tests/ # Test scripts and quality files
|
|── EDA & Analysis Scripts/ # SQL scripts for EDA, query analysis and query reports
| ├──EDA_Analysis.sql # EDA analysis like database exploration, dimension exploration, date range exloration etc.
| ├──Advance_Analysis.sql # Business analysis like change over time analysis, performance analysis, cumulative analysis etc.
| ├──report_customers.sql # Business report placing importance on customer dimension
| ├──report_products.sql # Business report placing importance on product dimension
│
├── README.md # Project overview
```
# Original Repository
[Link Here](https://github.com/DataWithBaraa/sql-data-warehouse-project)