Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/datawithbaraa/sql-data-warehouse-project
A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.
https://github.com/datawithbaraa/sql-data-warehouse-project
data-analysis data-analytics data-cleaning data-engineering data-lakehouse data-science data-warehouse data-warehousing datalake datascience datawarehouse datawarehousing etl etl-job etl-pipeline medallion-architecture sql sql-query sql-server sqlserver
Last synced: about 1 month ago
JSON representation
A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.
- Host: GitHub
- URL: https://github.com/datawithbaraa/sql-data-warehouse-project
- Owner: DataWithBaraa
- License: mit
- Created: 2024-12-30T09:15:55.000Z (about 1 month ago)
- Default Branch: main
- Last Pushed: 2024-12-30T09:56:02.000Z (about 1 month ago)
- Last Synced: 2024-12-30T10:27:13.987Z (about 1 month ago)
- Topics: data-analysis, data-analytics, data-cleaning, data-engineering, data-lakehouse, data-science, data-warehouse, data-warehousing, datalake, datascience, datawarehouse, datawarehousing, etl, etl-job, etl-pipeline, medallion-architecture, sql, sql-query, sql-server, sqlserver
- Language: TSQL
- Homepage:
- Size: 13.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Data Warehouse and Analytics Project
Welcome to the **Data Warehouse and Analytics Project** repository! š
This project demonstrates a comprehensive data warehousing and analytics solution, from building a data warehouse to generating actionable insights. Designed as a portfolio project, it highlights industry best practices in data engineering and analytics.
---
## š Project Overview
This project involves:
1. **Data Architecture**: Designing a robust data warehouse with **Bronze**, **Silver**, and **Gold** layers.
2. **ETL Pipelines**: Extracting, transforming, and loading data from source systems into the warehouse.
3. **Data Modeling**: Developing fact and dimension tables optimized for analytical queries.
4. **Analytics & Reporting**: Creating SQL-based reports and dashboards for actionable insights.
This repository is an excellent resource for professionals and students looking to showcase expertise in:
- SQL Development
- Data Engineering
- ETL Pipeline Design
- Data Modeling
- Data Analytics
---
## š Project Requirements
### Building the Data Warehouse (Data Engineering)
#### Objective
Develop a modern data warehouse using SQL Server to consolidate sales data, enabling analytical reporting and informed decision-making.
#### Specifications
- **Data Sources**: Import data from two source systems (ERP and CRM) provided as CSV files.
- **Data Quality**: Cleanse and resolve data quality issues prior to analysis.
- **Integration**: Combine both sources into a single, user-friendly data model designed for analytical queries.
- **Scope**: Focus on the latest dataset only; historization of data is not required.
- **Documentation**: Provide clear documentation of the data model to support both business stakeholders and analytics teams.
---
### BI: Analytics & Reporting (Data Analytics)
#### Objective
Develop SQL-based analytics to deliver detailed insights into:
- **Customer Behavior**
- **Product Performance**
- **Sales Trends**
These insights empower stakeholders with key business metrics, enabling strategic decision-making.
For more details, refer to [docs/requirements.md](docs/requirements.md).
---
## š Repository Structure
```
data-warehouse-project/
ā
āāā datasets/ # Raw datasets used for the project (ERP and CRM data)
ā
āāā docs/ # Project documentation and architecture details
ā āāā architecture.drawio # Editable Draw.io file of the project's architecture
ā āāā data_catalog.md # Catalog of datasets, including field descriptions and metadata
ā āāā data_flow.drawio # Editable Draw.io file for the data flow diagram
ā āāā data_models.drawio # Editable Draw.io file for data models (star schema)
ā āāā business_rules.md # Business rules applied during ETL and analysis
ā āāā naming-conventions.md # Consistent naming guidelines for tables, columns, and files
ā
āāā scripts/ # SQL scripts for ETL and transformations
ā āāā bronze/ # Scripts for extracting and loading raw data
ā āāā silver/ # Scripts for cleaning and transforming data
ā āāā gold/ # Scripts for creating analytical models
ā
āāā tests/ # Test scripts and quality files
ā
āāā README.md # Project overview and instructions
āāā LICENSE # License information for the repository
āāā .gitignore # Files and directories to be ignored by Git
āāā requirements.txt # Dependencies and requirements for the project
```
---
## šÆ Goals
- To showcase proficiency in data engineering and analytics.
- To provide a practical example of building and using a data warehouse for business intelligence.
- To demonstrate SQL expertise for ETL, data modeling, and analytical reporting
---
## š ļø Tools & Technologies
- **SQL Server**: Data warehouse and analytics implementation.
- **SQL**: ETL pipelines and analytics development.
- **DrawIO**: For Drawing Diagrams and Data Models.
- **Notion**: For Project Management.
---
## š”ļø License
This project is licensed under the [MIT License](LICENSE). You are free to use, modify, and share this project with proper attribution.
## š About Me
Hi there! I'm **Baraa Khatib Salkini**, also known as **Data With Baraa**. Iām an IT professional and passionate YouTuber on a mission to share knowledge and make working with data enjoyable and engaging!
Let's stay in touch! Feel free to connect with me on the following platforms:
[](http://bit.ly/3GiCVUE)
[](https://linkedin.com/in/baraa-khatib-salkini)
[](https://www.datawithbaraa.com)
[](https://bit.ly/BaraaNewsletter)
[](https://paypal.me/baraasalkini)
[](https://www.youtube.com/@datawithbaraa)