{"id":28355164,"url":"https://github.com/husskhosravi/elt-data-warehouse-snowflake","last_synced_at":"2026-02-03T00:02:03.496Z","repository":{"id":292449910,"uuid":"980948324","full_name":"husskhosravi/elt-data-warehouse-snowflake","owner":"husskhosravi","description":"Design and implement a full ELT data pipeline using Snowflake and S3, featuring star schema modelling, SCD Type 1 \u0026 2 handling, and incremental load automation","archived":false,"fork":false,"pushed_at":"2025-05-16T23:48:37.000Z","size":722,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-04T09:39:29.826Z","etag":null,"topics":["aws-s3","data-engineering","data-warehouse","dimensional-modeling","scd-type-1","scd-type-2","snowflake","sql","star-schema"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/husskhosravi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-10T02:34:06.000Z","updated_at":"2025-05-16T23:48:40.000Z","dependencies_parsed_at":"2025-05-10T03:49:23.678Z","dependency_job_id":null,"html_url":"https://github.com/husskhosravi/elt-data-warehouse-snowflake","commit_stats":null,"previous_names":["husskhosravi/elt-data-warehouse-snowflake"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/husskhosravi/elt-data-warehouse-snowflake","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/husskhosravi%2Felt-data-warehouse-snowflake","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/husskhosravi%2Felt-data-warehouse-snowflake/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/husskhosravi%2Felt-data-warehouse-snowflake/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/husskhosravi%2Felt-data-warehouse-snowflake/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/husskhosravi","download_url":"https://codeload.github.com/husskhosravi/elt-data-warehouse-snowflake/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/husskhosravi%2Felt-data-warehouse-snowflake/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261020227,"owners_count":23098105,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-s3","data-engineering","data-warehouse","dimensional-modeling","scd-type-1","scd-type-2","snowflake","sql","star-schema"],"created_at":"2025-05-28T03:41:25.151Z","updated_at":"2026-02-03T00:02:03.491Z","avatar_url":"https://github.com/husskhosravi.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# 📊 End-to-End Data Warehousing Project with Snowflake, S3 and ELT\n\n## ✨ Project Overview\n\nThis project showcases my ability to design and implement an end-to-end data warehousing pipeline using **AWS S3**, **Snowflake**, and **SQL-based ELT**. I replicated a real-world scenario involving OLTP data ingestion, modelling a star schema, implementing Slowly Changing Dimensions (SCD), and handling both **initial** and **incremental loads**.\n\nThe project includes creating a raw data layer, staging layer, and dimensional data warehouse (DW) with **SCD Type 1 and Type 2** support, surrogate key handling, and performance-oriented design decisions.\n\n---\n\n## 🌐 Data Pipeline Flow\n\n![flow_diagram3](https://github.com/user-attachments/assets/0bbb009f-b5f2-4574-8eed-7d40125e7d65)\n\n---\n\n### 🔁 Pipeline Sequence Breakdown\n\n1. **📦 OLTP (Flat File) → 3NF Normalisation**\n\n   * Input: `orders_transaction_snapshot.csv`\n   * Goal: Break denormalised data into normalised tables (`cities`, `pincode`, `customers`, `products`, etc.)\n   * Output: 3NF-formatted SQL tables (simulate OLTP)\n\n2. **☁️ COPY INTO Snowflake from S3**\n\n   * Upload the 3NF `.csv` files to S3\n   * Load into **raw schema** in Snowflake using `COPY INTO`\n\n3. **🔄 Staging Layer Transformation**\n\n   * Join 3NF entities to create **flattened, enriched views**\n   * Example: `stg_customers`, `stg_products` from multiple raw tables\n\n4. **🌟 Star Schema (Dimensional Model)**\n\n   * Create **dimension tables** (e.g. `customer_dim`, `product_dim`) and **fact tables**\n   * Apply SCD logic (Type 1 or 2), surrogate keys, date ranges\n   * Structure aligned with **analytics/OLAP consumption**\n\n---\n\n## 🔧 Tech Stack \u0026 Tools\n\n* **Cloud Warehouse:** Snowflake\n* **Storage:** AWS S3 (source layer)\n* **Modelling:** Star Schema\n* **Scripting:** SQL (DDL + DML + CTEs)\n* **SCD Management:** Type 1 \u0026 Type 2\n* **Transformation Strategy:** ELT (Extract-Load-Transform)\n* **Data Partitioning:** Calendar-based folders in S3\n* **Optional Tooling:** Lucidchart (for ERD), Excel (for validation)\n\n---\n\n## 🤝 Key Learnings\n\n### ✍️ Data Modelling\n\n* Designed **conceptual, logical and physical** data models\n* Practised **normalisation (1NF, 2NF, 3NF)** and when to denormalise for DW\n* Built **ER diagrams** and star schemas with clear fact/dimension separation\n\n### 📁 Data Warehouse Design\n\n* Created **raw, staging and consumption layers**\n* Implemented **SCD Type 1** for customers (overwrite) and **SCD Type 2** for products (historical tracking)\n* Generated **surrogate keys** for performance and compatibility\n* Managed **dimension and fact table loads** with correct sequencing\n* Built reusable **calendar dimension** with recursive CTEs\n\n### ⚙️ ELT \u0026 Automation\n\n* Loaded data from S3 to Snowflake using `COPY INTO` statements\n* Applied `INSERT` and `UPDATE` logic to handle incremental loads\n* Developed logic for detecting and processing delta data using timestamp columns like `last_updated_at`\n* Created reusable SQL scripts for staging and DW layer loads to support repeatable execution and scheduling\n* Partitioned S3 by date folder structure (e.g., `/2024/12/20/`) for daily automation\n\n---\n\n## 🧱 Why I Started from a 3NF OLTP Model\n\nTo simulate a real-world enterprise scenario, I started from a highly normalised **OLTP dataset in 3rd Normal Form (3NF)** which is typical of transactional systems.\n\nOLTP systems handle high volumes of short, atomic transactions like orders or payments, prioritising data integrity and write efficiency. They use Third Normal Form (3NF) to reduce redundancy, enforce entity separation, and prevent update anomalies, ensuring clean, consistent source data for downstream analytics.\n\nThis allowed me to:\n- ✅ Practise real-world **data modelling**\n- ✅ Transition from **normalised OLTP** → **star schema**\n- ✅ Extract clean, deduplicated dimension entities\n- ✅ Build **referential integrity** from raw rows\n\nThe 3NF model was generated using SQL with window functions and joins from the OLTP source.\n\n### 📂 Sample Orders Transaction File\nThe dataset used here is a **denormalised transaction file** that mimics an OLTP export. It contains flattened order data with repeated values across rows — ideal for normalisation.\n\n[Sample OLTP Orders File](./sample_data/orders_transaction_snapshot.csv)\n\nI applied **3rd Normal Form (3NF)** transformations on this file using SQL. These transformations decomposed the flat structure into atomic entities (e.g., `cities`, `pincode`, `customers`, etc.) based on the ER diagram. Below is an example of how I extracted the `cities` table from this transaction file:\n\n### 🛠 Example 3NF Extraction: `cities` Table\n```sql\nWITH cte AS (\n  SELECT city, state, region, country,\n         ROW_NUMBER() OVER (PARTITION BY city ORDER BY orders.order_date) AS rn\n  FROM orders\n)\nSELECT ROW_NUMBER() OVER (ORDER BY city) AS id,\n       city, region, country\nINTO cities\nFROM cte\nWHERE rn = 1;\n```\n### 🛠 Example 3NF Extraction: `pincode` Table\n```sql\nWITH cte AS (\n  SELECT postal_code, cities.id AS city_id,\n         ROW_NUMBER() OVER (PARTITION BY postal_code ORDER BY postal_code) AS rn\n  FROM orders\n  INNER JOIN cities ON cities.city = orders.city\n  WHERE postal_code IS NOT NULL\n)\nSELECT ROW_NUMBER() OVER (ORDER BY postal_code) AS id,\n       postal_code, city_id\nINTO pincode\nFROM cte\nWHERE rn = 1;\n```\n### 🧱 Example: Customer Dimension Creation\n\nAfter loading 3NF-normalised tables into Snowflake’s `raw` schema, I created staging views to flatten and enrich the data before loading into the data warehouse. Below is the logic used to build the `stg_customers` table by joining `customers`, `pincode`, and `cities`.\n\n```sql\nCREATE OR REPLACE TABLE stg.stg_customers AS\nSELECT\n    cus.id AS customer_id,\n    cus.customer_name,\n    cus.username,\n    cus.segment,\n    p.postal_code,\n    c.city,\n    c.state,\n    c.region,\n    c.country\nFROM raw.pincode p\nINNER JOIN raw.cities c ON p.city_id = c.id\nINNER JOIN raw.customers cus ON cus.pincode_id = p.id;\n```\nThis staging table is then used to populate the final dw.customer_dim, applying SCD Type 1 logic.\n\n---\n\n### 📜 Key SQL Scripts\n* Full pipeline from raw → staging → dw :: [Link](./scripts/schema-scripts.sql)\n\n* Incremental load logic for *customer_dim* (SCD Type 1) :: [Link](./scripts/SCD-type1.sql)\n\n* Full SCD Type 2 handling for *product_dim*, including surrogate key, date versioning :: [Link](./scripts/SCD-type2.sql)\n\n---\n\n## 🌟 Covered in this project\n\n* Real-world **Data Engineering pipeline design**\n* **Dimensional modelling** for analytics use cases\n* Strong command of **SQL transformations and SCD implementations**\n* Handling **incremental data loads** and auditability\n\n---\n\n## 📄 Project Structure\n\n```\nelt-data-warehouse-snowflake/\n├── README.md\n├── /scripts/\n│   ├── schema-scripts.sql\n│   ├── SCD-type1.sql\n│   └── SCD-type2.sql\n├── /sample_data/\n│   ├── customers.csv\n│   └── products.csv\n│   └── ...\n├── /diagrams/\n│   ├── ERD.png\n│   └── star_schema.png\n│   └── pipeline_flow.png\n\n```\n\n---\n\n## 🚀 Future Enhancements\n\n* Implement **data quality checks** (null filters, type checks, referential integrity validation)\n* Add **dynamic parameterisation** for S3 folder paths and load dates\n* Use **orchestration tools** (e.g., dbt, Airflow, or Snowflake Tasks) to automate pipeline stages\n* Enhance **error handling and logging** during ELT steps\n* Extend with **BI dashboards** using Power BI / Tableau to demonstrate consumption layer utility\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhusskhosravi%2Felt-data-warehouse-snowflake","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhusskhosravi%2Felt-data-warehouse-snowflake","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhusskhosravi%2Felt-data-warehouse-snowflake/lists"}