{"id":27707719,"url":"https://github.com/smars-bin-hu/ecomdwh-batchdataprocessingplatform","last_synced_at":"2025-04-26T07:57:04.288Z","repository":{"id":278316601,"uuid":"933366839","full_name":"Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform","owner":"Smars-Bin-Hu","description":"This project aims to build an enterprise-grade offline data warehouse solution based on e-commerce platform order data.","archived":false,"fork":false,"pushed_at":"2025-04-24T05:39:35.000Z","size":27161,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-26T07:56:59.576Z","etag":null,"topics":["big-data","data-warehouse","hadoop","spark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Smars-Bin-Hu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-15T19:31:10.000Z","updated_at":"2025-04-24T05:39:39.000Z","dependencies_parsed_at":"2025-04-17T08:00:27.977Z","dependency_job_id":"1689ecc3-aded-4755-a2f8-d39a49e44cc1","html_url":"https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform","commit_stats":null,"previous_names":["smars-bin-hu/big-data-engineering-project1","smars-bin-hu/amod-5410-team-project-ecommerce-dwh","smars-bin-hu/ecomdwh-pipeline","smars-bin-hu/ecomdwh-batchdataprocessingplatform"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Smars-Bin-Hu%2FEComDWH-BatchDataProcessingPlatform","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Smars-Bin-Hu%2FEComDWH-BatchDataProcessingPlatform/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Smars-Bin-Hu%2FEComDWH-BatchDataProcessingPlatform/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Smars-Bin-Hu%2FEComDWH-BatchDataProcessingPlatform/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Smars-Bin-Hu","download_url":"https://codeload.github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250955392,"owners_count":21513499,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","data-warehouse","hadoop","spark"],"created_at":"2025-04-26T07:57:03.501Z","updated_at":"2025-04-26T07:57:04.276Z","avatar_url":"https://github.com/Smars-Bin-Hu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![1](https://github.com/user-attachments/assets/5b98ca67-3770-4d4a-b444-ad8b70c40557)\n\n# Enterprise-Grade Offline Data Warehouse Solution for E-Commerce\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/blob/main/src/README/quick-start.md\"\u003e\n      \u003cimg src=\"https://img.shields.io/badge/project-🚀quick_start-blue?style=for-the-badge\u0026logo=github\" alt=\"Sublime's custom image\"/\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/Smars-Bin-Hu/EComDWH-Pipeline/tree/main/src\"\u003e\n      \u003cimg src=\"https://img.shields.io/badge/project-source_code-green?style=for-the-badge\u0026logo=github\" alt=\"Sublime's custom image\"/\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/wiki\"\u003e\n      \u003cimg src=\"https://img.shields.io/badge/project-all%20documents-red?style=for-the-badge\u0026logo=github\" image\"/\u003e\n   \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/python-3.12.9-blue?style=plastic\u0026logo=python\u0026logoColor=yellow\u0026logoSize=auto\u0026color=blue\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/apache_spark-3.3.0-blue?style=plastic\u0026logo=apachespark\u0026logoSize=auto\u0026color=white\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/apache_hadoop-3.2.4-blue?style=plastic\u0026logo=apachehadoop\u0026logoColor=yellow\u0026logoSize=auto\u0026color=blue\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/apache_hive-3.1.3-blue?style=plastic\u0026logo=apachehive\u0026logoColor=yellow\u0026logoSize=auto\u0026color=yellow\"/\u003e\n  \u003cbr\u003e\n  \u003cimg src=\"https://img.shields.io/badge/apache_zookeeper-3.8.4-79bb2e?style=plastic\u0026color=79bb2e\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/mysql-8.0.39-blue?style=plastic\u0026logo=mysql\u0026logoColor=blue\u0026logoSize=auto\u0026color=blue\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/clickhouse-24.12-FFCC01?style=plastic\u0026logo=clickhouse\u0026logoColor=yellow\u0026logoSize=auto\u0026color=FFCC01\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Prometheus-2.52.0-f2f2e8?style=plastic\u0026logo=prometheus\u0026logoColor=red\u0026logoSize=auto\u0026color=white\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Grafana-10.3.1-f2f2e8?style=plastic\u0026logo=Grafana\u0026logoColor=red\u0026logoSize=auto\u0026color=white\"/\u003e\n\u003c/p\u003e\n\nThis project aims to build an enterprise-grade offline data warehouse solution based on e-commerce platform order data. By leveraging **Docker containers** to simulate a big data platform, it achieves a complete workflow from ETL processing to data warehouse modeling, OLAP analysis, and data visualization. \n\nThe core value of this project lies in its implementation of **enterprise-grade data warehouse modeling**, integrating e-commerce order data with relevant business themes through standardized dimension modeling and fact table design, ensuring data accuracy, consistency, and traceability. Meanwhile, **the deployment of a big data cluster via Docker containers** simplifies environment management and operational costs, offering a flexible deployment model for distributed batch processing powered by Spark. Additionally, the project incorporates **CI/CD automation**, enabling rapid iterations while maintaining the stability and reliability of the data pipeline. Storage and computation are also **highly optimized** to maximize hardware resource utilization.\n\nTo monitor and manage the system effectively, a **Grafana-based cluster monitoring system** has been implemented, providing real-time insights into cluster health metrics and assisting in performance tuning and capacity planning. Finally, by integrating **business intelligence (BI) and visualization solutions**, the project transforms complex data warehouse analytics into intuitive dashboards and reports, allowing business teams to make data-driven decisions more efficiently.\n\nBy combining these critical features—including:\n\n| ✅ Core Feature | 🔥 Core Highlights | 📦 Deliverables |\n|-----------|------------------|---------------|\n| **1. [Data Warehouse Modeling and Documentation](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tree/main?tab=readme-ov-file#1-data-warehouse-modeling-and-documentation)** | - Full dimensional modeling process (Star Schema / Snowflake Schema) \u003cbr\u003e - Standardized development norms (ODS/DWD/DWM/DWS/ADS five-layer modeling) \u003cbr\u003e - Business Matrix: defining \u0026 managing dimensions \u0026 fact tables | - Data warehouse design document (Markdown/PDF) \u003cbr\u003e - Hive SQL modeling code \u003cbr\u003e - Database ER diagram |\n| **2. [Cluster Deployment](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tree/main?tab=readme-ov-file#2-a-self-built-distributed-big-data-platform)** | - Fully containerized deployment with Docker for quick replication \u003cbr\u003e - High-availability environment: Hadoop + Hive + Spark + Zookeeper + ClickHouse | - Docker images (open-source Dockerfile) \u003cbr\u003e - `.env` configuration file \u003cbr\u003e - `docker-compose.yml` (one-click cluster startup) \u003cbr\u003e - Infra configuration files (Hadoop, Hive, Spark, Zookeeper) |\n| **3. [Distributed Batch Processing](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tree/main?tab=readme-ov-file#3-distributed-batch-processing)** | - ETL processing using Spark for Oracle relational data \u003cbr\u003e - Multi-layer processing: ODS → DWD → DWM → DWS → ADS \u003cbr\u003e - Efficient data transformation \u0026 aggregation | - Spark ETL code (PySpark) \u003cbr\u003e - SparkSQL scripts \u003cbr\u003e - Data flow diagram |\n| **4. [CI/CD Automation](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tree/main?tab=readme-ov-file#4-cicd-automation)** | - Automated Airflow DAG deployment (auto-sync with code updates) \u003cbr\u003e - Automated Spark job submission (eliminates manual `spark-submit`) \u003cbr\u003e - Hive table schema change detection (automatic alerts) | - GitHub Actions / Jenkins pipeline \u003cbr\u003e - CI/CD code and documentation \u003cbr\u003e - Sample log screenshots |\n| **5. [Storage \u0026 Computation Optimization](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tree/main?tab=readme-ov-file#5-storage--computation-optimization)** | - SQL optimization (dynamic partitioning, indexing, storage partitioning) \u003cbr\u003e - Spark tuning: Salting, Skew Join Hint, Broadcast Join, `reduceByKey` vs. `groupByKey` \u003cbr\u003e - Hive tuning: Z-Order sorting (boost ClickHouse queries), Parquet + Snappy compression | - Pre \u0026 post optimization performance comparison \u003cbr\u003e - Spark optimization code \u003cbr\u003e - SQL execution plan screenshots |\n| **6. [DevOps - Monitoring and Alerting](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tree/main?tab=readme-ov-file#6-devops---monitoring-and-alerting)** | - Prometheus + Grafana for performance monitoring Hadoop Cluster / MySQL \u003cbr\u003e - AlertManager for alerting and email receiving | - Prometheus, Grafana configuration files \u003cbr\u003e - Grafana dashboard screenshots \u003cbr\u003e |\n| **7. [Business Intelligence \u0026 Visualization](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tree/main?tab=readme-ov-file#7-business-intelligence--visualization)** | - PowerBI dashboards for data analysis \u003cbr\u003e - Real business-driven visualizations \u003cbr\u003e - Providing actionable business insights | - PowerBI visualization screenshots \u003cbr\u003e - Business analysis report \u003cbr\u003e - Key business metric explanations (BI Insights) |\n\nthis project delivers a professional, robust, and highly efficient solution for enterprises dealing with large-scale data processing and analytics.\n\n## ⚙️ Core Deliverables\n\n### 1. Data Warehouse Modeling and Documentation\n\nThis project demonstrates my ability to build a data warehouse from the ground up following enterprise-grade standards. I independently designed and documented a complete SOP for data warehouse development, covering every critical step in the modeling roadmap. From initial business data research to final model delivery, I established a standardized methodology that ensures clarity, scalability, and maintainability. The SOP includes detailed best practices on data warehouse layering, table naming conventions, field naming rules, and lifecycle management for warehouse tables. For more information, please refer to the documentation below.\n\n\u003cdetails\u003e\n  \u003csummary\u003e🔗 Click to Show DWH Dimensional Modelling Documents and Code\u003c/summary\u003e\n  \n  - [DWH Modelling Standard Operation Procedure (SOP)](./docs/doc/dwh-modelling-sop.md)\n  - [Business Data Research](./docs/doc/business_data_research.md)\n  \n  Data Warehouse Development Specification\n  \n  - [Data Warehouse Layering Specification](./docs/doc/data-warehouse-development-specification/data-warehouse-layering-specification.md)\n  - [Table Naming Conventions](./docs/doc/data-warehouse-development-specification/table-naming-convertions.md)\n  - [Data Warehouse Column Naming Conventions](./docs/doc/data-warehouse-development-specification/partitioning-column-naming-conventions.md)\n  - [Data Table Lifecycle Management Specification](./docs/doc/data-warehouse-development-specification/data-table-lifecycle-management-specification.md)\n\n\n  [🔨 Code - Hive DDL](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tree/main/src/warehouse_modeling)(for Data Warehouse All Layers including ods, dwd, dwm, dws, dwt, dim (Operational Data Storage, DW detail, DW middle, DW summary, DW theme, DW Dimension, Analytical Data Storage-CK)\n\n\u003c/details\u003e\n\n![image](https://github.com/user-attachments/assets/ec924ea9-1acf-48a3-99ba-1546c1e8c3a9)\n\u003cp align=\"center\"\u003e\u003cem\u003eFigure 1: DWH Dimensional Modelling SOP\u003c/em\u003e\u003c/p\u003e\n\n![image](https://github.com/user-attachments/assets/ab21c750-052f-4c10-baf0-bc97e5ed8274)\n\u003cp align=\"center\"\u003e\u003cem\u003eFigure 2: DWH Dimensional Modelling Methodology Diagram\u003c/em\u003e\u003c/p\u003e\n\n![ECom-DWH-Pipeline](https://github.com/Smars-Bin-Hu/my-draw-io/blob/main/ECom-DWH-Datapipeline-Proejct/ECom-DWH-Pipeline.drawio.svg)\n\u003cp align=\"center\"\u003e\u003cem\u003eFigure 3: DWH Dimensional Modelling Architecture\u003c/em\u003e\u003c/p\u003e\n\n\n### 2. A Self-Built Distributed Big Data Platform\n\nThis distributed data platform was built entirely from scratch by myself. Starting with a base Ubuntu 20.04 docker image, I manually installed and configured each component step by step, ultimately creating a fully functional three-node Hadoop cluster with distributed storage and computing capabilities. The platform is fully containerized, featuring a highly available HDFS and YARN architecture. It supports Hive for data warehousing, Spark for distributed computing, Airflow for workflow orchestration, and Prometheus + Grafana for performance monitoring. A MySQL container manages metadata for both Hive and Airflow and is also monitored by Prometheus. An Oracle container simulates the backend of a business system and serves as a data source for the data warehouse. All container images are open-sourced and published to [🔨 GitHub Container Registry](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/pkgs/container/proj1-dwh-cluster), making it easy for anyone to deploy the same platform locally.\n\n[🔨 Code - Docker Compose File](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/blob/main/docker-compose-bigdata.yml)\n\n[🔨 Code - Configuration Files for Cluster: Hadoop, ZooKeeper, Hive, MySql, Spark, Prometheus\u0026Grafana, Airflow](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tree/main/src/infra)\n\n[🔨 Code - Container Internal Scripts: Hadoop, ZooKeeper, Hive, MySql, Spark, Prometheus\u0026Grafana, Airflow](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tree/main/src/scripts)\n\n[🔨 Code - Common Used Snippets for Cluster: Hadoop, ZooKeeper, Hive, MySql, Spark, Prometheus\u0026Grafana, Airflow](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tree/main/src/snippets)\n\n\u003cimg width=\"1500\" alt=\"image\" src=\"https://github.com/user-attachments/assets/576ce494-1d96-4b3c-992e-96addc1e6f43\" /\u003e\n\u003cp align=\"center\"\u003e\u003cem\u003eFigure 1: All Containers Window\u003c/em\u003e\u003c/p\u003e\n\n![ECom-DWH-Pipeline](https://github.com/Smars-Bin-Hu/my-draw-io/blob/main/ECom-DWH-Datapipeline-Proejct/ECom-DWH-Tech-Arc.drawio.svg)\n\u003cp align=\"center\"\u003e\u003cem\u003eFigure 2: Data Platform Architecture\u003c/em\u003e\u003c/p\u003e\n\n\n\n### 3. Distributed Batch Processing\n\n1. [🔨 Code - Extract and Load pipeline (OLTP -\u003e DWH, DWH -\u003e OLAP)](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tree/main/src/data_pipeline)\n\n2. [🔨 Code - Batch Processing (Transform)](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tree/main/src/batch_processing)\n\n3. 🔨 Code - Scheduling based on Airflow (DAG)\n\n### 4. CI/CD Automation\n\n\u003ca href=\"https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/actions/workflows/main.yml\"\u003e\n    \u003cimg src=\"https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/actions/workflows/main.yml/badge.svg\" image\"/\u003e\n\u003c/a\u003e\n\n1. GitHub Actions Code\n\n[🔨 Code - workflows.main YAML](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/blob/main/.github/workflows/main.yml)\n\n2. Key Screenshots\n\n\u003cimg width=\"1500\" alt=\"image\" src=\"https://github.com/user-attachments/assets/00fc170c-14ae-4f9d-a7e1-1480cb4d0112\" /\u003e\n\u003cp align=\"center\"\u003e\u003cem\u003eFigure 1: Data platform launching and stop automation\u003c/em\u003e\u003c/p\u003e\n\n\u003cimg width=\"1500\" alt=\"image\" src=\"https://github.com/user-attachments/assets/4a0a07fe-a629-4a10-a232-b01e0ed3aed2\" /\u003e\n\u003cp align=\"center\"\u003e\u003cem\u003eFigure 2: Sample Log Screenshot I \u003c/em\u003e\u003c/p\u003e\n\n\u003cimg width=\"1500\" alt=\"image\" src=\"https://github.com/user-attachments/assets/4c79ca68-286d-4f19-88a6-9917b565bc9e\" /\u003e\n\u003cp align=\"center\"\u003e\u003cem\u003eFigure 3: Sample Log Screenshot II \u003c/em\u003e\u003c/p\u003e\n\n3. [🔗 Link - Automation Workflow Web UI](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/actions)\n\n### 5. Storage \u0026 Computation Optimization\n\n### 6. DevOps - Monitoring and Alerting\n\n[🔨 Code - Monitoring Services Configuaration Files: Prometheus, Grafana, AlertManager](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tree/main/src/infra/monitoring-config)\n\n[🔨 Code - Monitoring Services Start\u0026Stop Scripts: Prometheus, Grafana, AlertManager](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tree/main/src/scripts/monitoring)\n\n[🔨 Code - Container Metrics Exporter Start\u0026Stop Scripts: `my-start-node-exporter.sh` \u0026 `my-stop-node-exporter.sh`](https://github.com/Smars-Bin-Hu/EComDWH-BatchDataProcessingPlatform/tree/main/src/scripts/hadoop-master)\n\n\u003cimg width=\"1500\" alt=\"Prometheus\" src=\"https://github.com/user-attachments/assets/2f157fd1-1e41-4090-9c74-0fc3e97e385e\" /\u003e\n\u003cp align=\"center\"\u003e\u003cem\u003eFigure 1: Prometheus\u003c/em\u003e\u003c/p\u003e\n\u003cimg width=\"1501\" alt=\"Grafana-Hadoop-Cluster-instance-hadoop-master\" src=\"https://github.com/user-attachments/assets/7c4d94ca-3262-46be-adb3-18c3777a314f\" /\u003e\n\u003cp align=\"center\"\u003e\u003cem\u003eFigure 2: Grafana-Hadoop-Cluster-instance-hadoop-master\u003c/em\u003e\u003c/p\u003e\n\u003cimg width=\"1495\" alt=\"Grafana-MySQLD\" src=\"https://github.com/user-attachments/assets/7df991be-3642-4ad4-9672-3d8483423178\" /\u003e\n\u003cp align=\"center\"\u003e\u003cem\u003eFigure 3: Grafana-MySQLD\u003c/em\u003e\u003c/p\u003e\n\n### 7. Business Intelligence \u0026 Visualization\n\n[🔗 Link - PowerBI Public Access(Expirable)](https://app.powerbi.com/view?r=eyJrIjoiMzVjYTQ3NmMtODllZS00N2JhLWFkNWItMWI4MmYyNDZjMDc1IiwidCI6IjI0MGI3OWM1LTZiZWYtNDYwOC1hNDE3LTY1NjllODQzNTQ1YyJ9)\n\nUse Microsoft PowerBI connect to the Clickhouse and extract the **analytical data storage** layer\n\u003cimg width=\"1500\" alt=\"image\" src=\"https://github.com/user-attachments/assets/21eaea88-0fac-488d-9696-3be5720b4ac3\" /\u003e\n\u003cp align=\"center\"\u003e\u003cem\u003eFigure 1: PowerBI Dashboard Demo\u003c/em\u003e\u003c/p\u003e\n\n## Tech Stack\n\nThis project sets up a high-availability big data platform, including the following components:\n\n![Apache Spark](https://img.shields.io/badge/Spark-FDEE21?style=for-the-badge\u0026logo=apachespark\u0026logoColor=black) \t![Apache Hadoop](https://img.shields.io/badge/Hadoop-66CCFF?style=for-the-badge\u0026logo=apachehadoop\u0026logoColor=black) ![Apache ZooKeeper](https://img.shields.io/badge/Zookeeper-8e8c3a?style=for-the-badge\u0026color=8e8c3a) ![Apache Airflow](https://img.shields.io/badge/Airflow-017CEE?style=for-the-badge\u0026logo=apacheairflow\u0026logoColor=white) ![Apache Hive](https://img.shields.io/badge/Hive-FDEE21?style=for-the-badge\u0026logo=apachehive\u0026logoColor=black)  ![ClickHouse](https://img.shields.io/badge/ClickHouse-FFCC01?style=for-the-badge\u0026logo=clickhouse\u0026logoColor=white) ![Prometheus](https://img.shields.io/badge/Prometheus-f2f2e8?style=for-the-badge\u0026logo=prometheus\u0026color=f2f2e8) ![Grafana](https://img.shields.io/badge/Grafana-252523?style=for-the-badge\u0026logo=grafana\u0026color=252523)  ![MySQL](https://img.shields.io/badge/MySQL-blue?style=for-the-badge\u0026logo=mysql\u0026logoColor=yellow\u0026color=blue) ![Oracle Database](https://img.shields.io/badge/Oracle_Database-red?style=for-the-badge\u0026color=red) ![Microsoft PowerBI](https://img.shields.io/badge/Microsoft_PowerBI-pink?style=for-the-badge\u0026color=pink) \n![Docker](https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge\u0026logo=docker\u0026logoColor=white) \n\n| Components             | Features                       | Version |\n|------------------------|--------------------------------|---------|\n| Apache Hadoop          | Big Data Distributed Framework | 3.2.4   |\n| Apache Zookeeper       | High Availability              | 3.8.4   |\n| Apache Spark           | Distributed Computing          | 3.3.0   |\n| Apache Hive            | Data Warehousing               | 3.1.3   |\n| Apache Airflow         | Workflow Scheduling            | 2.7.2   |\n| MySQL                  | Metastore                      | 8.0.39  |\n| Oracle Database        | Workflow Scheduling            | 19.0.0  |\n| Azure Cloud ClickHouse | OLAP Analysis                  | 24.12   |\n| Microsoft PowerBI      | BI Dashboard                   | latest  |\n| Prometheus             | Monitoring                     | 2.52.0  |\n| Grafana                | Monitoring GUI                 | 10.3.1  |\n| Docker                 | Containerization               | 28.0.1  |\n\n## 📁 Project Directory\n\n```bash\n/bigdata-datawarehouse-project\n│── /.github/workflows            # CI/CD automation workflows via GitHub Actions\n│── /docs                         # docs (all business and technologies documents about this project)\n│── /src\n    │── /data_pipeline            # data pipeline code (ETL/ELT Logic, output)\n    │── /warehouse_modeling       # DWH modelling（Hive SQL etc.）\n    │── /batch_processing         # Data Batch processing (PySpark + SparkSQL)\n    │── /dags                # Task Scheduler(Airflow)\n    │── /infra                    # infrastructure deployment(Docker, configuration files)\n    │── /snippets                 # common used commands and snippets\n    │── /README                   # Source Code Use Instruction Markdown Files\n    │── README.md                 # Navigation of Source Code Use Instruction\n    │── main_data_pipeline.py     # operate the data_pipeline module to do the `Extract` and `Load` jobs\n    │── main_batch_processing.py  # operate the batch_processing module to do the `Transform` jobs\n│── /tests                        # all small features unit testing snippets (DWH modelling, data pipeline, dags etc.) \n│── README.md                     # Introduction about project\n│── docker-compose-bigdata.yml    # Docker Compose to launch the docker cluster\n│── .env                          # `public the .env on purpose` for docker-compose file use\n│── .gitignore                    # Git ignore some directory not to be committed to the remote repo\n│── .gitattributes                # Git repository attributes config\n│── LICENSE                       # COPYRIGHT for this project\n│── mysql-metadata-restore.sh     # container operational level scripts: restore mysql container metadata\n│── mysql-metastore-dump.sh       # container operational level scripts: dump mysql container metadata\n│── push-to-ghcr.sh               # container operational level scripts: push the images to GitHub Container Registry\n│── start-data-clients.sh         # container operational level scripts: start hive, spark etc\n│── start-hadoop-cluster.sh       # container operational level scripts: start hadoop HA cluster \n│── start-other-services.sh       # container operational level scripts: start airflow, prometheus, grafana etc\n│── stop-data-clients.sh          # container operational level scripts: stop hive, spark etc\n│── stop-hadoop-cluster.sh        # container operational level scripts: stop hadoop HA cluster \n│── stop-other-services.sh        # container operational level scripts: stop airflow, prometheus, grafana etc\n```\n\n## 🚀 Quick Start `/src`\n\n### [🔗 Source Code Instruction for Use](./src/README.md)\n### \n### \n\n## 📌 Project Documents `/docs`\n\n#### 1. Business logic \u0026\u0026 Tech Selection\n\n- Business Logic\n- [Project Tech Architecture](./docs/doc/tech-architecture.md)\n\n#### 2. Development Specification\n\n[DWH Modelling Standard Operation Procedure (SOP)](./docs/doc/dwh-modelling-sop.md)\n\n- [Business Data Research](./docs/doc/business_data_research.md)\n- Data Warehouse Development Specification\n  - [Data Warehouse Layering Specification](./docs/doc/data-warehouse-development-specification/data-warehouse-layering-specification.md)\n  - [Table Naming Conventions](./docs/doc/data-warehouse-development-specification/table-naming-convertions.md)\n  - [Data Warehouse Column Naming Conventions](./docs/doc/data-warehouse-development-specification/partitioning-column-naming-conventions.md)\n  - [Data Table Lifecycle Management Specification](./docs/doc/data-warehouse-development-specification/data-table-lifecycle-management-specification.md)\n\n- Python Development Specification\n  - Package Modulize\n \n- SQL Development Specification\n  - [Development Specification](./docs/doc/data-warehouse-development-specification/development-specification.md)\n\n#### 3. Troubleshooting\n\n  - [Future Bugs to Fix](./docs/doc/error-handling/future-fix.md)\n  - [04_MAR_2025](./docs/doc/error-handling/04_MAR_2025.md)\n  - [05_MAR_2025](./docs/doc/error-handling/05_MAR_2025.md)\n  - [06_MAR_2025](./docs/doc/error-handling/06_MAR_2025.md)\n\n\n#### 4. Infrastructure \u0026 Building\n\n  - 核心架构docker容器分布图\n  - Hadoop 3节点 的搭建和配置\n  - Hive 节点的搭建和配置\n  - Spark 节点的搭建和配置\n  - mysql 节点的搭建和配置\n  - oracle 节点的搭建和配置\n  - airflow 节点的搭建和配置 (airflow.cfg 里 mysql 和 localexecutor 的配置）\n  - `docker-compose` 文件的配置\n\n#### 5. Development\n\n  - Data Warehousing\n    - ods\n    - dwd\n  - Data Pipeline ETL\n    - Spark on Yarn to connect Oracle (Hello World)\n    - Spark to extract data and load to HDFS\n    - OOP\n  - Scheduler (Airflow)\n  - some files under /scripts\n\n#### 6. Optimization\n\n  - [Too many INFO logs: Reducing Spark Console Log Levels](./docs/doc/optimization/reducing-spark-console-log-levels.md)\n\n#### 7. Testing\n  - spark_connect_oracle.py  \n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](./LICENSE) file for details.  \nCreated and maintained by **Smars-Bin-Hu**.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmars-bin-hu%2Fecomdwh-batchdataprocessingplatform","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsmars-bin-hu%2Fecomdwh-batchdataprocessingplatform","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmars-bin-hu%2Fecomdwh-batchdataprocessingplatform/lists"}