{"id":32395362,"url":"https://github.com/jiatangzhi/master_thesis","last_synced_at":"2026-05-04T00:32:57.487Z","repository":{"id":319110695,"uuid":"1077626940","full_name":"jiatangzhi/master_thesis","owner":"jiatangzhi","description":"This project implements my master’s thesis on building a scalable, ACID-compliant data lakehouse architecture for IoT and industrial workloads, integrating AWS Glue, S3, Athena, and Grafana with Iceberg to evaluate Copy-on-Write vs Merge-on-Read performance.","archived":false,"fork":false,"pushed_at":"2025-10-16T14:23:39.000Z","size":2317,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-17T17:16:09.695Z","etag":null,"topics":["apache-iceberg","aws-glue","aws-s3","batch-processing","data-engineering","data-lakehouse","distributed-systems","grafana","iot-data","mqtt","open-table-format","python3","schema-evolution","spark"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jiatangzhi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-16T14:07:26.000Z","updated_at":"2025-10-16T15:44:25.000Z","dependencies_parsed_at":null,"dependency_job_id":"6e72f169-976a-4a45-bf29-7c5b6651e676","html_url":"https://github.com/jiatangzhi/master_thesis","commit_stats":null,"previous_names":["jiatangzhi/master_thesis"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/jiatangzhi/master_thesis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiatangzhi%2Fmaster_thesis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiatangzhi%2Fmaster_thesis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiatangzhi%2Fmaster_thesis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiatangzhi%2Fmaster_thesis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jiatangzhi","download_url":"https://codeload.github.com/jiatangzhi/master_thesis/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiatangzhi%2Fmaster_thesis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32590275,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-03T22:12:39.696Z","status":"ssl_error","status_checked_at":"2026-05-03T22:09:10.534Z","response_time":103,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-iceberg","aws-glue","aws-s3","batch-processing","data-engineering","data-lakehouse","distributed-systems","grafana","iot-data","mqtt","open-table-format","python3","schema-evolution","spark"],"created_at":"2025-10-25T06:21:06.564Z","updated_at":"2026-05-04T00:32:57.463Z","avatar_url":"https://github.com/jiatangzhi.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🧠 Design and Optimization of a Cloud-Based Transactional Data Lake for Evolving Data Models\n\nThis repository contains the implementation and documentation of my **Master’s Thesis** project — an end-to-end cloud-based **transactional data lakehouse** built with **Apache Iceberg** and **AWS** services.  \n\nThe project bridges the gap between traditional data lakes and warehouses by introducing **ACID compliance**, **schema evolution**, and **real-time adaptability**, optimized for large-scale and evolving analytical workloads.\n\n---\n\n## 📖 Abstract\n\nThe rapid expansion of data-driven applications demands scalable and transactional cloud architectures.  \nThis project designs and evaluates a **data lakehouse** that integrates the flexibility of **data lakes** with the transactional integrity of **data warehouses**, leveraging **Apache Iceberg** as the open table format.  \n\nIt benchmarks **Copy-on-Write (CoW)** and **Merge-on-Read (MoR)** strategies to assess ingestion throughput, query performance, and compaction efficiency, while demonstrating schema evolution, rollback, and time travel capabilities.\n\n---\n\n## 🧩 Architecture Overview\n\nThe proposed lakehouse is composed of **five modular layers**:\n\n1. **Ingestion Layer** — IoT data ingestion using **AWS IoT Core** and **MQTT** protocols.  \n2. **Storage Layer** — Persistent, scalable storage in **Amazon S3** (raw + curated zones).  \n3. **Processing Layer** — ETL pipelines using **AWS Glue** and **Apache Spark** with metadata tracked in **AWS Glue Data Catalog**.  \n4. **API Layer** — Query access via **Amazon Athena** for SQL-based analytics.  \n5. **Consumption Layer** — Interactive visualization and real-time monitoring through **Grafana** and **Amazon CloudWatch**.\n\n---\n\n## 🧪 Research Focus\n\nThe main research objective is to design and optimize a **transactional, cloud-native data lakehouse** capable of handling high-ingestion IoT data streams while maintaining performance, consistency, and reliability.\n\n### 🔍 Comparative Evaluation\n- **Apache Iceberg vs Apache Hudi**\n- **Copy-on-Write (CoW)** vs **Merge-on-Read (MoR)**\n- Analysis of **query latency**, **ingestion throughput**, **compaction cost**, and **metadata scalability**\n\n### 🧠 Core Features\n- ACID-compliant table management  \n- Schema evolution and snapshot rollback  \n- Metadata pruning and partition optimization  \n- ETL automation through AWS Glue  \n- IaC deployment using **AWS CDK**  \n\n---\n\n## ⚙️ Technologies Used\n\n| Category | Technologies |\n|-----------|--------------|\n| Cloud Services | AWS S3, Glue, Athena, IoT Core, CloudWatch, Managed Grafana |\n| Data Frameworks | Apache Iceberg, Apache Hudi, Apache Spark |\n| Programming | Python, PySpark, Boto3 |\n| Infrastructure | AWS CDK (IaC), Virtual Machines (Edge Simulation) |\n| Visualization | Grafana, CloudWatch Dashboards |\n\n---\n\n## 🧰 Industrial Simulator\n\nA local **industrial monitoring simulator** was developed to generate realistic IoT metrics (CPU, memory, disk I/O, network I/O).  \nEach virtual device (VM) publishes telemetry payloads to AWS IoT Core via **MQTT**, storing raw CSV data in S3 before transformation into **Parquet** format for Iceberg tables.\n\n---\n\n## 📊 Performance Experiments\n\nBenchmarks included:\n- Read and write latency under CoW vs MoR strategies  \n- Query time before and after compaction  \n- Schema evolution tests (column addition/removal)  \n- Snapshot rollback and time-travel verification  \n\nResults demonstrated that **Iceberg’s metadata layer** significantly improves query performance and storage efficiency, making it a robust foundation for **scalable, cost-effective cloud lakehouses**.\n\n---\n\n## 🗂️ License\n\nThis project is licensed under the **MIT License**.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjiatangzhi%2Fmaster_thesis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjiatangzhi%2Fmaster_thesis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjiatangzhi%2Fmaster_thesis/lists"}