{"id":42715535,"url":"https://github.com/edugmenes/azure-data-engineering","last_synced_at":"2026-01-29T15:38:13.192Z","repository":{"id":333710682,"uuid":"1138403029","full_name":"edugmenes/azure-data-engineering","owner":"edugmenes","description":"This repository contains my first end-to-end Data Engineering project, built using Microsoft Azure Cloud and Azure Databricks with PySpark.","archived":false,"fork":false,"pushed_at":"2026-01-23T18:43:58.000Z","size":236,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-24T08:37:15.063Z","etag":null,"topics":["azure","cloud","data","data-engineering","data-lakehouse","data-structures","databricks","delta-lake","etl-pipelines","lakehouse","lakehouse-architectures","medallion-architecture","microsoft-azure","pyspark","spark"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/edugmenes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-20T16:19:07.000Z","updated_at":"2026-01-23T18:44:05.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/edugmenes/azure-data-engineering","commit_stats":null,"previous_names":["edugmenes/azure-databricks-project","edugmenes/azure-data-engineering"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/edugmenes/azure-data-engineering","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edugmenes%2Fazure-data-engineering","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edugmenes%2Fazure-data-engineering/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edugmenes%2Fazure-data-engineering/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edugmenes%2Fazure-data-engineering/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/edugmenes","download_url":"https://codeload.github.com/edugmenes/azure-data-engineering/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edugmenes%2Fazure-data-engineering/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28880318,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-29T10:31:27.438Z","status":"ssl_error","status_checked_at":"2026-01-29T10:31:01.017Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","cloud","data","data-engineering","data-lakehouse","data-structures","databricks","delta-lake","etl-pipelines","lakehouse","lakehouse-architectures","medallion-architecture","microsoft-azure","pyspark","spark"],"created_at":"2026-01-29T15:38:13.131Z","updated_at":"2026-01-29T15:38:13.185Z","avatar_url":"https://github.com/edugmenes.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# End-to-End Azure Data Engineering\n\nThis repository contains my **first Data Engineering project**, built using **Microsoft Azure Cloud** and **Azure Databricks**.\n\nThe project focuses on designing and implementing **ETL pipelines** using **PySpark** following the **Medallion Architecture (Bronze, Silver, Gold)**, a modern and widely adopted pattern for building scalable and reliable data platforms.\n\n\n\n## 🚀 Project Overview\n\nThe main objective of this project is to demonstrate how raw data can be ingested, transformed, and curated into analytics-ready datasets using cloud-native tools and best practices.\n\nThe solution covers:\n- Data ingestion from raw sources\n- Data transformation and cleansing\n- Data modeling for analytics consumption\n- Distributed data processing with PySpark\n\n\n\n## 🏗️ Architecture\n![Architecture Diagram](/archives/images/Captura%20de%20tela%202026-01-20%20123843.png)\n\n### Azure Event Hubs\nPurpose: Real-time data ingestion\n\nRole in project:\n- Captures streaming events (clicks, logs, IoT, transactions)\n- Highly scalable and fault tolerant\n\n✨ Without Event Hubs: You’d miss live data or overload systems\n\n### Azure Data Factory (ADF)\nPurpose: Orchestration \u0026 batch ingestion\n\nRole in project:\n- Schedules pipelines\n- Moves data from source → data lake\n- Triggers Databricks jobs\n\n✨ Think of it as the control center\n\n### Azure Databricks (Apache Spark)\nPurpose: Data processing \u0026 transformation\n\nRole in project:\n- Processes huge volumes of data efficiently\n- Implements Bronze → Silver → Gold logic\n- Handles both batch and streaming data\n\n✨ This is the engine of the architecture\n\n### Azure Data Lake Storage Gen2 (ADLS)\nPurpose: Central storage layer\n\nRole in project:\n- Stores all data (Bronze, Silver, Gold)\n- Cheap, scalable, secure\n- Optimized for analytics\n\n✨ This is your single source of truth\n\n### Delta Lake\nPurpose: Reliability \u0026 governance on top of ADLS\n\nRole in project:\n- ACID transactions\n- Schema enforcement\n- Time travel (data versioning)\n- Efficient reads/writes\n\n✨ Delta Lake turns “files” into real analytical tables\n\n\n\n## 🧮 Data Model\nThe project is structured using a **Medallion Architecture** \n\n    \u003cstorage-account\u003e/\u003ccontainer\u003e/\n    │\n    └── projet1/\n        ├── resources/                  # Source and target data\n        │   ├── source/                 # CSV files received from the source\n        │   └── target/                 # Exports files for customers\n        │\n        ├── 01-bronze/                  # Raw data\n        │   ├── customers/\n        │   │   └── customers.parquet\n        │   ├── sales/\n        │   │   └── sales.parquet\n        │   ...\n        │    \n        ├── 02-silver/                  # Clean data\n        │   ├── customers/\n        │   │   └── customers.parquet\n        │   ├── sales/\n        │   │   └── sales.parquet\n        │   ...\n        │\n        ├── 03-gold/                    # Aggregated data\n        │   ├── sales_per_category/\n        │   ├── sales_per_city/\n        │   ...\n        │\n        ├── metadata/                   # Metadata and logs\n        │   ├── bronze/\n        │   ├── silver/\n        │   ├── gold/\n        │   ├── ddl/                    # CREATE TABLE scripts\n        │   ├── logs/                   # ETL execution logs\n        │   └── checkpoints/            # Autoloader checkpoints / streaming\n        │\n        └── tmp/                        # Temporary staging\n\n### 🟤 Bronze Layer\n- Raw data ingestion\n- Minimal transformation\n- Preserves source data as-is\n\n### ⚪ Silver Layer\n- Data cleansing and normalization\n- Data enrichment\n- Application of business rules\n\n### 🟡 Gold Layer\n- Curated, analytics-ready datasets\n- Optimized for reporting and BI use cases\n\n\n\n## 🛠️ Technologies Used\n\n- Microsoft Azure\n- Azure Databricks\n- Apache Spark (PySpark)\n- Delta Lake\n- Medallion Architecture\n\n\n\n## 🎯 Key Learnings\n\nThrough this project, I gained hands-on experience with:\n- Cloud-based data platforms\n- Distributed data processing using Spark\n- Building scalable ETL pipelines\n- Applying modern Data Engineering design patterns\n- Managing data across multiple data layers\n\n\n\n## 📌 Notes\n\nThis is a **learning project**, created to apply theoretical concepts in a practical environment using industry-standard tools.\n\nFuture improvements may include:\n- Pipeline orchestration\n- Data quality checks\n- Performance optimization\n- Monitoring and logging\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedugmenes%2Fazure-data-engineering","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fedugmenes%2Fazure-data-engineering","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedugmenes%2Fazure-data-engineering/lists"}