{"id":30151468,"url":"https://github.com/evialex/adventure-works-de-project","last_synced_at":"2026-05-05T04:02:49.514Z","repository":{"id":306784293,"uuid":"1027212204","full_name":"EviAleX/Adventure-Works-DE-Project","owner":"EviAleX","description":"Azure End To End Data Engineering Project","archived":false,"fork":false,"pushed_at":"2025-08-08T11:25:15.000Z","size":2007,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-08T13:12:44.219Z","etag":null,"topics":["azure","databricks","etl","etl-pipeline","git","powerbi","spark"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EviAleX.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-27T14:53:55.000Z","updated_at":"2025-08-08T11:25:20.000Z","dependencies_parsed_at":"2025-07-27T16:47:16.354Z","dependency_job_id":"2cdb117d-5c99-4960-bd11-67f4ce278660","html_url":"https://github.com/EviAleX/Adventure-Works-DE-Project","commit_stats":null,"previous_names":["evialex/adventure-works-de-project"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/EviAleX/Adventure-Works-DE-Project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EviAleX%2FAdventure-Works-DE-Project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EviAleX%2FAdventure-Works-DE-Project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EviAleX%2FAdventure-Works-DE-Project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EviAleX%2FAdventure-Works-DE-Project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EviAleX","download_url":"https://codeload.github.com/EviAleX/Adventure-Works-DE-Project/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EviAleX%2FAdventure-Works-DE-Project/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269873158,"owners_count":24488993,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-11T02:00:10.019Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","databricks","etl","etl-pipeline","git","powerbi","spark"],"created_at":"2025-08-11T11:02:15.206Z","updated_at":"2026-05-05T04:02:44.470Z","avatar_url":"https://github.com/EviAleX.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Adventure-Works-DE-Project\n\nThis project demonstrates a complete end-to-end data engineering solution built on the Microsoft Azure platform. The pipeline ingests the AdventureWorks dataset from a public GitHub repository, processes it through a multi-stage architecture, and serves it to Power BI for business intelligence and analytics.\n\n![Architecture Diagram](./images/architecture.png)\n\n---\n## 🏛️**Architecture Overview**\n\nThe solution leverages a modern data stack on Azure, following a medallion architecture (Bronze, Silver, Gold layers) to ensure data quality and scalability.\n\n- **Orchestration**: Azure Data Factory (ADF)\n- **Data Lake**: Azure Data Lake Storage (ADLS) Gen2\n- **Data Transformation**: Azure Databricks (using Spark)\n- **Data Warehousing**: Azure Synapse Analytics\n- **Business Intelligence**: Power BI\n\n---\n## 🚀**Pipeline Flow**\nThe data moves through four distinct stages: \n\n### 1. **Ingestion (Bronze Layer)**\n- **Azure Data Factory** uses a dynamic copy activity to the raw datasets via an HTTP connector from Github\n\n\u003cimg width=\"1917\" height=\"1065\" alt=\"image\" src=\"https://github.com/user-attachments/assets/426cbb34-106a-41fd-866f-f62c159264a0\" /\u003e\n\n- The raw data is landed in the **Bronze** container in Azure Data Lake Storage without any modifications.\n\n\u003cimg width=\"1919\" height=\"1065\" alt=\"image\" src=\"https://github.com/user-attachments/assets/30f9e43a-0c9e-4a7c-87a3-79627f4fa40d\" /\u003e\n\n---\n### 2. **Transformation (Silver Layer)**\n- **Azure Databricks** reads the raw data from the Bronze layer\n- A PySpark job perfoms key transformations, including cleaning records, normalizing data formats, and structuring the data\n\n\u003cimg width=\"1919\" height=\"1065\" alt=\"image\" src=\"https://github.com/user-attachments/assets/5e67ab0a-d317-429d-8f46-8f438063c3da\" /\u003e\n\n- The cleaned, transformed data is saved in the **Silver** container in the efficient **Parquet** format \n\n\u003cimg width=\"1919\" height=\"1068\" alt=\"image\" src=\"https://github.com/user-attachments/assets/b71b4343-3a65-4a6d-9c6c-664f5a5d15c1\" /\u003e\n\n### 3. **Warehousing (Gold Layer)**\n- **Azure Synapse Analytics** connects to the Silver container using a serverless SQL pool.\n- External tables and views are created on top of the Parquet files to structure the data for analysis.\n\n\u003cimg width=\"1919\" height=\"1066\" alt=\"image\" src=\"https://github.com/user-attachments/assets/16f66c47-7c21-43b4-922c-47f49ae855c4\" /\u003e\n\n\u003cimg width=\"1919\" height=\"1066\" alt=\"image\" src=\"https://github.com/user-attachments/assets/9b1d3b08-3248-4702-9648-f4aeb5fa259d\" /\u003e\n\n- This final, curated data represents the Gold layer, ready for reporting.\n\n\u003cimg width=\"1919\" height=\"1066\" alt=\"image\" src=\"https://github.com/user-attachments/assets/327e180e-3d15-44ad-bcff-a3e050d2c8be\" /\u003e\n\n---\n\n### 4. **Visualization** \n- **Power BI** connects directly to Azure Synapse Analytics.\n- Simple dashboards and reports are built to provide actionable insights from the curated data. Although creating beautiful dashboard was not part of project, so it is kinda raw. \n\n\u003cimg width=\"1919\" height=\"1151\" alt=\"image\" src=\"https://github.com/user-attachments/assets/c85424e0-ac1e-4d10-be07-1d4761fc5a50\" /\u003e\n\n---\n## **Key Takeaways** ✅\nThis project showcases a robust, automated, and scalable data engineering solution on Azure. It effectively transforms raw, source data into high-value business insights, demonstrating a complete data lifecycle.\n\n---\n## **Acknowledgment** 🎉\nThis project was inspired by the work of [Ansh Lamba](https://github.com/anshlambagit). For a detailed video walkthrough of similar project, please check out [his Youtube channel](https://www.youtube.com/watch?v=0GTZ-12hYtU\u0026t=15907s\u0026ab_channel=AnshLamba). \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevialex%2Fadventure-works-de-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevialex%2Fadventure-works-de-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevialex%2Fadventure-works-de-project/lists"}