https://github.com/ekote/build-your-first-end-to-end-lakehouse-solution

Build Your First End-to-End Lakehouse Solution (aka.ms/fabconlake)
https://github.com/ekote/build-your-first-end-to-end-lakehouse-solution

apache-spark data-engineering data-factory data-pipeline data-science dataflows delta-lake lakehouse machine-learning microsoft-azure microsoft-fabric parquet powerbi tutorial warehouse workshop

Last synced: 7 days ago
JSON representation

Build Your First End-to-End Lakehouse Solution (aka.ms/fabconlake)

• Host: GitHub
• URL: https://github.com/ekote/build-your-first-end-to-end-lakehouse-solution
• Owner: ekote
• Created: 2024-03-14T16:27:56.000Z (11 months ago)
• Default Branch: fabcon
• Last Pushed: 2024-09-11T22:12:14.000Z (5 months ago)
• Last Synced: 2025-02-02T01:31:47.910Z (19 days ago)
• Topics: apache-spark, data-engineering, data-factory, data-pipeline, data-science, dataflows, delta-lake, lakehouse, machine-learning, microsoft-azure, microsoft-fabric, parquet, powerbi, tutorial, warehouse, workshop
• Language: Jupyter Notebook
• Homepage: https://azuredataconf.com/#!/workshop/Build%20Your%20First%20End-to-End%20Lakehouse%20Solution/6194
• Size: 78 MB
• Stars: 15
• Watchers: 3
• Forks: 19
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Build Your First End to End Lakehouse Solution

Join the workshop to master building end-to-end data solutions with Microsoft Fabric. Learn to integrate, transform, and manage data in a lakehouse, utilizing Fabric pipelines, dataflows, notebooks, and Spark. Understand how BI analysts and data scientists utilize lakehouse data to enhance decision-making.

**Workshop Goals**

- Master Fabric Data Integration and Data Engineering and Data Science.

- Develop a complete data workflow: ingestion, preparation, serving, and operationalization.

**Project Context: Urban Mobility Transformation**

- Use Microsoft Fabric to analyze New York City's taxi data for improved urban planning and transportation safety.

- Aim: Better traffic forecasting, route management, and safety measures, leading to enhanced urban transport services and infrastructure.

**Lakehouse Solution Benefits**

- Unified urban transport data analysis.

- Enhanced fleet management and traffic forecasting.

- Data-driven urban development insights.

- Improved transportation reliability and safety.

- Adaptable data infrastructure for future urban mobility trends.

![Build Your First End to End Lakehouse Solution](screenshots/start/diagram.png)

# Agenda

> [!TIP]

> You can progress through these exercises at your own pace. While we have structured logical breaks within the session, these are merely suggestions. You are not required to stop if you prefer to continue working. These breaks are provided to accommodate those who may need them. Feel free to continue through the material as fits your learning style and needs.

> [!IMPORTANT]

> 9:00 am - 9:30 am - [Introduction, Set Up and Overview of Fabric Data Platform](exercise-0-setup/start.md)

> 

> 9:30 am - 10:00 am - [Exercise 1 - Ingest data with data pipelines and shortcuts](./exercise-1/exercise-1.md) 

> 

> 10:00 am - 10:15 am - Coffee break 15 minutes

> 

> 10:15 am - 10:45 am - [Exercise 1 - Ingest data with data pipelines and shortcuts](./exercise-1/exercise-1.md)

> 

> 10:45 am - 11:45 pm - [Exercise 2 - Transform data using Notebooks and Spark clusters](./exercise-2/exercise-2.md)

> 

> 11:45 am - 12:45 pm - Lunch 60 minutes

> 

> 12:45 pm - 01:15 pm - [Exercise 3 - Collaborate inside Notebooks and share Lakehouse. Use SQL Endpoint and SSMS](./exercise-3/exercise-3.md)

> 

> 01:15 pm - 01:25 pm - Break 10 minutes

> 

> 01:25 pm - 02:25 pm - [Exercise 4 - Serve and consume data using Power BI and Data Science ](./exercise-4/exercise-4.md)

> 

> 02:25 pm - 02:35 pm - Break 10 minutes

> 

> 02:35 pm - 03:05 pm - [Exercise 5 - Latest Fabric Features](./exercise-5/exercise-5.md)

> 

> 03:05 pm - 04:00 pm - Buffer, Recap and [Extra exercises](exercise-extra/extra.md)

>