Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/thunchanokbow/walmart-end-to-end-with-azure-databricks

The data is extracted in batches using Azure Data Factory. and transformed using Spark in Azure Databricks. The transformed data is stored in Synapse Analytics in columnar format. Finally, a dashboard is created using Looker Studio or Power BI to display sales data, customer analysis, and other information
https://github.com/thunchanokbow/walmart-end-to-end-with-azure-databricks

dashboard data-factory databricks gen2 synapse-analytics walmart

Last synced: about 2 months ago
JSON representation

Host: GitHub
URL: https://github.com/thunchanokbow/walmart-end-to-end-with-azure-databricks
Owner: thunchanokbow
Created: 2024-02-22T07:30:15.000Z (11 months ago)
Default Branch: main
Last Pushed: 2024-03-24T03:54:28.000Z (10 months ago)
Last Synced: 2024-03-24T04:27:50.681Z (10 months ago)
Topics: dashboard, data-factory, databricks, gen2, synapse-analytics, walmart
Language: Jupyter Notebook
Homepage:
Size: 8.77 MB
Stars: 0
Watchers: 1
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

        # Walmart-end-to-end-with-Azure-Databricks

The data is extracted in batches using Azure Data Factory and transformed using Spark in Azure Databricks. The transformed data is stored in Synapse Analytics in columnar format. Finally, a dashboard is created using Looker Studio or Power BI, As a result of my project, fresh food has become their top category! This will help with the decision to promote fresh food by placing them in more visible locations and on the website or application. Their most popular fresh food is MorningStar Farms Veggie Burgers, with customers picking up a total of 184 units daily. With this information, the company is able to contact MorningStar Farm's to order more products and discuss pricing options. For more information about data source.[Here](https://www.kaggle.com/datasets/thedevastator/product-prices-and-sizes-from-walmart-grocery) 

## Project Overview

![0](/images/0.png)

## Contents 

[Create Storage Account](sections/01-storage-accounts.md).
 

- [Azure Blob Storage](sections/01-storage-accounts.md).


- [Azure Data Lake Storage Gen2](sections/01-storage-accounts.md#Create-Azure-Data-Lake-Storage-Gen2).


[Create Azure Data Factory](sections/02-azure-data-factory.md).


- [Create Linked Services](sections/02-azure-data-factory.md#Create-Linked-Services).


  - [Source Linked Services](sections/02-azure-data-factory.md#Source-Linked-Services).


  - [Sink Linked Services](sections/02-azure-data-factory.md#Sink-Linked-Services).


  - [Databricks Linked Services](sections/02-azure-data-factory.md#Databricks-Linked-Services).
 

- [Create Dataset](sections/02-azure-data-factory.md#Create-Dataset).


  - [Dataset of Azure Blob Storage](sections/02-azure-data-factory.md#Dataset-of-Azure-Blob-Storage).


  - [Dataset of Azure Data Lake Storage Gen2](sections/02-azure-data-factory.md#Dataset-of-Azure-Data-Lake-Storage-Gen2).


- [Extract data using Azure Data Factory](sections/02-azure-data-factory.md#Extract-data-using-Azure-Data-Factory).
 

[Create Azure Databricks Services](sections/03-azure-databricks.md).


- [Create Cluster](sections/03-azure-databricks.md#Create-Cluster).


- [Mounting Data Lake Storage](sections/03-azure-databricks.md#Mounting-Data-Lake-Storage).


- [Using Databricks to transform data](sections/03-azure-databricks.md#Using-Databricks-to-transform-data).


- [Run Databricks Notebook using Azure Data Factory](sections/03-azure-databricks.md#Run-Databricks-Notebook-using-Azure-Data-Factory).
 

[Create Synapse Analytics workspace](sections/04-synapse-analytics.md).


- [Create Lake database](sections/04-synapse-analytics.md).


- [Create Table from data lake](sections/04-synapse-analytics.md).


- [Load data into Azure Synapse Analytics](sections/04-synapse-analytics.md).


[Import Azure Synapse Analytics (SQL DW) Data into the Power BI](sections/04-synapse-analytics.md).


- [Connect to a data source in Power BI](sections/04-synapse-analytics.md).


- [Creating Data visualizations](sections/04-synapse-analytics.md).