{"id":27000295,"url":"https://github.com/cesar312/retail-sales-data-lake-predictive-analytics","last_synced_at":"2026-05-02T19:33:04.017Z","repository":{"id":282492698,"uuid":"948242761","full_name":"Cesar312/retail-sales-data-lake-predictive-analytics","owner":"Cesar312","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-02T01:00:35.000Z","size":521,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-04T03:20:30.209Z","etag":null,"topics":["cluster","dashboard","data-lake","databricks","dbfs","eda","filestore","linear-regression","mlflow","mlflow-tracking-server","predictive-analytics","pyspark","python","scikitlearn-machine-learning","sql","visualization"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Cesar312.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-14T01:42:22.000Z","updated_at":"2025-04-02T01:00:38.000Z","dependencies_parsed_at":"2025-03-15T01:18:45.888Z","dependency_job_id":"1de469b2-5890-4d39-be70-ed72cea51606","html_url":"https://github.com/Cesar312/retail-sales-data-lake-predictive-analytics","commit_stats":null,"previous_names":["cesar312/retail-sales-data-lake-analytics-prediction-model","cesar312/retail-sales-data-lake-predictive-analytics"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Cesar312/retail-sales-data-lake-predictive-analytics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Cesar312%2Fretail-sales-data-lake-predictive-analytics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Cesar312%2Fretail-sales-data-lake-predictive-analytics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Cesar312%2Fretail-sales-data-lake-predictive-analytics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Cesar312%2Fretail-sales-data-lake-predictive-analytics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Cesar312","download_url":"https://codeload.github.com/Cesar312/retail-sales-data-lake-predictive-analytics/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Cesar312%2Fretail-sales-data-lake-predictive-analytics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32547647,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-02T19:18:06.202Z","status":"ssl_error","status_checked_at":"2026-05-02T19:16:21.335Z","response_time":132,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cluster","dashboard","data-lake","databricks","dbfs","eda","filestore","linear-regression","mlflow","mlflow-tracking-server","predictive-analytics","pyspark","python","scikitlearn-machine-learning","sql","visualization"],"created_at":"2025-04-04T03:19:54.172Z","updated_at":"2026-05-02T19:33:03.997Z","avatar_url":"https://github.com/Cesar312.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Retail Sales Data Lake, Analytics \u0026 Prediction Model  \n\nThis repository contains the end-to-end process of ingesting, cleaning, transforming, analyzing, and predicting sales trends from a retail store. The analysis and transformation were performed on Databricks clusters using PySpark and SQL, and using MLflow to track machine learning experiments for predictive modeling.\n\n## Table of Contents  \n- [Overview](#overview)\n- [Dataset](#dataset)\n- [Technologies Used](#technologies-used)\n- [Project Structure](#project-structure)\n- [Data Lake Architecture](#data-lake-architecture)\n- [Data Preparation](#data-preparation)\n- [Data Analysis](#data-analysis)\n- [Visualizations \\\u0026 Dashboard](#visualizations--dashboard)\n- [Machine Learning Model \\\u0026 MLflow Tracking](#machine-learning-model--mlflow-tracking)\n- [How to Reproduce](#how-to-reproduce)\n- [Future Improvements](#future-improvements)\n- [License](#license)\n\n### Overview  \nThis project demonstrates how to: \n- **Store and clean raw sales data** in a **data lake** (Databricks FileStore).\n- **Analyze and visualize** trends using **PySpark DataFrames** and **SQL queries**.\n- **Predict future sales trends** using **Machine Learning with MLflow tracking**.\n- **Deploy the trained model** to predict sales for upcoming months.\n\n### Dataset  \nThe dataset contains the following features:\n- **Order ID**\n- **Product**\n- **Quantity Ordered**\n- **Price Each**\n- **Order Date**\n- **Purchase Address**\n- **Derived Features: City, State, ReportYear, ReportMonth**\n\nYou can find more details on the columns and data in the [Retail-Sales-Data-Preparation.ipynb](Retail-Sales-Data-Preparation.ipynb) notebook and the SQL scripts.\n\n### Technologies Used  \n- **Databricks** for cluster management, notebooks, SQL analytics, experiment tracking, and deployment\n- **PySpark** for data manipulation and transformations\n- **SQL** (Databricks SQL) for analytical queries\n- **Databricks Files System (DBFS)** as the data lake storage location\n- **Parquet / Delta** Lake for optimized storage and querying\n- **MLflow** for model tracking and logging\n- **Scikit-learn** for the prediction model\n\n### Project Structure  \n- **[Retail-Sales-Data_Preparation.ipynb](Retail-Sales-Data-Preparation.ipynb)**  \nContains the PySpark code for reading the raw CSV data, cleaning columns, handling missing values, and applying transformations.\n- **[Retail-Sales-Queries.sql](./sql/Retail-Sales-Queries.sql)**  \nA collection of SQL queries that explore the sales data, run summary statistics, and handle intermediate analysis steps.\n- **[Retail-Sales-Analytics.sql](./sql/Retail-Sales-Analytics.sql)**  \nThe final set of SQL queries that derive key business metrics, such as monthly sales, product-level aggregations, and top-selling categories.\n- **[Retail-Sales-Predictive-Analytics.ipynb](Retail-Sales-Predictive-Analytics.ipynb)**  \nWith PySpark, we predict monthly sales by city using a linear regression model. The experiments were tracked using MLflow, and the trained model was saved for future predictions.\n\n### Data Lake Architecture  \n- **Ingest**  \n  - The raw CSV data is uploaded to DBFS at a path such as \n    `dbfs:/FileStore/salesdata/input/`.\n  - This location represents the ***raw zone*** of our data lake.  \n- **Transform**  \n  - Using PySpark in a Databricks notebook, the data was loaded from the raw zone.\n  - We cast columns (e.g. Quantity Ordered from `String` to `Int`), clean corrupted rows, added new columns, and fix other anomalies.  \n- **Publish**  \n  - The refined data is saved as Parquet / Delta Lake files in \n    `dbfs:/FileStore/salesdata/published/`\n  - This acts as the ***curated zone*** of the data lake, optimized for analytics.  \n- **Analyze**  \n  - The published data is queried via Databricks SQL or Spark DataFrames, enabling fast aggregations and computations.  \n- **Prediction**  \n  - Using PySpark ML, a pipeline transforms features into a format suitable for ML model training.\n  - Define a linear regression model to a workflow that predicts retail sales while tracking the experiments with evaluation metrics and logging.\n- **Deployment**\n  - From the MLflow tracking server, access the trained model to make real-time predictions. \n\n### Data Preparation  \nThe Retail-Sales-Data-Preparation.ipynb notebook includes:  \n- Reading the Raw CSV Files\n- Cleaning \u0026 Transformations\n  - Removing header rows that slipped into the data\n  - Converting columns to proper data types\n  - Splitting out `Purchase Address` into `City`, `State`, etc. \n- Writing to the Curated Zone\n\n### Data Analysis  \nAnalysis was performed via PySpark DataFrame APIs and SQL:\n- **Exploratory Queries**\n  - Finding total revenue by month\n  - Pinpoint peak ordering times \n  - Identifying popular product combinations in sales orders  \n- **Advanced Analytics**\n  - Partitioning data by Year/Month for efficient queries\n  - Summarizing results in aggregated tables  \n\n### Visualizations \u0026 Dashboard  \nThe final step was creating a dashboard in Databricks to visualize:\n- **Monthly Revenue Trends**\n- **Peak Ordering Times**\n- **Geographic Breakdown of Orders**\n- **Popular Product Combinations**\n\n### Machine Learning Model \u0026 MLflow Tracking   \n- Preprocessed the data with feature engineering techniques \n- Train Linear Regression model to predict monthly sales by city\n- Logged experiments in MLflow with evaluation metrics\n- Saved model and loaded it for future sales predictions\n- Deployed MLflow pipeline to track and compare multiple models\n\n### How to Reproduce  \n- **Clone Repository**  \n   `git clone https://github.com/Cesar312/retail-sales-data-lake-predictive-analytics.git`\n- **Create Databricks Clusters**  \n   - Standard cluster for the EDA and SQL queries\n   - ML cluster for model training, experiment tracking, and deployment of the prediction model  \n- **Upload Data to Databricks**  \n   - Upload raw CSV files to `dbfs:/FileStore/salesdata/input/`\n- **Run the PySpark Notebook**  \n   - Open `Retail-Sales-Data-Preparation.ipynb`\n   - Execute cells to clean and publish data\n- **Train the Model**  \n   - Open `Retail-Sales-Predictive-Analytics.ipynb`\n   - Execute cells to train and track experiments\n   - View the evaluation metrics in MLflow UI\n- **Predict Future Sales**  \n   - Load the trained model\n   - Make predictions for new months\n\n### Future Improvements  \n- Automate model retraining on new data updates\n- Test advanced models like XGBoost or Time-Series Forecasting\n- Create a backend API microservice to serve predictions by deploying the trained ML model retrieved from the Databricks MLflow registry\n  \n### License  \nThis project is licensed under the MIT License -- see the [LICENSE.txt](LICENSE.txt) file for details.  \n\n#### Databricks Community Edition\nThis project was developed and executed using [Databricks Community Edition](https://community.cloud.databricks.com/).\nThe code and queries are shared for educational purposes. Databricks itself is a proprietary platform, and\nthis repository does not include any proprietary Databricks software.\n\n###### Last Modified Date: 2025-04-01  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcesar312%2Fretail-sales-data-lake-predictive-analytics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcesar312%2Fretail-sales-data-lake-predictive-analytics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcesar312%2Fretail-sales-data-lake-predictive-analytics/lists"}