{"id":30151526,"url":"https://github.com/beatrice-127/ecommerce-data-etl","last_synced_at":"2026-05-02T10:43:36.491Z","repository":{"id":306282979,"uuid":"1024503135","full_name":"Beatrice-127/ecommerce-data-ETL","owner":"Beatrice-127","description":"An Azure-based ETL project for e-commerce data pipelines, including data extraction, transformation, and loading workflows using cloud tools.","archived":false,"fork":false,"pushed_at":"2025-07-31T18:36:36.000Z","size":776,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-11T11:02:26.651Z","etag":null,"topics":["azure","data-cleaning","data-engineering","data-transformation","databricks","etl","postgresql","spark","sql"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Beatrice-127.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-22T19:58:46.000Z","updated_at":"2025-07-31T18:36:40.000Z","dependencies_parsed_at":"2025-07-24T20:37:54.999Z","dependency_job_id":"f887f741-f746-4233-91bd-3fa5538602b2","html_url":"https://github.com/Beatrice-127/ecommerce-data-ETL","commit_stats":null,"previous_names":["beatrice-127/ecommerce-data-etl"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Beatrice-127/ecommerce-data-ETL","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Beatrice-127%2Fecommerce-data-ETL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Beatrice-127%2Fecommerce-data-ETL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Beatrice-127%2Fecommerce-data-ETL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Beatrice-127%2Fecommerce-data-ETL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Beatrice-127","download_url":"https://codeload.github.com/Beatrice-127/ecommerce-data-ETL/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Beatrice-127%2Fecommerce-data-ETL/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269873158,"owners_count":24488993,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-11T02:00:10.019Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","data-cleaning","data-engineering","data-transformation","databricks","etl","postgresql","spark","sql"],"created_at":"2025-08-11T11:02:17.252Z","updated_at":"2026-05-02T10:43:36.446Z","avatar_url":"https://github.com/Beatrice-127.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"#  E-Commerce Data Pipeline (Azure ETL Project)\n\n\n\nThis project simulates a **production-grade end-to-end data pipeline** built on **Azure Data Services**, using the **Medallion Architecture** (Bronze → Silver → Gold). It transforms messy, fragmented e-commerce data into analytics-ready layers that support BI dashboards and machine learning workflows.\n\nThe pipeline ingests data from the [Brazilian E-Commerce Public Dataset by Olist](https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce), which includes over 100,000 anonymized orders, covering customer behavior, product attributes, reviews, and shipping logistics.\n\n\n\n## Architecture Overview\n\n\u003cimg src=\"images/ecomm_engineer.png\" alt=\"pipeline\" width=\"80%\"/\u003e\n\nThis pipeline follows a layered architecture with well-defined separation of concerns:\n\n1. **Ingestion**: Raw data from CSV files is ingested using **Azure Data Factory** into **ADLS Gen2 (Bronze layer)**.\n2. **Processing**: **Azure Databricks (PySpark)** transforms and joins datasets into the **Silver layer**.\n3. **Enrichment**: External lookup data from **MongoDB** is merged to add business context (e.g., category names).\n4. **Analytics**: The **Gold layer** is used by **Azure Synapse** and **Tableau** for BI and modeling purposes.\n\n\n\n### What's next?\n\nI believe this dataset opens the door to many exciting analytical possibilities. Here’s what I plan to build next:\n\n- [ ] **Interactive KPI Dashboard** – Built with Tableau to monitor core business metrics.\n- [x] **RFM Segmentation** – Using Recency, Frequency, and Monetary value to group customers via visualization and **unsupervised learning** algorithms.\n- [ ] **Customer behavior** – Combining the [Marketing Funnel by Olist](https://www.kaggle.com/datasets/olistbr/marketing-funnel-olist) to analysis how to promote the business.\n\n---\n\nStay tuned! This project is continuously evolving as I explore new ideas and techniques. 😊\n\n\n\n\n\n## Data Ingestion: Raw Data to Bronze layer\n\nThe **Bronze layer** typically serves as the initial landing zone for data originating from external source systems. In this project, to better mimic a real-world scenario, data is sourced from **GitHub**, a **SQL Server**, and **MongoDB**.\n\nTo ingest data from GitHub and SQL Server, we use **Azure Data Factory (ADF)**, with the following pipeline:\n\n![ADFpipeline](images/ADFpipeline.png)\n\nThis pipeline accesses the raw data URLs using a `ForEach` loop, where the source links are stored in a **JSON configuration file**. This setup makes future updates easier and more scalable.\n\nWhile using such a pipeline to upload just a few small files might seem like overkill, the process helped me understand how to configure **linked services**, **datasets**, and **permissions** in Azure — which actually took quite a bit of time to get right!\n\nHowever, imagine managing dozens of data sources or setting up a pipeline that runs daily — this structure quickly becomes not only necessary but powerful.\n\n\n\n\n\n## Data Transformation: Bronze layer to Silver layer\n\nThe **Silver layer** is where raw data from the Bronze layer undergoes a series of transformation steps to reach a “just-enough” state — ready for analytics and business logic. It provides an integrated **enterprise view** of key entities, transactions, and relationships.\n\nBut what does “just-enough” really mean? It depends on the use case. Since I’m currently the sole user of this pipeline, and given the low cost of modern cloud storage, I choose to retain as much data as possible and defer heavier transformations until they’re actually needed.\n\nAll transformations are performed on **Azure Databricks using PySpark**, and the resulting data is stored in **ADLS Gen2 in `PARQUET` format** for optimized storage and query performance.\n\n\n\n### Data Cleaning \u0026 Validation\n\n- **Handle missing values**: Drop rows containing null from critical tables (e.g. `orders`, `order_items`) where necessary.\n\n  ```python\n  def clean_dataframe(df, name):\n      print(f\"Cleaning {name}\")\n      return df.dropDuplicates().na.drop('all')\n  \n  orders_df = clean_dataframe(orders_df, 'orders')\n  items_df = clean_dataframe(items_df, 'order_items')\n  ```\n\n  For `review_df`, many rows contain nulls — which reflects a realistic business scenario where users often don’t leave reviews. These rows are preserved.\n\n\n- **Remove duplicates**: Especially in `geolocation_df`, where the data is heavily duplicated due to anonymization.\n\n  ```python\n  total = geolocation_df.count()\n  unique = geolocation_df.dropDuplicates().count()\n  print(f\"Total Rows: {total}\")\n  print(f\"Duplicated Rows: {total - unique}\")\n  print(f\"% of Duplication: {(total - unique) / total:.2%}\")\n  \n  # Output:\n  # Total Rows: 1000163\n  # Duplicated Rows: 261831\n  # % of Duplication: 26.18%\n  ```\n\n  Drop duplicates:\n\n  ```python\n  geolocation_df = geolocation_df.dropDuplicates()\n  ```\n\n\n- **Data type conversion**: For example, converting `order_delivered_customer_date` from timestamp to date:\n\n  ```python\n  orders_df = orders_df.withColumn(\n      \"order_delivered_customer_date\", \n      to_date(col(\"order_delivered_customer_date\"))\n  )\n  ```\n\n- **Delivery Time Features**: To better analyze delivery performance, some features are created. For example:\n\n  ```python\n  orders_df = orders_df.withColumn(\"actual_delivery_time\", \n                  datediff(\"order_delivered_customer_date\", \"order_purchase_timestamp\"))\n  ```\n  \n\n\n\n### Data conforming\n\nData from **MongoDB** has a different schema and structure. We convert it to a Spark DataFrame as follows:\n\n```python\nclient = MongoClient(uri)\ncollection = client[database]['product_categories']\n\nmongo_data = pd.DataFrame(list(collection.find()))\nmongo_data.drop('_id', axis=1, inplace=True)\nmongo_spark_df = spark.createDataFrame(mongo_data)\n```\n\n\n\n### Data matching for integration purpose\n\n- **Join datasets**: All datasets (except `review_df`) are joined into `final_df`, resulting in a unified dataset with **37 features** and **118,434 rows** — ready for analytics or further processing.\n\n\n\n## Data Serving: Silver layer to Gold layer\n\nThe **Gold layer** is the analytics-ready version of the data — cleaned, enriched, and structured for direct use in BI dashboards and machine learning models.\n\n###  Initial Plan: Synapse (Failed... for now)\n\nOriginally, I intended to connect Azure Synapse to my **ADLS Gen2 Silver layer**, so that the data could be directly used in **Power BI**.\n\nHowever, I encountered persistent access errors despite correctly configuring permissions (e.g., setting Synapse as a Contributor to the storage account):\n\n```sql\nCREATE SCHEMA gold;\n\nCREATE VIEW gold.final AS\nSELECT *\nFROM OPENROWSET(\n    BULK 'https://olistdatastoragelibei.dfs.core.windows.net/olistdata/silver/',\n    FORMAT = 'PARQUET'\n) AS result1;\n```\n\nError message:\n\n```sql\nContent of directory on path 'https://olistdatastoragelibei.dfs.core.windows.net/olistdata/silver/' cannot be listed.\n```\n\nI tried everything I could find online (permissions, networking, identity...), but nothing worked.  \nI'm currently contacting Microsoft support — if you’ve solved this before, please reach out!\n\n\n\n### Plan B: PostgreSQL on Localhost\n\nFor now, I’ve exported the **Silver layer output** to **PostgreSQL** on my laptop and built the **Gold layer** there.\n\nAll queries are documented in the `.sql` files in this project. Here's a sample of what I did:\n\n####  Create New Tables\n\nTo support time-based aggregations, I created a new table with extracted time features:\n\n```sql\nCREATE TABLE order_time AS\nSELECT \n    order_id,\n    customer_id,\n    customer_unique_id,\n    EXTRACT(YEAR FROM order_purchase_timestamp) AS y,\n    EXTRACT(QUARTER FROM order_purchase_timestamp) AS q,\n    EXTRACT(MONTH FROM order_purchase_timestamp) AS m,\n    DATE(order_purchase_timestamp) AS d,\n    EXTRACT(HOUR FROM order_purchase_timestamp) AS h\nFROM olist_data\nWHERE TO_CHAR(order_purchase_timestamp, 'YYYY-MM') \n      NOT IN ('2016-09', '2016-12', '2018-09', '2018-10');  -- outliers\n```\n\n\u003e These four months contained only 25 rows, which seemed anomalous and were excluded.\n\n\n\n#### Create Views for Business Use\n\nSome views are created for Business Use. \n\n##### ``order_time_and_value``: Enabling KPIs like MAU, GMV, ARPU\n\n```sql\nCREATE VIEW order_time_and_value AS\nSELECT\n\ta.order_id,\n\ta.customer_id,\n\ta.customer_unique_id,\n\tround(a.total_order_value::numeric, 2) as total_order_value,\n\ty,q,m,d,h\n\tFROM order_with_customer_summary a\n\tINNER JOIN order_time b\n\tON a.order_id=b.order_id;\n```\nThis view enables tracking key business metrics such as **Monthly Active Users (MAU)**\n```sql\n-- MAU\nSELECT \n    TO_CHAR(d, 'YYYY-MM') AS month,\n    COUNT(DISTINCT customer_unique_id) AS MAU\nFROM order_time_and_value\nGROUP BY TO_CHAR(d, 'YYYY-MM')\nORDER BY month;\n```\nWith a simple visualization generated in **pgAdmin 4**, we can already begin to observe meaningful patterns in customer activity:\n\n\u003cimg src=\"images/mau.png\" alt=\"MAU\" width=\"60%\"/\u003e\n\n\u003e Olist may be in trouble — growth appears to have stalled and may even be reversing. We'll explore this further in the upcoming analysis.\n\n\n\n##### `rfm_scored`: Customer Segmentation via RFM\n\nViews were created for **RFM analysis**, using quantile scoring:\n\n  ```sq\n  CREATE VIEW rfm_scored AS\n  SELECT\n  \tcustomer_unique_id,\n      recency,\n  \tfrequency,\n      monetary,\n  \n      -- scoring:\n  \t6 - NTILE(5) OVER (ORDER BY recency) AS r_score,   \n  \tNTILE(5) OVER (ORDER BY frequency) AS f_score,     \n      NTILE(5) OVER (ORDER BY monetary) AS m_score       \n  \n  FROM rfm_base\n  ```\n\n These views are designed to feed into future customer segmentation and marketing analysis.\n\n\n\n\n\n## Governance \u0026 Access Control\n\nA production-grade data pipeline isn’t just about transformation and delivery — it also requires thoughtful governance. In this project, access control is designed to reflect real-world data security practices, leveraging **Access Control (IAM)** in Azure and **Microsoft Entra ID** for identity management.\n\n- **Bronze layer** access is restricted to ingestion users only.\n\n-  **Silver and Gold layers** are accessible to BI and analytics teams.\n-  Data stored in **Parquet format**, combined with **Synapse views**, lays the foundation for implementing **row-level security** in future stages.\n\nThis access model follows the principle of **least privilege**, ensuring that users and services only have access to the data they actually need.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbeatrice-127%2Fecommerce-data-etl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbeatrice-127%2Fecommerce-data-etl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbeatrice-127%2Fecommerce-data-etl/lists"}