{"id":50325104,"url":"https://github.com/johnathon-smith/global_partners_orders_aws_pipeline","last_synced_at":"2026-05-29T05:30:28.093Z","repository":{"id":359147590,"uuid":"1243956715","full_name":"johnathon-smith/global_partners_orders_aws_pipeline","owner":"johnathon-smith","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-20T18:36:10.000Z","size":139,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-20T20:26:15.363Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/johnathon-smith.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-19T20:39:22.000Z","updated_at":"2026-05-20T18:36:14.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/johnathon-smith/global_partners_orders_aws_pipeline","commit_stats":null,"previous_names":["johnathon-smith/global_partners_orders_aws_pipeline"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/johnathon-smith/global_partners_orders_aws_pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnathon-smith%2Fglobal_partners_orders_aws_pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnathon-smith%2Fglobal_partners_orders_aws_pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnathon-smith%2Fglobal_partners_orders_aws_pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnathon-smith%2Fglobal_partners_orders_aws_pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/johnathon-smith","download_url":"https://codeload.github.com/johnathon-smith/global_partners_orders_aws_pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnathon-smith%2Fglobal_partners_orders_aws_pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33639050,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-29T02:00:06.066Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-29T05:30:27.316Z","updated_at":"2026-05-29T05:30:28.084Z","avatar_url":"https://github.com/johnathon-smith.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS Glue + Delta Lake Analytics Pipeline\n\n![Project Status](https://img.shields.io/badge/status-complete-brightgreen)\n![AWS](https://img.shields.io/badge/cloud-AWS-orange)\n![Delta Lake](https://img.shields.io/badge/storage-Delta%20Lake-blue)\n![Python](https://img.shields.io/badge/python-3.11-blue)\n![Streamlit](https://img.shields.io/badge/dashboard-Streamlit-red)\n![CI/CD](https://img.shields.io/badge/CI%2FCD-GitHub%20Actions-black)\n\n## Overview\n\nThis project is an end-to-end AWS data engineering pipeline that ingests restaurant order data from **Microsoft SQL Server on AWS RDS**, processes it through a medallion-style **S3 Delta Lake**, and serves curated analytics through a **Streamlit Cloud dashboard**.\n\nThe pipeline is designed to demonstrate practical, production-aware data engineering skills while staying focused enough for a portfolio project. It includes batch ingestion, Delta Lake table management, dimensional modeling, SCD Type 2 logic, curated analytics tables, dashboard consumption, observability, alerting, and CI/CD deployment for AWS Glue scripts.\n\nThe final system includes:\n\n* AWS RDS SQL Server as the source system\n* AWS Glue Workflow for orchestration and scheduling\n* AWS Glue Spark jobs for ingestion and transformations\n* Amazon S3 as the data lake\n* Delta Lake for raw, refined, and curated table storage\n* SCD Type 2 user dimension tracking loyalty changes over time\n* Curated analytics tables for customer, churn, loyalty, location, and timing analysis\n* Streamlit Cloud dashboard reading curated Delta tables directly from S3\n* CloudWatch logging and metrics\n* EventBridge failure detection\n* SNS email alerts\n* SQS dead-letter queue for failed alert delivery\n* GitHub Actions CI/CD with Ruff validation and S3 script deployment\n* GitHub OIDC authentication with a least-privilege AWS IAM role\n\n---\n\n## Table of Contents\n\n* [Architecture](#architecture)\n* [Business Problem](#business-problem)\n* [Tech Stack](#tech-stack)\n* [Data Pipeline Flow](#data-pipeline-flow)\n* [Source Data](#source-data)\n* [Data Lake Design](#data-lake-design)\n* [Refined Data Model](#refined-data-model)\n* [Curated Analytics Layer](#curated-analytics-layer)\n* [Streamlit Dashboard](#streamlit-dashboard)\n* [Observability and Alerting](#observability-and-alerting)\n* [CI/CD Implementation](#cicd-implementation)\n* [Repository Structure](#repository-structure)\n* [Key Engineering Decisions](#key-engineering-decisions)\n* [Security Notes](#security-notes)\n* [Interview Talking Points](#interview-talking-points)\n* [Future Improvements](#future-improvements)\n\n---\n\n## Architecture\n\n![Final Architecture Diagram](docs/images/architecture/final-architecture-diagram.png)\n\nThe project uses AWS Glue Workflow as the central orchestrator. Glue jobs extract data from SQL Server, write ingestion files to S3, convert those files into Delta Lake raw tables, transform the data into refined fact and dimension tables, and then create dashboard-ready curated analytical tables.\n\nHigh-level flow:\n\n```text\nAWS RDS SQL Server\n        ↓\nAWS Glue Workflow\n        ↓\nGlue Ingestion Jobs\n        ↓\nS3 Ingestion Layer - Parquet\n        ↓\nGlue Raw Jobs\n        ↓\nS3 Raw Layer - Delta Lake\n        ↓\nGlue Refined Jobs\n        ↓\nS3 Refined Layer - Delta Lake\n        ↓\nGlue Curated Jobs\n        ↓\nS3 Curated Layer - Delta Lake\n        ↓\nStreamlit Cloud Dashboard\n```\n\nSupporting production-style components:\n\n```text\nCloudWatch Logs and Metrics\nEventBridge Glue failure detection\nSNS email alerts\nSQS dead-letter queue for failed alert delivery\nGitHub Actions CI/CD\nGitHub OIDC AWS deployment role\n```\n\nImportant architecture clarification:\n\nGlue Workflow handles pipeline orchestration and scheduling. EventBridge is not used to schedule or orchestrate the data pipeline. EventBridge is only used as an observability component that listens for failed Glue job state changes and routes those events to SNS.\n\n---\n\n## Business Problem\n\nThe source system contains restaurant order activity, customer loyalty information, item-level order details, option/customization data, and calendar metadata. The goal is to transform this operational data into analytics-ready tables that can answer questions such as:\n\n* Which customers generate the most lifetime value?\n* Which customers are VIPs, active, new, or at risk of churn?\n* Which customers have not ordered recently?\n* Which restaurant locations generate the most revenue?\n* Do loyalty-associated orders behave differently from non-loyalty orders?\n* Which dayparts, weekends, weekdays, and holidays perform best?\n\nInstead of putting heavy business logic directly inside the dashboard, this project precomputes curated analytical tables in AWS Glue. This keeps the Streamlit app lightweight and makes the business logic easier to validate.\n\n---\n\n## Tech Stack\n\n| Area                  | Technology                       |\n| --------------------- | -------------------------------- |\n| Source Database       | AWS RDS for Microsoft SQL Server |\n| Orchestration         | AWS Glue Workflow                |\n| Processing            | AWS Glue Spark Jobs, PySpark     |\n| Storage               | Amazon S3                        |\n| Table Format          | Delta Lake                       |\n| Dashboard             | Streamlit Cloud                  |\n| Dashboard Data Access | deltalake, pyarrow, pandas       |\n| Visualization         | Streamlit, Altair                |\n| Logging               | Amazon CloudWatch Logs           |\n| Failure Detection     | Amazon EventBridge               |\n| Alerts                | Amazon SNS                       |\n| Alert DLQ             | Amazon SQS                       |\n| CI/CD                 | GitHub Actions                   |\n| Code Quality          | Ruff                             |\n| AWS Auth for CI/CD    | GitHub OIDC + IAM Role           |\n\n---\n\n## Data Pipeline Flow\n\n### 1. Ingestion Layer\n\nGlue ingestion jobs connect to SQL Server on AWS RDS using JDBC and extract the source tables into S3 as Parquet files.\n\nSource tables:\n\n```text\ndate_dim\norder_items\norder_item_options\n```\n\nIngestion layer paths:\n\n```text\ns3://global-partners-data-bucket-001/ingestion/date_dim/\ns3://global-partners-data-bucket-001/ingestion/order_items/\ns3://global-partners-data-bucket-001/ingestion/order_item_options/\n```\n\nThe ingestion layer is the landing zone for source extracts.\n\n---\n\n### 2. Raw Layer\n\nRaw Glue jobs convert ingestion Parquet files into Delta Lake tables while keeping the data close to the original source structure.\n\nRaw layer paths:\n\n```text\ns3://global-partners-data-bucket-001/raw/date_dim/\ns3://global-partners-data-bucket-001/raw/order_items/\ns3://global-partners-data-bucket-001/raw/order_item_options/\n```\n\nRaw tables use Delta Lake format and include Delta transaction logs under `_delta_log/`.\n\nRaw write pattern:\n\n```text\nappend\n```\n\nRaw metadata columns include:\n\n```text\nload_dts\n```\n\n---\n\n### 3. Refined Layer\n\nThe refined layer transforms raw source-aligned data into clean analytical fact and dimension tables.\n\nRefined layer paths:\n\n```text\ns3://global-partners-data-bucket-001/refined/fact_orders/\ns3://global-partners-data-bucket-001/refined/dim_dates/\ns3://global-partners-data-bucket-001/refined/dim_users/\ns3://global-partners-data-bucket-001/refined/dim_order_items/\ns3://global-partners-data-bucket-001/refined/dim_options/\n```\n\nThe refined layer applies:\n\n* 2023 filtering\n* column casting\n* field standardization\n* dimensional modeling\n* Delta Lake merge/upsert logic\n* SCD Type 2 logic for `dim_users`\n* point-in-time `user_sk` lookup for `fact_orders`\n\n---\n\n### 4. Curated Layer\n\nThe curated layer turns refined fact and dimension data into dashboard-ready analytical tables.\n\nCurated layer paths:\n\n```text\ns3://global-partners-data-bucket-001/curated/customer_ltv/\ns3://global-partners-data-bucket-001/curated/customer_segmentation/\ns3://global-partners-data-bucket-001/curated/churn_risk/\ns3://global-partners-data-bucket-001/curated/location_performance/\ns3://global-partners-data-bucket-001/curated/loyalty_impact/\ns3://global-partners-data-bucket-001/curated/order_timing_analysis/\n```\n\nThese tables are read directly by the Streamlit dashboard.\n\n---\n\n## Source Data\n\nThe source system contains three SQL Server tables.\n\n### `date_dim`\n\nCalendar dimension data for every day of 2023.\n\n| Column         | Description                         |\n| -------------- | ----------------------------------- |\n| `date_key`     | Full calendar date                  |\n| `day_of_week`  | Day of the week                     |\n| `week`         | Week number in the year             |\n| `month`        | Month name                          |\n| `year`         | Calendar year                       |\n| `is_weekend`   | Whether the date falls on a weekend |\n| `is_holiday`   | Whether the date is a holiday       |\n| `holiday_name` | Name of the holiday, if applicable  |\n\n---\n\n### `order_items`\n\nFood order line item data. The source table spans multiple years, but this project processes 2023 records.\n\n| Column                | Description                             |\n| --------------------- | --------------------------------------- |\n| `app_name`            | Ordering platform or channel            |\n| `restaurant_id`       | Restaurant/location identifier          |\n| `creation_time_utc`   | Order timestamp in UTC                  |\n| `order_id`            | Unique order identifier                 |\n| `user_id`             | Customer identifier                     |\n| `printed_card_number` | Tokenized or masked loyalty card number |\n| `is_loyalty`          | Loyalty membership flag                 |\n| `currency`            | Transaction currency                    |\n| `lineitem_id`         | Unique order line item identifier       |\n| `item_category`       | Item category                           |\n| `item_name`           | Item name                               |\n| `item_price`          | Item price                              |\n| `item_quantity`       | Quantity ordered                        |\n\nImportant business logic note:\n\nFor this project, `order_total` is calculated as:\n\n```text\nsum(item_price) grouped by order_id\n```\n\nIf `item_price` were later confirmed to represent a unit price instead of a line-item amount, the more accurate formula may become:\n\n```text\nsum(item_price * item_quantity)\n```\n\n---\n\n### `order_item_options`\n\nOption/customization data tied to order line items.\n\n| Column              | Description                          |\n| ------------------- | ------------------------------------ |\n| `order_id`          | Parent order identifier              |\n| `lineitem_id`       | Parent line item identifier          |\n| `option_group_name` | Option group/category                |\n| `option_name`       | Selected option                      |\n| `option_price`      | Option price                         |\n| `option_quantity`   | Number of times the option was added |\n\n---\n\n## Data Lake Design\n\nFinal S3 data lake structure:\n\n```text\ns3://global-partners-data-bucket-001/\n  ingestion/\n    date_dim/\n    order_items/\n    order_item_options/\n\n  raw/\n    date_dim/\n    order_items/\n    order_item_options/\n\n  refined/\n    fact_orders/\n    dim_dates/\n    dim_users/\n    dim_order_items/\n    dim_options/\n\n  curated/\n    customer_ltv/\n    customer_segmentation/\n    churn_risk/\n    location_performance/\n    loyalty_impact/\n    order_timing_analysis/\n\n  quarantine/\n\n  scripts/\n    glue_jobs/\n      ingestion/\n      raw/\n      refined/\n      curated/\n```\n\nLayer summary:\n\n| Layer      | Format     | Purpose                                             |\n| ---------- | ---------- | --------------------------------------------------- |\n| Ingestion  | Parquet    | Landing zone for source extracts                    |\n| Raw        | Delta Lake | Source-aligned Delta tables with ingestion metadata |\n| Refined    | Delta Lake | Clean fact and dimension tables                     |\n| Curated    | Delta Lake | Dashboard-ready analytical tables                   |\n| Quarantine | Reserved   | Location for invalid/problem records                |\n| Scripts    | Python     | Glue scripts deployed by GitHub Actions             |\n\n---\n\n## Refined Data Model\n\n![Refined Data Model](docs/images/s3/s3-data-lake-refined-layer.png)\n\nThe refined layer contains one main fact table and four supporting dimension/detail tables.\n\n```text\nfact_orders\ndim_dates\ndim_users\ndim_order_items\ndim_options\n```\n\n---\n\n### `fact_orders`\n\nMain order-level fact table.\n\nGrain:\n\n```text\nOne row per order_id\n```\n\nFields:\n\n```text\napp_name\nrestaurant_id\norder_id\nuser_sk\nuser_id\norder_date\norder_timestamp\norder_total\nnum_unique_lineitems\nnum_order_items\ncurrency\norder_year\norder_month\netl_updated_at\n```\n\nKey logic:\n\n* Aggregates `order_items` to one row per order\n* Calculates `order_total` as `sum(item_price)` by `order_id`\n* Calculates item counts and quantities\n* Joins to `dim_users` to retrieve the correct point-in-time `user_sk`\n* Partitioned by `order_year` and `order_month`\n\nMerge key:\n\n```text\norder_id\n```\n\n---\n\n### `dim_users`\n\nSCD Type 2 user dimension that tracks loyalty attribute changes over time.\n\nGrain:\n\n```text\nOne row per user version\n```\n\nFields:\n\n```text\nuser_sk\nuser_id\nprinted_card_number\nis_loyalty\neffective_start_date\neffective_end_date\nis_current\nrecord_hash\netl_updated_at\n```\n\nSCD2 tracked fields:\n\n```text\nprinted_card_number\nis_loyalty\n```\n\nWhy SCD Type 2 is used:\n\nThe source data showed that loyalty fields can change for the same `user_id`. The pipeline preserves historical loyalty status so downstream analytics can answer point-in-time questions such as:\n\n```text\nWas this customer a loyalty member at the time of this order?\n```\n\nSurrogate key logic:\n\n```text\nuser_sk = hash(user_id + effective_start_date + record_hash)\n```\n\n---\n\n### `dim_dates`\n\nCalendar dimension for 2023.\n\nGrain:\n\n```text\nOne row per date\n```\n\nFields:\n\n```text\ndate_key\nday_of_week\nweek\nmonth\nyear\nis_weekend\nis_holiday\nholiday_name\netl_updated_at\n```\n\nMerge key:\n\n```text\ndate_key\n```\n\n---\n\n### `dim_order_items`\n\nLine-item detail table connected to the order-level fact table.\n\nGrain:\n\n```text\nOne row per order_id + lineitem_id\n```\n\nFields:\n\n```text\norder_id\nlineitem_id\nitem_category\nitem_name\nitem_price\nitem_quantity\netl_updated_at\n```\n\nMerge key:\n\n```text\norder_id + lineitem_id\n```\n\n---\n\n### `dim_options`\n\nOption/customization detail table for order line items.\n\nGrain:\n\n```text\nOne row per selected option/customization for a line item\n```\n\nFields:\n\n```text\norder_id\nlineitem_id\noption_group_name\noption_name\noption_price\noption_quantity\netl_updated_at\n```\n\nMerge key:\n\n```text\norder_id + lineitem_id + option_group_name + option_name\n```\n\n---\n\n## Curated Analytics Layer\n\n![Curated Layer](docs/images/s3/s3-data-lake-curated-layer.png)\n\nThe curated layer contains six dashboard-ready analytical Delta tables.\n\n```text\ncustomer_ltv\ncustomer_segmentation\nchurn_risk\nlocation_performance\nloyalty_impact\norder_timing_analysis\n```\n\n---\n\n### `customer_ltv`\n\nPurpose:\n\nAnalyze lifetime customer value.\n\nGrain:\n\n```text\nOne row per user_id\n```\n\nFields:\n\n```text\nuser_id\nltv\nltv_category\nnum_orders\navg_order_value\netl_updated_at\npipeline_run_id\n```\n\nBusiness logic:\n\n```text\nltv = sum(order_total)\nnum_orders = count distinct order_id\navg_order_value = average order_total\n```\n\nLTV category logic:\n\n```text\nTop 20% by LTV = High\nMiddle 60% by LTV = Medium\nBottom 20% by LTV = Low\n```\n\n---\n\n### `customer_segmentation`\n\nPurpose:\n\nSegment customers using recency, frequency, and monetary behavior.\n\nGrain:\n\n```text\nOne row per user_id\n```\n\nFields:\n\n```text\nuser_id\ndays_since_last_order\nnum_orders_last_3_months\ntotal_spend_last_3_months\nsegment\netl_updated_at\npipeline_run_id\n```\n\nSegment labels:\n\n```text\nVIP\nNew Customer\nChurn Risk\nActive\n```\n\nBecause the dataset is historical, the analysis anchor date is the max `order_date` in `fact_orders`, not the real current date.\n\n---\n\n### `churn_risk`\n\nPurpose:\n\nIdentify customers who may be at risk based on inactivity and spend changes.\n\nGrain:\n\n```text\nOne row per user_id\n```\n\nFields:\n\n```text\nuser_id\ndays_since_last_order\navg_days_between_orders\npercent_change_in_spend\nat_risk\netl_updated_at\npipeline_run_id\n```\n\nAt-risk logic:\n\n```text\nat_risk = true when days_since_last_order \u003e= 45\n```\n\nPercent change in spend compares the most recent rolling 90-day period to the previous rolling 90-day period.\n\n---\n\n### `location_performance`\n\nPurpose:\n\nAnalyze restaurant/location performance.\n\nGrain:\n\n```text\nOne row per restaurant_id for the full analysis period\n```\n\nFields:\n\n```text\nrestaurant_id\ntotal_revenue\navg_order_value\navg_orders_per_day\navg_orders_per_week\nrevenue_rank\netl_updated_at\npipeline_run_id\n```\n\nRevenue ranking uses `dense_rank()` so tied restaurants do not create gaps in rank values.\n\n---\n\n### `loyalty_impact`\n\nPurpose:\n\nCompare purchasing behavior between loyalty and non-loyalty activity.\n\nGrain:\n\n```text\nOne row per is_loyalty value\n```\n\nFields:\n\n```text\nis_loyalty\ncustomer_count\ntotal_orders\ntotal_revenue\navg_ltv\navg_order_value\navg_orders_per_customer\nrevenue_per_customer\netl_updated_at\npipeline_run_id\n```\n\nThis table joins:\n\n```text\nfact_orders.user_sk = dim_users.user_sk\n```\n\nThis allows the dashboard to compare loyalty status at the time of the order using the SCD Type 2 user dimension.\n\nImportant caveat:\n\nThis table shows correlation between loyalty status and purchasing behavior. It does not prove the loyalty program caused higher spending.\n\n---\n\n### `order_timing_analysis`\n\nPurpose:\n\nAnalyze order behavior by date and time-of-day segment.\n\nGrain:\n\n```text\nOne row per order_date + daypart\n```\n\nFields:\n\n```text\norder_date\norder_year\norder_month\nday_of_week\nmonth\nis_weekend\nis_holiday\nholiday_name\ndaypart\ntotal_orders\ntotal_revenue\navg_order_value\nunique_customers\netl_updated_at\npipeline_run_id\n```\n\nDaypart logic:\n\n```text\nMorning: 6:00 AM - 10:59 AM\nLunch: 11:00 AM - 1:59 PM\nAfternoon: 2:00 PM - 4:59 PM\nDinner: 5:00 PM - 8:59 PM\nLate Night: 9:00 PM - 5:59 AM\n```\n\nPartitioning:\n\n```text\norder_year\norder_month\n```\n\n---\n\n## Streamlit Dashboard\n\n![Streamlit Executive Overview](docs/images/streamlit/executive-overview.png)\n\nThe Streamlit dashboard is deployed to Streamlit Cloud and reads curated Delta tables directly from S3 using the `deltalake` package.\n\nDashboard tabs:\n\n```text\nExecutive Overview\nCustomer LTV\nCustomer Segmentation\nChurn Risk\nLocation Performance\nLoyalty Impact\nOrder Timing Analysis\n```\n\nThe app uses Streamlit secrets for AWS credentials and Delta table paths.\n\nExample Delta reading pattern:\n\n```python\nfrom deltalake import DeltaTable\n\ndelta_table = DeltaTable(table_path, storage_options=storage_options)\ndf = delta_table.to_pandas()\n```\n\nA global helper converts `decimal.Decimal` columns to `float64` after loading each Delta table. This fixed Altair chart rendering issues caused by Decimal values from Delta Lake.\n\nDashboard screenshots:\n\n| Tab                   | Screenshot                                        |\n| --------------------- | ------------------------------------------------- |\n| Executive Overview    | `docs/images/streamlit/executive-overview.png`    |\n| Customer LTV          | `docs/images/streamlit/customer-ltv.png`          |\n| Customer Segmentation | `docs/images/streamlit/customer-segmentation.png` |\n| Churn Risk            | `docs/images/streamlit/churn-risk.png`            |\n| Location Performance  | `docs/images/streamlit/location-performance.png`  |\n| Loyalty Impact        | `docs/images/streamlit/loyalty-impact.png`        |\n| Order Timing Analysis | `docs/images/streamlit/order-timing-analysis.png` |\n\n---\n\n## Observability and Alerting\n\nThe project includes a production-aware observability layer.\n\nObservability flow:\n\n```text\nAWS Glue Jobs\n        ↓\nCloudWatch Logs and Metrics\n\nGlue Job State Change Event\n        ↓\nEventBridge Rule\n        ↓\nSNS Topic\n        ↓\nEmail Subscription\n        ↓ if alert delivery fails\nSQS Dead-Letter Queue\n        ↓\nCloudWatch Alarm on DLQ Messages\n```\n\nComponents:\n\n| Component        | Purpose                                                    |\n| ---------------- | ---------------------------------------------------------- |\n| CloudWatch Logs  | Stores Glue job output, driver, executor, and system logs  |\n| EventBridge      | Detects Glue job `FAILED`, `TIMEOUT`, and `STOPPED` states |\n| SNS              | Sends email alerts for failed Glue jobs                    |\n| SQS DLQ          | Captures failed SNS alert deliveries                       |\n| CloudWatch Alarm | Alerts when messages accumulate in the DLQ                 |\n\nEventBridge event pattern:\n\n```json\n{\n  \"source\": [\"aws.glue\"],\n  \"detail-type\": [\"Glue Job State Change\"],\n  \"detail\": {\n    \"state\": [\"FAILED\", \"TIMEOUT\", \"STOPPED\"]\n  }\n}\n```\n\nThe alerting path was tested by intentionally failing a Glue job. The failed job wrote logs to CloudWatch, EventBridge detected the failure, SNS sent an email alert, and the job was fixed and rerun successfully afterward.\n\n---\n\n## CI/CD Implementation\n\n![GitHub Actions Workflow](docs/images/cicd/github-actions-workflow.png)\n\nThe project uses GitHub Actions for lightweight CI/CD.\n\nCI/CD flow:\n\n```text\nGitHub Repository\n        ↓\nGitHub Actions Workflow\n        ↓\nValidate Python Code with Ruff\n        ↓\nDeploy Glue Scripts to S3\n        ↓\nAWS Glue Jobs Use Updated Scripts\n```\n\nWorkflow behavior:\n\n| Trigger                  | Behavior                                                      |\n| ------------------------ | ------------------------------------------------------------- |\n| Pull request to `main`   | Run Python validation only                                    |\n| Push to `main`           | Run validation, then deploy Glue scripts if validation passes |\n| Manual workflow dispatch | Allows controlled force deploy when `force_deploy=true`       |\n\nGlue scripts are deployed to:\n\n```text\ns3://global-partners-data-bucket-001/scripts/glue_jobs/\n```\n\nThe workflow uses GitHub OIDC to assume a least-privilege AWS IAM role. This avoids storing long-term AWS access keys in GitHub Secrets.\n\nThe IAM deployment role can only modify objects under:\n\n```text\ns3://global-partners-data-bucket-001/scripts/glue_jobs/*\n```\n\nIt does not have access to RDS, Glue job execution, raw data, refined data, curated data, quarantine data, or administrator actions.\n\n---\n\n## Repository Structure\n\n```text\naws-glue-delta-lake-analytics-pipeline/\n│\n├── glue_jobs/\n│   ├── ingestion/\n│   │   ├── ingestion_date_dim.py\n│   │   ├── ingestion_order_items.py\n│   │   └── ingestion_order_item_options.py\n│   │\n│   ├── raw/\n│   │   ├── raw_date_dim.py\n│   │   ├── raw_order_items.py\n│   │   └── raw_order_item_options.py\n│   │\n│   ├── refined/\n│   │   ├── refined_dim_dates.py\n│   │   ├── refined_dim_users.py\n│   │   ├── refined_dim_order_items.py\n│   │   ├── refined_dim_options.py\n│   │   └── refined_fact_orders.py\n│   │\n│   └── curated/\n│       ├── curated_customer_ltv.py\n│       ├── curated_customer_segmentation.py\n│       ├── curated_churn_risk.py\n│       ├── curated_location_performance.py\n│       ├── curated_loyalty_impact.py\n│       └── curated_order_timing_analysis.py\n│\n├── streamlit_app/\n│   ├── app.py\n│   ├── requirements.txt\n│   ├── runtime.txt\n│   ├── secrets.example.toml\n│   └── .streamlit/\n│       └── config.toml\n│\n├── docs/\n│   └── images/\n│       ├── architecture/\n│       ├── data-model/\n│       ├── s3/\n│       ├── glue/\n│       ├── streamlit/\n│       ├── observability/\n│       └── cicd/\n│\n├── .github/\n│   └── workflows/\n│       └── ci-cd.yml\n│\n├── requirements-dev.txt\n├── pyproject.toml\n├── README.md\n└── .gitignore\n```\n\n---\n\n## Key Engineering Decisions\n\n### Why AWS Glue Workflow instead of Step Functions?\n\nGlue Workflow is sufficient for this batch pipeline because all major processing tasks are AWS Glue jobs. It keeps orchestration and scheduling simple while avoiding unnecessary infrastructure. Step Functions would provide more granular workflow control in a larger production system, but for this portfolio project Glue Workflow is the better fit.\n\n---\n\n### Why Delta Lake?\n\nDelta Lake provides transaction logs, reliable table storage on S3, and merge/upsert support. This is important for the refined and curated layers because those tables need to insert new records and update existing records without blindly appending duplicates.\n\n---\n\n### Why append in raw and merge in refined/curated?\n\nThe raw layer preserves source-aligned ingested records, so it uses append. The refined and curated layers represent clean analytical entities, so they use Delta merge/upsert logic based on stable business keys.\n\n---\n\n### Why SCD Type 2 for `dim_users`?\n\nThe source data showed that loyalty attributes can change for the same user. SCD Type 2 preserves historical user versions and allows the fact table to join each order to the correct user loyalty status at the time of purchase.\n\n---\n\n### Why keep both `user_sk` and `user_id` in `fact_orders`?\n\n`user_sk` is the dimensional foreign key used to join to the correct SCD Type 2 user version. `user_id` is retained as the source-system natural key for traceability, validation, debugging, and lineage.\n\n---\n\n### Why GitHub Actions instead of AWS CodePipeline?\n\nThe CI/CD requirement is lightweight: validate Python code and upload Glue scripts to S3. GitHub Actions keeps the workflow close to the repository, avoids unnecessary AWS infrastructure, and still supports secure AWS authentication through GitHub OIDC and a least-privilege IAM role.\n\n---\n\n### Why EventBridge if Glue Workflow is the orchestrator?\n\nEventBridge is not used for orchestration or scheduling. Glue Workflow remains the orchestrator. EventBridge is only used for observability because AWS Glue emits job state change events, and EventBridge is a clean way to route failed job events to SNS for alerting.\n\n---\n\n## Security Notes\n\nThis project uses several security-conscious practices:\n\n* AWS credentials are not committed to GitHub\n* Streamlit Cloud credentials are stored in Streamlit secrets\n* Streamlit uses a dedicated IAM user with read-only access to the curated S3 prefix\n* GitHub Actions uses OIDC instead of long-term AWS access keys\n* The GitHub Actions deployment role is limited to the S3 Glue script prefix\n* The CI/CD role cannot access raw, refined, curated, RDS, or Glue execution resources\n* SNS alert delivery failures are captured in an SQS dead-letter queue\n\nExample Streamlit IAM scope:\n\n```text\ns3://global-partners-data-bucket-001/curated/*\n```\n\nExample GitHub Actions IAM scope:\n\n```text\ns3://global-partners-data-bucket-001/scripts/glue_jobs/*\n```\n\n---\n\n## Interview Talking Points\n\n### Full project pitch\n\nI built an end-to-end AWS data engineering pipeline that extracts restaurant order data from SQL Server on AWS RDS, lands it in an S3 ingestion layer, converts it into Delta Lake format in a raw layer, and transforms it into refined fact and dimension tables using AWS Glue jobs orchestrated by AWS Glue Workflow. The refined layer includes an SCD Type 2 `dim_users` table to preserve historical loyalty status, and `fact_orders` stores the correct user surrogate key for point-in-time analysis. I then built a curated analytics layer with customer LTV, segmentation, churn risk, location performance, loyalty impact, and order timing analysis tables. Finally, I deployed a Streamlit Cloud dashboard that reads curated Delta tables directly from S3 and visualizes customer, location, loyalty, churn, and time-based business metrics. I also added CloudWatch, EventBridge, SNS, SQS observability and GitHub Actions CI/CD to make the project more production-aware.\n\n### Observability pitch\n\nI added an observability layer around the Glue pipeline without changing the orchestration design. Glue Workflow still handles orchestration and scheduling, while CloudWatch stores logs and metrics for each job. EventBridge listens for Glue job state changes and routes failed, timed-out, or stopped jobs to SNS for email alerts. I also configured an SQS dead-letter queue for failed SNS alert delivery.\n\n### CI/CD pitch\n\nI added a lightweight CI/CD pipeline with GitHub Actions. Pull requests and pushes run Python validation with Ruff, and pushes to main deploy the Glue job scripts to S3. The Glue jobs reference those S3 script paths, so deploying updated files updates the code used by the jobs. I used GitHub OIDC with a least-privilege AWS IAM role instead of storing long-term AWS access keys in GitHub.\n\n---\n\n## Future Improvements\n\nPotential improvements for a future production version:\n\n* Add Infrastructure as Code with Terraform or AWS CDK\n* Add automated Glue job definition deployment instead of manually configured Glue jobs\n* Add data quality checks for required fields, duplicate keys, and invalid values\n* Add structured quarantine handling for rejected records\n* Add unit tests for transformation logic where practical\n* Add integration tests against small sample datasets\n* Add table-level data freshness checks\n* Add cost monitoring for Glue, S3, and Streamlit usage\n* Add dashboard-level authentication for non-public deployments\n* Add a cached Parquet export fallback for Streamlit if direct Delta reads become slow\n\n---\n\n## Project Status\n\nThis project is complete.\n\nCompleted components:\n\n* SQL Server source setup\n* Glue Workflow orchestration and scheduling\n* S3 ingestion layer\n* Raw Delta Lake layer\n* Refined fact and dimension model\n* SCD Type 2 `dim_users`\n* Curated analytics layer\n* Streamlit Cloud dashboard\n* CloudWatch logging\n* EventBridge failure detection\n* SNS email alerts\n* SQS alert dead-letter queue\n* CloudWatch DLQ alarm\n* GitHub Actions CI/CD\n* GitHub OIDC deployment role\n* Ruff validation\n* Manual force-deploy path\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnathon-smith%2Fglobal_partners_orders_aws_pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjohnathon-smith%2Fglobal_partners_orders_aws_pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnathon-smith%2Fglobal_partners_orders_aws_pipeline/lists"}