{"id":30117058,"url":"https://github.com/markphamm/skytrax-reviews","last_synced_at":"2025-08-10T10:36:35.282Z","repository":{"id":224863019,"uuid":"764429446","full_name":"MarkPhamm/skytrax-reviews","owner":"MarkPhamm","description":"A comprehensive ETL pipeline for analyzing passenger satisfaction data. Features a modern data architecture with Apache Airflow for extraction, dbt/Snowflake for transformation, Python/Pandas for cleaning, and interactive dashboards for visualization with NextJS. ","archived":false,"fork":false,"pushed_at":"2025-08-06T21:54:00.000Z","size":161977,"stargazers_count":5,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-06T23:35:23.070Z","etag":null,"topics":["airflow-dags","astronomer","aws-s3","chartjs","chromadb","cicd","crawler-python","dbt","deepseek-r1","dockers","github-actions","langchain","nextjs","snowflake","tailwindcss"],"latest_commit_sha":null,"homepage":"https://british-airways-dashboard-website.vercel.app/","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MarkPhamm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-02-28T03:54:11.000Z","updated_at":"2025-08-06T21:54:04.000Z","dependencies_parsed_at":"2024-06-23T15:45:24.144Z","dependency_job_id":"7f3bbdaf-cfc0-4498-9d1b-f50fadafde6d","html_url":"https://github.com/MarkPhamm/skytrax-reviews","commit_stats":null,"previous_names":["markphamm/british-airway","markphamm/skytrax-reviews"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/MarkPhamm/skytrax-reviews","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkPhamm%2Fskytrax-reviews","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkPhamm%2Fskytrax-reviews/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkPhamm%2Fskytrax-reviews/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkPhamm%2Fskytrax-reviews/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MarkPhamm","download_url":"https://codeload.github.com/MarkPhamm/skytrax-reviews/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkPhamm%2Fskytrax-reviews/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269712253,"owners_count":24463208,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-10T02:00:08.965Z","response_time":71,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow-dags","astronomer","aws-s3","chartjs","chromadb","cicd","crawler-python","dbt","deepseek-r1","dockers","github-actions","langchain","nextjs","snowflake","tailwindcss"],"created_at":"2025-08-10T10:36:34.373Z","updated_at":"2025-08-10T10:36:35.269Z","avatar_url":"https://github.com/MarkPhamm.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Skytrax Global Airlines Analytics Project\n\u003cimg width=\"2000\" height=\"1333\" alt=\"image\" src=\"https://github.com/user-attachments/assets/95ba2599-7690-4a35-981e-99dc9704ee40\" /\u003e\n\n\nAccess our Dashboard: [Global Airlines Dashboard](https://british-airways-dashboard-website.vercel.app/)\n\n---\n\n## Repositories\n\n| Repository                                                                                               | Owner         | Purpose                                                                                                                                |\n| -------------------------------------------------------------------------------------------------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------- |\n| **[skytrax\\_data\\_cleaning](https://github.com/DucLe-2005/british_airways_data_cleaning)**            | DucLe‑2005    | Cleans raw scraped data and standardizes formats using modular Python functions.                                                       |\n| **[skytrax\\_extract\\_load](https://github.com/MarkPhamm/skytrax_reviews_extract_load)**             | MarkPhamm   | Scrapes customer reviews for *all* airlines on Skytrax, stages them in S3, then loads to Snowflake via Airflow‑compatible ETL scripts. |\n| **[skytrax\\_transformation](https://github.com/MarkPhamm/skytrax_reviews_extract_load)**            | MarkPhamm     | Handles dbt‑based data transformation on Snowflake with CI/CD workflows via GitHub Actions.                                            |\n| **[skytrax\\_dashboard\\_website](https://github.com/nguyentienTCU/all_airlines_dashboard_website)** | nguyentienTCU | A dashboard website for visualising insights from processed airline reviews.                                                           |\n\n---\n\n## Team Structure\n\n### Project Leadership\n\n* **Mentor – Stakeholder:** [Nhan Tran](https://www.linkedin.com/in/panicpotatoe/)\n* **Analytics Engineer – Team Lead:** [Mark Pham](https://www.linkedin.com/in/minhbphamm/)\n\n### Engineering Teams\n\n* **Data Engineering:** [Leonard Dau](https://www.linkedin.com/in/leonard-dau-722399238/), [Thieu Nguyen](https://www.linkedin.com/in/thieunguyen1402/), [Viet Lam Nguyen](https://www.linkedin.com/in/lam-nguyen-viet-051a57305)\n* **Software Engineering:** [Tien Nguyen](https://www.linkedin.com/in/tien-nguyen-598758329), [Anh Duc Le](https://www.linkedin.com/in/duc-le-517420205/)\n* **Data Science:** [Robin Tran](https://www.linkedin.com/in/robin-tran/), [Trung Dam](https://www.linkedin.com/in/trung-dam-86962a235/)\n* **Scrum Master:** [Hien Dinh](https://www.linkedin.com/in/hiendinhq)\n\n---\n\n## 1. Project Overview\n\nThis end‑to‑end analytics initiative ingests, processes, and visualises **customer‑review data for *every* airline covered by Skytrax** (AirlineQuality.com). The architecture leverages industry‑standard tooling and cloud services to provide a robust, scalable foundation for airline‑wide sentiment, operational, and competitive analysis.\n\n**Self‑selection bias:** Reviews on Skytrax are self‑reported. Passengers with extreme experiences (positive *or* negative) are more likely to post, so KPIs derived from this data skew away from the broader flying population. Our goal is therefore *directional insight*, not population‑level generalisation.\n\n---\n\n## 2. Architecture Overview\n\n![BritishAirways](https://github.com/user-attachments/assets/2a9d45e6-be1b-4582-a9a0-3b7fb7536d9f)\n\n### 2.1 Extraction Layer\n\n#### Overview\n\nThe extraction layer gathers review data for *all* airlines from Skytrax, stores it in S3 and prepares it for downstream processing.\n\n* **Repository:** [all\\_airlines\\_extract\\_load](https://github.com/vietlam2002/all_airlines_extract_load)\n\n#### 2.1.1 Technology Stack\n\n* Python 3.12 with Pandas\n* Apache Airflow\n* AWS S3\n* Docker\n* Snowflake\n\n#### 2.1.2 Data Source\n\nSkytrax review pages, e.g.\n`https://www.airlinequality.com/airline-reviews/{airline‑slug}/`\n\nCaptured fields include: star ratings, review text, flight details, passenger metadata, and category scores.\n\n#### 2.1.3 Extraction Process\n\n```python\n# From main_dag.py – Extract task definition\nscrape_skytrax_data = BashOperator(\n    task_id=\"scrape_skytrax_data\",\n    bash_command=\"chmod -R 777 /opt/***/data \u0026\u0026 python /opt/airflow/tasks/scraper_extract/scraper.py\"\n)\n```\n\nSteps\n\n1. Iterate through the Skytrax airline index.\n2. Request paginated review HTML for each carrier.\n3. Parse and normalise each review record.\n4. Persist results to `raw_data.csv`.\n\n#### 2.1.4 Data Cleaning \u0026 Initial Transformation\n\n```python\nclean_data = BashOperator(\n    task_id=\"clean_data\",\n    bash_command=\"python /opt/airflow/tasks/transform/transform.py\"\n)\n```\n\nCleaning tasks standardise date formats, handle nulls, and enforce data‑type consistency before staging to S3.\n\n#### 2.1.5 AWS S3 Integration\n\n```python\nupload_cleaned_data_to_s3 = BashOperator(\n    task_id=\"upload_cleaned_data_to_s3\",\n    bash_command=\"chmod -R 777 /opt/airflow/data \u0026\u0026 python /opt/airflow/tasks/upload_to_s3.py\"\n)\n```\n\n* Secure IAM roles\n* Server‑side encryption\n* Versioning enabled\n\n#### 2.1.6 Workflow Orchestration\n\n```python\nwith DAG(\n    dag_id=\"skytrax_pipeline\",\n    schedule_interval=\"@daily\",\n    default_args=default_args,\n    start_date=start_date,\n    catchup=True,\n    max_active_runs=1,\n):\n    scrape_skytrax_data \u003e\u003e note \u003e\u003e clean_data \u003e\u003e note_clean_data \u003e\u003e upload_cleaned_data_to_s3\n```\n\n#### 2.1.7 Snowflake Integration\n\n```python\nsnowflake_copy_operator = BashOperator(\n    task_id=\"snowflake_copy_from_s3\",\n    bash_command=\"pip install snowflake-connector-python python-dotenv \u0026\u0026 python /opt/airflow/tasks/snowflake_load.py\"\n)\n```\n\n---\n\n### 2.2 Data Cleaning Layer\n\n* **Repository:** [all\\_airlines\\_data\\_cleaning](https://github.com/DucLe-2005/all_airlines_data_cleaning)\n* **Stack:** Python 3.12.5, Pandas, NumPy, Matplotlib, Seaborn\n\nKey steps mirror the British Airways version but operate across carriers:\n\n1. **Column Standardisation** – snake\\_case, special‑character cleanup.\n2. **Date Formatting** – ISO 8601 for both submission and flight dates.\n3. **Text Cleaning** – verification flag extraction; nationality normalisation.\n4. **Route Parsing** – origin, destination, and connections.\n5. **Aircraft Standardisation** – unified Airbus/Boeing nomenclature.\n6. **Rating Conversion** – numeric Int64 fields for uniform analysis.\n\nOutputs feed directly to Snowflake for transformation.\n\n---\n\n### 2.3 Transformation Layer\n\n* **Repository:** [all\\_airlines\\_transformation](https://github.com/MarkPhamm/all_airlines_transformation)\n* **Stack:** dbt (Core), Snowflake, Airflow (Astronomer), GitHub Actions\n\n#### 2.3.1 Data Model\n\nA star schema identical in design to the airline‑specific version:\n\n| Table             | Purpose                                                 |\n| ----------------- | ------------------------------------------------------- |\n| **fct\\_review**   | One row per review per flight with quantitative metrics |\n| **dim\\_customer** | Passenger information                                   |\n| **dim\\_aircraft** | Aircraft attributes                                     |\n| **dim\\_location** | Airport / city keys for origin, destination, transit    |\n| **dim\\_date**     | Calendar table for submission \u0026 flight dates            |\n\nIncremental dbt jobs maintain freshness while minimising warehouse spend.\n\n#### 2.3.2 Data Quality Framework\n\n* Schema \u0026 relationship tests\n* Custom business‑logic assertions (e.g. rating within 0–10)\n* Freshness \u0026 completeness checks\n\nCI/CD triggers on code pushes, PRs, weekly schedules, and manual invocations.\n\n---\n\n### 2.4 Visualisation Layer\n\n* **Repository:** [all\\_airlines\\_dashboard\\_website](https://github.com/nguyentienTCU/all_airlines_dashboard_website)\n* **Live Site:** [Global Airlines Analytics Dashboard](https://global-airlines-dashboard.vercel.app/)\n* **Stack:** Next.js, TailwindCSS, Chart.js, LangChain, ChromaDB\n\n#### 2.4.1 Dashboard Highlights\n\n* **Interactive KPI Cards:** Overall satisfaction, NPS‑like scores, category averages.\n* **Multi‑Dimensional Filters:** Airline, aircraft, route, cabin class, traveller type.\n* **Data Explorer:** Drag‑and‑drop or SQL‑like querying for power users.\n* **RAG Chatbot:** Natural‑language Q\\\u0026A across the full corpus of reviews.\n\n---\n\n## 3. Key Business Insights (Illustrative)\n\n### 3.1 Economy‑Class Passenger Trends\n![Problem 1 Details](https://github.com/MarkPhamm/British-Airway/assets/99457952/665ff202-218a-4862-a130-98ce4c8584b9)\n**Findings**\n\n* Across airlines, ground‑staff service and boarding efficiency dominate complaints.\n* Major international hubs (e.g. LHR, CDG, JFK) see the highest negative volume—often tied to long security queues and staff shortages.\n* 92 % of low‑rating Economy reviews cite *at‑airport* factors rather than in‑flight experience.\n\n![Problem 1](https://github.com/MarkPhamm/British-Airway/assets/99457952/fad27d46-f9c1-4187-94af-02da65d3f10b)\n**Recommendations**\n\n* Collaborate with ground‑handling partners to boost staffing during peak waves.\n* Deploy self‑service kiosks and real‑time queue monitoring.\n\n### 3.2 Premium‑Cabin Passenger Expectations\n\n\n**Findings**\n\n* Business \u0026 First passengers focus on seat comfort, bedding quality, and connectivity speed.\n* Consistency gaps between aircraft sub‑fleets (older cabins vs. refurbished) drive dissatisfaction.\n* Food quality is the second‑largest driver of 4‑star‑and‑below ratings.\n\n**Recommendations**\n\n* Accelerate fleet‑wide seat upgrade programmes.\n* Introduce chef‑curated rotating menus with regional options.\n* Guarantee minimum bandwidth per passenger on Wi‑Fi plans.\n\n---\n\n## 4. Next Steps\n\n1. **Expand Data Sources** – Integrate on‑time‑performance and DOT complaint data for richer modelling.\n2. **Real‑Time Ingestion** – Move to CDC‑style pipelines to surface insights within hours of review publication.\n3. **Predictive Modelling** – Use sentiment plus operational variables to forecast future NPS movement by airline and route.\n4. **Monetisation** – Offer benchmarking dashboards to airlines and airports via subscription.\n\n---\n\n*© 2025 Skytrax Global Airlines Analytics Project*\n\n\n\n## 2. Architecture Overview\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarkphamm%2Fskytrax-reviews","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarkphamm%2Fskytrax-reviews","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarkphamm%2Fskytrax-reviews/lists"}