{"id":25535978,"url":"https://github.com/tom474/data-pipeline-with-aws","last_synced_at":"2026-04-03T23:38:11.952Z","repository":{"id":278310126,"uuid":"933749796","full_name":"tom474/data_pipeline_with_aws","owner":"tom474","description":"[RMIT 2024C] EEET2574 - Big Data for Engineering - Group Project","archived":false,"fork":false,"pushed_at":"2025-02-19T04:08:41.000Z","size":3607,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-19T05:20:51.717Z","etag":null,"topics":["aws","data-engineering","data-science","data-visualization","machine-learning","mongodb","python","spark"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tom474.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-16T15:59:24.000Z","updated_at":"2025-02-19T04:14:25.000Z","dependencies_parsed_at":"2025-02-19T05:20:58.523Z","dependency_job_id":"fc275c2a-d62f-4563-b097-6570db89addc","html_url":"https://github.com/tom474/data_pipeline_with_aws","commit_stats":null,"previous_names":["tom474/data_pipeline_with_aws"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tom474%2Fdata_pipeline_with_aws","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tom474%2Fdata_pipeline_with_aws/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tom474%2Fdata_pipeline_with_aws/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tom474%2Fdata_pipeline_with_aws/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tom474","download_url":"https://codeload.github.com/tom474/data_pipeline_with_aws/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239775905,"owners_count":19695018,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","data-engineering","data-science","data-visualization","machine-learning","mongodb","python","spark"],"created_at":"2025-02-20T04:24:42.015Z","updated_at":"2025-12-30T20:12:27.190Z","avatar_url":"https://github.com/tom474.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Pipeline with AWS  \n\nThis project builds a **data pipeline** using **AWS services** for real-time and batch data processing, integrating **machine learning** for predictive analytics.\n\n## Tech Stack  \n\n- Python  \n- AWS S3  \n- AWS EC2  \n- AWS Kinesis  \n- AWS Lambda  \n- AWS SageMaker  \n- MongoDB  \n\n## System Architecture  \n\n![System Architecture Diagram](https://github.com/tom474/data_pipeline_with_aws/blob/main/assets/system-architecture-diagram.png?raw=true)  \n\n## Features  \n\n### Streaming Data ETL Pipeline  \n\n- Real-time weather data ingestion from **OpenWeatherMap API**.  \n- Data processing using **AWS EC2, Kinesis, and Lambda**.  \n- Data storage in **MongoDB** for real-time analytics.  \n\n### Historical Data \u0026 Testing Data ETL Pipeline  \n\n- Integration of **U.S. Civil Flights \u0026 Weather Meteo datasets**.  \n- Data cleaning and transformation using **AWS SageMaker**.  \n- Processed data stored in **AWS S3** for analysis.  \n\n\n### Prediction Model Data Pipeline  \n\n- Training dataset creation using **historical flight \u0026 weather data**.  \n- Model training in **AWS SageMaker** using **Gradient Boosting, Decision Tree, and Random Forest**.  \n- Predictions generated for real-time flight delay forecasting.  \n\n### Visualization Data Pipeline  \n\n- Processed data stored in **MongoDB** for visualization.  \n- Dashboards built using **MongoDB Charts** for insights:\n  - [LAX Overview Dashboard](https://charts.mongodb.com/charts-big-data-for-engineering-lzggfsp/public/dashboards/2b54cb6f-21e0-4cba-b024-a898e3606f0f?fbclid=IwZXh0bgNhZW0CMTAAAR3SBTXZwQnorRUTwApbq4Z5FwMAzy2O3D82_Zm7hFOw6NYDMW2UijQFqeI_aem_TsqlkYcRpEPGmuLCP3pdYw)\n  - [LAX Weather Impact Dashboard](https://charts.mongodb.com/charts-big-data-for-engineering-lzggfsp/public/dashboards/2240afae-d905-4c0e-ad54-959e0f0a83ab?fbclid=IwZXh0bgNhZW0CMTAAAR3H2n8Hnx7prUGv0Ty__CRx8lBfiO88N0Sa-wPp-ThXvG1m6vdTFWIknnU_aem_ZSH8xQzG7SYU3poAiOjNiw)\n  - [LAX Performance Benchmark Dashboard](https://charts.mongodb.com/charts-big-data-for-engineering-lzggfsp/public/dashboards/40528c4b-1ef1-4a0b-ae93-d3e20911d42b?fbclid=IwZXh0bgNhZW0CMTAAAR3H2n8Hnx7prUGv0Ty__CRx8lBfiO88N0Sa-wPp-ThXvG1m6vdTFWIknnU_aem_ZSH8xQzG7SYU3poAiOjNiw)\n  - [LAX Time Series Dashboard](https://charts.mongodb.com/charts-big-data-for-engineering-lzggfsp/public/dashboards/5c7d6348-0dd8-4b77-a39b-18e75e3c3b44?fbclid=IwZXh0bgNhZW0CMTAAAR0kTTF7D3HchOVbwhk-3JTARp5I6U-lBYCHiZ-hWSpHinfgTprVWfLzAy8_aem_oH6mf1EOlemXOgAUZePIPg)\n  - [LAX Geographical Insights Dashboard](https://charts.mongodb.com/charts-big-data-for-engineering-lzggfsp/public/dashboards/b8929560-fa61-454e-95da-6ecc6190b764?fbclid=IwZXh0bgNhZW0CMTAAAR3lp61_82Zx_S2QHsDcdZRLtbtkJ9FRlw94QLAJRo69vY73N3NN-adq_BA_aem_qhm9l3CXJibgx7RiAskkxg)\n  - [LAX Aircraft Analysis Dashboard](https://charts.mongodb.com/charts-big-data-for-engineering-lzggfsp/public/dashboards/9f1b8081-1940-4b85-b5f1-84120b0c31c1?fbclid=IwZXh0bgNhZW0CMTAAAR3lp61_82Zx_S2QHsDcdZRLtbtkJ9FRlw94QLAJRo69vY73N3NN-adq_BA_aem_qhm9l3CXJibgx7RiAskkxg) \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftom474%2Fdata-pipeline-with-aws","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftom474%2Fdata-pipeline-with-aws","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftom474%2Fdata-pipeline-with-aws/lists"}