https://github.com/adadalshabab/data-engineering-gcp-project
An end-to-end modern data engineering project, including deployment of ETL pipeline on Google Cloud Platform, using BigQuery for data analysis and leveraging Looker to generate an insight dashboard.
https://github.com/adadalshabab/data-engineering-gcp-project
bigquery data data-science data-visualization databases dataengineering-a engineering etl-pipeline looker-studio powerbi
Last synced: 5 months ago
JSON representation
An end-to-end modern data engineering project, including deployment of ETL pipeline on Google Cloud Platform, using BigQuery for data analysis and leveraging Looker to generate an insight dashboard.
- Host: GitHub
- URL: https://github.com/adadalshabab/data-engineering-gcp-project
- Owner: AdadAlShabab
- Created: 2024-08-04T10:38:00.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-08-04T12:36:41.000Z (almost 2 years ago)
- Last Synced: 2025-04-05T17:28:30.612Z (about 1 year ago)
- Topics: bigquery, data, data-science, data-visualization, databases, dataengineering-a, engineering, etl-pipeline, looker-studio, powerbi
- Language: Python
- Homepage:
- Size: 5.12 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Data Engineering ETL Google Cloud Platform Project
An end-to-end modern data engineering project, including deployment of ETL pipeline on Google Cloud Platform, using BigQuery for data analysis and leveraging Looker to generate an insight dashboard.
## Technology Stack
Languages:
* Python
* SQL
Google Cloud Platform:
* Google Storage
* Google Engine
* Big Query
* Looker Studio
Modern Data Pipeline Tool:
* Mage - https://www.mage.ai
## Data Source
The dataset is provided by TLC Trip Record Data, including yellow and green taxi trip records. It includes fields capturing pick-up and drop-off dates/times, pick-up and drop-off locations, trip distances, itemized fares, rate types, payment types, and driver-reported passenger counts.
Data source link: https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page
Data Dictionary: https://www.nyc.gov/assets/tlc/downloads/pdf/data_dictionary_trip_records_yellow.pdf
## Data Modeling

## ETL Pipeline

## Looker Dashboard
https://github.com/manyuzhang1996/NYC-Uber-Data-Engineering-Project-with-Google-Cloud-Platform/assets/111943220/4c7ad7b2-15b6-4a0c-aba8-8754cfef91d5
## Contact
Adad Al Shabab(sababadad74@gmail.com)