https://github.com/adadalshabab/data-engineering-gcp-project

An end-to-end modern data engineering project, including deployment of ETL pipeline on Google Cloud Platform, using BigQuery for data analysis and leveraging Looker to generate an insight dashboard.
https://github.com/adadalshabab/data-engineering-gcp-project

bigquery data data-science data-visualization databases dataengineering-a engineering etl-pipeline looker-studio powerbi

Last synced: 5 months ago
JSON representation

An end-to-end modern data engineering project, including deployment of ETL pipeline on Google Cloud Platform, using BigQuery for data analysis and leveraging Looker to generate an insight dashboard.

• Host: GitHub
• URL: https://github.com/adadalshabab/data-engineering-gcp-project
• Owner: AdadAlShabab
• Created: 2024-08-04T10:38:00.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2024-08-04T12:36:41.000Z (almost 2 years ago)
• Last Synced: 2025-04-05T17:28:30.612Z (about 1 year ago)
• Topics: bigquery, data, data-science, data-visualization, databases, dataengineering-a, engineering, etl-pipeline, looker-studio, powerbi
• Language: Python
• Homepage:
• Size: 5.12 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          

# Data Engineering ETL Google Cloud Platform Project

An end-to-end modern data engineering project, including deployment of ETL pipeline on Google Cloud Platform, using BigQuery for data analysis and leveraging Looker to generate an insight dashboard.

## Technology Stack

Languages: 

* Python

* SQL

Google Cloud Platform: 

* Google Storage

* Google Engine

* Big Query

* Looker Studio

Modern Data Pipeline Tool:

* Mage - https://www.mage.ai

## Data Source

The dataset is provided by TLC Trip Record Data, including yellow and green taxi trip records. It includes fields capturing pick-up and drop-off dates/times, pick-up and drop-off locations, trip distances, itemized fares, rate types, payment types, and driver-reported passenger counts.

Data source link: https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page

Data Dictionary: https://www.nyc.gov/assets/tlc/downloads/pdf/data_dictionary_trip_records_yellow.pdf

## Data Modeling

![Uber Data Model](https://github.com/manyuzhang1996/NYC-Uber-Data-Engineering-Project-with-Google-Cloud-Platform/assets/111943220/3b9c2377-cc26-498c-a2e7-c4c857b94b80)

## ETL Pipeline



## Looker Dashboard

https://github.com/manyuzhang1996/NYC-Uber-Data-Engineering-Project-with-Google-Cloud-Platform/assets/111943220/4c7ad7b2-15b6-4a0c-aba8-8754cfef91d5

## Contact

Adad Al Shabab(sababadad74@gmail.com)