https://github.com/smohanta23/uber_data-engineering_etl-project

This project demonstrates a comprehensive data engineering workflow using the Uber information dataset. It covers the full spectrum of data engineering pipelines, from data transformation to deployment on Google Cloud, with a focus on creating a scalable and insightful data model.
https://github.com/smohanta23/uber_data-engineering_etl-project

big-data-analytics bigquery cloudcomputing computeengine dashboard-application dataengineering datainsights datamodelling datapipeline datascience datavisualization etl-pipeline gcp-project googlecloudplatform mage opensource python uber uber-api

Last synced: about 1 month ago
JSON representation

This project demonstrates a comprehensive data engineering workflow using the Uber information dataset. It covers the full spectrum of data engineering pipelines, from data transformation to deployment on Google Cloud, with a focus on creating a scalable and insightful data model.

• Host: GitHub
• URL: https://github.com/smohanta23/uber_data-engineering_etl-project
• Owner: Smohanta23
• License: mit
• Created: 2024-08-15T06:21:13.000Z (3 months ago)
• Default Branch: master
• Last Pushed: 2024-08-15T08:06:30.000Z (3 months ago)
• Last Synced: 2024-09-24T23:01:18.205Z (about 2 months ago)
• Topics: big-data-analytics, bigquery, cloudcomputing, computeengine, dashboard-application, dataengineering, datainsights, datamodelling, datapipeline, datascience, datavisualization, etl-pipeline, gcp-project, googlecloudplatform, mage, opensource, python, uber, uber-api
• Language: Jupyter Notebook
• Homepage: https://lookerstudio.google.com/u/0/reporting/f383c480-7dab-461f-b426-8bbb1df2f13d/page/8pqOD?s=nQI06ax2wMY
• Size: 19.6 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Uber Data Analytics (Data Engineering ETL Project)

![uber-logo](Uber-logo.jpg)

In this project, I designed a comprehensive data engineering solution using an Uber dataset to build a robust data model. I implemented data transformation by writing Python scripts to convert flat files into structured fact and dimension tables. The project was deployed on Google Cloud, utilizing Compute Engine for virtual machines, BigQuery for data warehousing, and Data Studio for creating interactive dashboards. Mage, an open-source tool, was employed for seamless data transformation and integration. This hands-on project not only demonstrates practical skills in Python and SQL but also highlights key data engineering concepts such as dimensional modeling and cloud integration for scalable data solutions.

### Step 1: Designing a Process Flow on GCP



### Step 2: Building an ER Diagram for Uber Data-Flow

![Uber Data Model](Uber_ERD.png)

### Step 3: Analysing the data in Python (Feature Engineering)

https://github.com/UmairThakur/Uber-Data-Analysis-ETL-PIPELINE-DATA-ANALYSIS_PROJECT/assets/81063457/4208599f-c0a2-4747-ae1b-87751a37de6f

## Step 4: Developing the uber project and a bucket on the Google Cloud Platform, extracting the data, selecting the server and setting up the required permissions.



Note: Project ID and Project Number are hidden intentionally for copyright issues.

## Step 5: Creating a Virtual Machine Instance in GCP using GCP Compute Engine.



  

  

  



## Step 6: Connectting the VM to the Mage Project using SSH Linux Terminal and creating the mage project.



### Step 7: Building a data pipeline with Mage using blocks like data loader, transformer, and exporter (ETL).Incorporate your own extra transformation code into the data transformer, making the necessary adjustments.

### Step 8: After setting up the pipeline, add your GCP credentials to the `io_config.yaml` file. You can obtain these credentials from the APIs and Services section in Google Cloud Console.

### Step 9: Utilize BigQuery to perform ETL operations on the data, making it suitable for analysis such as creating dashboard and generating reports.

https://github.com/UmairThakur/Uber-Data-Analysis-ETL-PIPELINE-DATA-ANALYSIS_PROJECT/assets/81063457/7d03f9af-28c2-405c-a7ea-55dd45cffa1f

### Step 10: Finally, create a dashboard using your preferred dashboarding or reporting tool. I used Google Looker Studio, but you can also opt for other tools like Power BI, Tableau, or Qlik Sense.







## Have a look at my Uber Dashboard- [https://lookerstudio.google.com/s/nQI06ax2wMY](https://lookerstudio.google.com/u/0/reporting/f383c480-7dab-461f-b426-8bbb1df2f13d/page/8pqOD?s=nQI06ax2wMY)