https://github.com/jasontanx/bigquery-to-postgres-etl

Data ingestion from Google BigQuery to PostgreSQL
https://github.com/jasontanx/bigquery-to-postgres-etl

big-query data-engineering etl-pipeline postgresql

Last synced: 2 months ago
JSON representation

Data ingestion from Google BigQuery to PostgreSQL

• Host: GitHub
• URL: https://github.com/jasontanx/bigquery-to-postgres-etl
• Owner: jasontanx
• Created: 2023-03-23T14:42:04.000Z (about 2 years ago)
• Default Branch: main
• Last Pushed: 2023-04-05T12:45:06.000Z (about 2 years ago)
• Last Synced: 2025-02-01T02:19:54.233Z (4 months ago)
• Topics: big-query, data-engineering, etl-pipeline, postgresql
• Language: Python
• Homepage:
• Size: 18.6 KB
• Stars: 2
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# bigquery-to-postgres-etl

ETL --> (Extract - Load)

[ Source --> BigQuery, Destination --> Postgres]

![image](https://user-images.githubusercontent.com/116934441/227949498-df847c06-6865-40bc-b434-59996e28de93.png)

# Project Overview

**Description**: Personal project (BQ to Postgres data ingestion)

**Source**: GCP Big Query

**Apache Airflow (Orchestration Tool)** : Not Involved in this project*

**Destination**: Postgresql

**Language**: python 

**Main Tools Involved:**

1. Python (*IDE - Visual Studio Code*)

2. Google Cloud Platform account (*For extracting data from BigQuery*)

3. Docker (*Preferred*)

**Workflow**

1. Extract data from BigQuery via query

2. Save queried output into dataframe and output data as csv format

3. Ingest data into Postgres DB with new file name

**Final Outcome**

![image](https://user-images.githubusercontent.com/116934441/229813264-e66a18ec-b191-44b3-b3f0-ff384a73ce70.png)

**Extra Information**

1. Docker Compose tutorial (to run postgresql)

--> [DE Zoomcamp 1.2.5 - Running Postgres and pgAdmin with Docker-Compose](https://www.youtube.com/watch?v=hKI6PkPhpa0&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb&index=9)