https://github.com/fsistemas/bigquery-td

ETL to extract data from mysql load and merge in BigQuery
https://github.com/fsistemas/bigquery-td

bigquery etl mysql python sql

Last synced: about 1 month ago
JSON representation

ETL to extract data from mysql load and merge in BigQuery

• Host: GitHub
• URL: https://github.com/fsistemas/bigquery-td
• Owner: fsistemas
• Created: 2020-04-05T03:29:49.000Z (almost 5 years ago)
• Default Branch: master
• Last Pushed: 2022-05-03T19:02:53.000Z (almost 3 years ago)
• Last Synced: 2024-11-09T03:38:33.667Z (3 months ago)
• Topics: bigquery, etl, mysql, python, sql
• Language: Python
• Homepage:
• Size: 4.88 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: readme.md

Awesome Lists containing this project

README

        
# Testing Bigquery API - ETL to extract data from mysql using sql2json and load that information to BigQuery

## Necessary dependencies

```

pip install --upgrade google-cloud-bigquery

pip install python-dotenv

pip install pandas

```

## Useful links

https://googleapis.dev/python/bigquery/latest/index.html

https://cloud.google.com/bigquery/docs/quickstarts/quickstart-client-libraries#client-libraries-install-python

https://cloud.google.com/bigquery/docs/loading-data-local

https://cloud.google.com/bigquery/docs/reference/standard-sql/dml-syntax

## Insert/Update by ID, Delete rows does not exist in source

```

MERGE MyDataset.table_test01 test

USING MyDataset.table_test01_tmp tmp

ON test.ID = tmp.ID

WHEN MATCHED THEN

    UPDATE SET NAME = tmp.name, CREATION_DATE = tmp.CREATION_DATE

WHEN NOT MATCHED THEN

  INSERT ROW

```