https://github.com/epomatti/gcp-bigquery

Data sync via CDC from GCP Cloud SQL to Big Query using Datastream
https://github.com/epomatti/gcp-bigquery

bigquery cloud-sql datastream gcp

Last synced: 13 days ago
JSON representation

Data sync via CDC from GCP Cloud SQL to Big Query using Datastream

• Host: GitHub
• URL: https://github.com/epomatti/gcp-bigquery
• Owner: epomatti
• License: mit
• Created: 2023-02-08T02:59:10.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2023-02-09T15:57:19.000Z (almost 2 years ago)
• Last Synced: 2024-11-17T00:23:45.180Z (3 months ago)
• Topics: bigquery, cloud-sql, datastream, gcp
• Language: HCL
• Homepage:
• Size: 26.4 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# gcp-bigquery

Data sync via CDC from GCP Cloud SQL to Big Query using Datastream.



Make sure to login with gcloud:

```

gcloud auth application-default login

```

First `cd` into the infra directory.

Create the `.auto.tfvars` parameter:

```hcl

project_id        = ""

region            = "us-central1"

database_tier     = "db-g1-small"

database_username = "dbadmin"

database_password = "p4ssw0rd"

```

Create the database:

```sh

terraform init

terraform apply -auto-approve

```

Connect to the database and execute execute the SQL commands in the `database.sql` file.

This database already has automated backups and point-in-time recovery enabled.

Now in the GCP console [create a stream in Datastream](https://console.cloud.google.com/datastream/streams) that sends the changes in the database to BigQuery.

Data should be available in the BigQuery dashboard for queries.