https://github.com/tinybirdco/gcs-cloud-function
Guide and scripts to automate csv import to Tinybird from Google Cloud Storage using cloud functions
https://github.com/tinybirdco/gcs-cloud-function
Last synced: 11 months ago
JSON representation
Guide and scripts to automate csv import to Tinybird from Google Cloud Storage using cloud functions
- Host: GitHub
- URL: https://github.com/tinybirdco/gcs-cloud-function
- Owner: tinybirdco
- Created: 2021-03-23T15:36:21.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2021-11-11T17:04:40.000Z (over 4 years ago)
- Last Synced: 2025-02-27T05:25:32.664Z (over 1 year ago)
- Language: Python
- Size: 4.88 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# GCS + cloud function -> Automatically ingesting to Tinybird from Google Cloud Storage
This repository contains a instructions to configure a Google Cloud function that is automatically triggered each time a new CSV file is uploaded to a given Google Cloud Storage bucket.
The Google Cloud function function appends the contents of the CSV file to a Tinybird Data Source.
## How to deploy the Cloud Function
Install the `gcloud` command line tool and run these commands:
```sh
# edit the .env.yaml file to set your Tinybird token available from https://ui.tinybird.co/tokens
cp .env.yaml.sample .env.yaml
# set some environment variables before deploying
PROJECT_NAME=
SERVICE_ACCOUNT_NAME=
BUCKET_NAME=
REGION=
TB_FUNCTION_NAME=
# grant permissions to deploy the cloud function and read from storage to the service account
gcloud projects add-iam-policy-binding $PROJECT_NAME --member serviceAccount:$SERVICE_ACCOUNT_NAME --role roles/storage.admin
gcloud projects add-iam-policy-binding $PROJECT_NAME --member serviceAccount:$SERVICE_ACCOUNT_NAME --role roles/iam.serviceAccountTokenCreator
gcloud projects add-iam-policy-binding $PROJECT_NAME --member serviceAccount:$SERVICE_ACCOUNT_NAME --role roles/editor
# deploy the cloud function
gcloud functions deploy $TB_FUNCTION_NAME \
--runtime python38 \
--trigger-resource $BUCKET_NAME \
--trigger-event google.storage.object.finalize \
--region $REGION \
--env-vars-file .env.yaml \
--service-account $SERVICE_ACCOUNT_NAME
```
Once finished, go to the Google Cloud console and check the cloud function is there and the `TB_HOST`, `TB_TOKEN`, and `FILE_REGEXP` variables required to run the script are available.
