https://github.com/tosh2230/pubsub-dataflow-bigquery

Google Cloud Dataflow for 'Exactly-Once' streaming insertion, from Google Cloud Pub/Sub to Google BigQuery.
https://github.com/tosh2230/pubsub-dataflow-bigquery

bigquery dataflow gcp google-cloud google-cloud-platform pubsub

Last synced: 5 months ago
JSON representation

Google Cloud Dataflow for 'Exactly-Once' streaming insertion, from Google Cloud Pub/Sub to Google BigQuery.

• Host: GitHub
• URL: https://github.com/tosh2230/pubsub-dataflow-bigquery
• Owner: tosh2230
• License: mit
• Created: 2021-07-19T12:40:05.000Z (over 4 years ago)
• Default Branch: main
• Last Pushed: 2023-01-14T07:14:05.000Z (about 3 years ago)
• Last Synced: 2025-01-21T02:29:40.423Z (about 1 year ago)
• Topics: bigquery, dataflow, gcp, google-cloud, google-cloud-platform, pubsub
• Language: Python
• Homepage:
• Size: 40 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# pubsub-dataflow-bigquery

A sample code of Google Cloud Dataflow for 'Exactly-Once' streaming insertion, from Google Cloud Pub/Sub to Google BigQuery.

```sh

bash ./setup.sh

cd ./dataflow

docker build -t pubsub-dataflow-bigquery .

docker run --rm -v ~/.config:/root/.config -e GCP_PROJECT=your-project-id pubsub-dataflow-bigquery bash /tmp/start_streaming.sh

```