https://github.com/flowerinthenight/bqstream

A simple library to help facilitate streaming to BigQuery.
https://github.com/flowerinthenight/bqstream

bigquery go golang streaming

Last synced: 4 days ago
JSON representation

A simple library to help facilitate streaming to BigQuery.

• Host: GitHub
• URL: https://github.com/flowerinthenight/bqstream
• Owner: flowerinthenight
• License: mit
• Created: 2020-10-22T01:39:22.000Z (about 4 years ago)
• Default Branch: main
• Last Pushed: 2023-08-21T15:55:55.000Z (about 1 year ago)
• Last Synced: 2024-06-20T06:37:58.914Z (5 months ago)
• Topics: bigquery, go, golang, streaming
• Language: Go
• Homepage:
• Size: 53.7 KB
• Stars: 0
• Watchers: 3
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
![main](https://github.com/flowerinthenight/bqstream/workflows/main/badge.svg)

[![Go Reference](https://pkg.go.dev/badge/github.com/flowerinthenight/bqstream.svg)](https://pkg.go.dev/github.com/flowerinthenight/bqstream)

**NOTE: BigQuery recommends using its [Storage Write API](https://cloud.google.com/bigquery/docs/write-api) instead of its legacy streaming API which this library uses.**

## bqstream

A simple wrapper library for streaming data to BigQuery. By default, it will create streaming goroutines equal to the number of cores it's running on.

At the moment, authentication is done by providing the following environment variable:

```

GOOGLE_APPLICATION_CREDENTIALS=/path/to/service/account/file.json

```

## Usage

To use the library, you can do something like this:

```go

// The data that we will be streaming to BigQuery.

type Data struct {

	Key   string

	Value string

}

// Save implements the ValueSaver interface.

func (d *Data) Save() (map[string]bigquery.Value, string, error) {

	return map[string]bigquery.Value{

		"key":   d.Key,

		"value": d.Value,

	}, "", nil

}

// Create a streamer for a table in Tokyo region.

streamer = bqstream.New("project01", "asia-northeast1", "dataset01", "table01")

// Stream the data.

streamer.Add([]*Data{

	&Data{

		Key:   "samplekey",

		Value: "samplevalue",

	},

})

```