https://github.com/gjbae1212/go-bqworker
go-esworker is an async worker that data can bulk insert, update to the BigQuery.
https://github.com/gjbae1212/go-bqworker
async bigquery bigquery-bulk gcp go golang parallel worker
Last synced: about 1 month ago
JSON representation
go-esworker is an async worker that data can bulk insert, update to the BigQuery.
- Host: GitHub
- URL: https://github.com/gjbae1212/go-bqworker
- Owner: gjbae1212
- License: mit
- Created: 2020-02-19T04:54:07.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2022-08-08T06:09:12.000Z (almost 4 years ago)
- Last Synced: 2025-08-15T07:33:40.865Z (10 months ago)
- Topics: async, bigquery, bigquery-bulk, gcp, go, golang, parallel, worker
- Language: Go
- Homepage:
- Size: 366 KB
- Stars: 3
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# go-bqworker
**go-esworker** is an async worker that data can bulk insert, update to the BigQuery.
It's a library for **golang**.
## Getting Started
### Install
```bash
$ go get -u github.com/gjbae1212/go-bqworker
```
### Usage
```go
// pseudo code
package main
import (
"context"
"time"
"github.com/gjbae1212/go-bqworker"
)
func main() {
streamer, _ := bqworker.NewStreamer([]*bqworker.TableSchemaGroup{},
bqworker.WithQueueSize(7000), bqworker.WithWorkerSize(6), bqworker.WithWorkerStack(500),
bqworker.WithMaxRetry(2), bqworker.WithWorkerWaitDuration(time.Second),
bqworker.WithDispatcherLoopWaitDuration(time.Hour), bqworker.WithErrorHandler(func(err error) {}))
streamer.Start()
// async
streamer.AddRow(ctx, row)
// sync
streamer.AddRowSync(ctx, row)
}
```
## LICENSE
This project is following The MIT.
