https://github.com/gjbae1212/go-bqworker

go-esworker is an async worker that data can bulk insert, update to the BigQuery.
https://github.com/gjbae1212/go-bqworker

async bigquery bigquery-bulk gcp go golang parallel worker

Last synced: 8 days ago
JSON representation

go-esworker is an async worker that data can bulk insert, update to the BigQuery.

• Host: GitHub
• URL: https://github.com/gjbae1212/go-bqworker
• Owner: gjbae1212
• License: mit
• Created: 2020-02-19T04:54:07.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2022-08-08T06:09:12.000Z (over 2 years ago)
• Last Synced: 2024-06-21T03:15:57.745Z (5 months ago)
• Topics: async, bigquery, bigquery-bulk, gcp, go, golang, parallel, worker
• Language: Go
• Homepage:
• Size: 366 KB
• Stars: 3
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# go-bqworker

 

   

   

               

   



**go-esworker** is an async worker that data can bulk insert, update to the BigQuery.  

It's a library for **golang**.

## Getting Started

### Install

```bash

$ go get -u github.com/gjbae1212/go-bqworker

```

### Usage

```go

// pseudo code

package main

import (

    "context"

	"time"

    "github.com/gjbae1212/go-bqworker"

)

func main() {

	streamer, _ := bqworker.NewStreamer([]*bqworker.TableSchemaGroup{},

		bqworker.WithQueueSize(7000), bqworker.WithWorkerSize(6), bqworker.WithWorkerStack(500),

		bqworker.WithMaxRetry(2), bqworker.WithWorkerWaitDuration(time.Second),

		bqworker.WithDispatcherLoopWaitDuration(time.Hour), bqworker.WithErrorHandler(func(err error) {}))

	streamer.Start()

	

   // async

   streamer.AddRow(ctx, row)     

   // sync 

   streamer.AddRowSync(ctx, row)

}  

```

## LICENSE

This project is following The MIT.