https://github.com/wintermi/bqwrite-test

A command line application designed to provide a method to test the BigQuery Streaming API or BigQuery Storage Write API, allowing you to get a view of the potential throughput available via a given host.
https://github.com/wintermi/bqwrite-test

bigquery google-cloud google-cloud-platform

Last synced: 5 days ago
JSON representation

A command line application designed to provide a method to test the BigQuery Streaming API or BigQuery Storage Write API, allowing you to get a view of the potential throughput available via a given host.

• Host: GitHub
• URL: https://github.com/wintermi/bqwrite-test
• Owner: wintermi
• License: apache-2.0
• Created: 2021-11-14T05:10:30.000Z (about 3 years ago)
• Default Branch: main
• Last Pushed: 2024-08-16T13:31:44.000Z (3 months ago)
• Last Synced: 2024-08-16T14:56:46.808Z (3 months ago)
• Topics: bigquery, google-cloud, google-cloud-platform
• Language: Go
• Homepage:
• Size: 163 KB
• Stars: 3
• Watchers: 0
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# BigQuery Streaming API Test Client

[![Workflows](https://github.com/wintermi/bqwrite-test/workflows/Go/badge.svg)](https://github.com/wintermi/bqwrite-test/actions)

[![Go Report](https://goreportcard.com/badge/github.com/wintermi/bqwrite-test)](https://goreportcard.com/report/github.com/wintermi/bqwrite-test)

[![License](https://img.shields.io/github/license/wintermi/bqwrite-test.svg)](https://github.com/wintermi/bqwrite-test/blob/main/LICENSE)

[![Release](https://img.shields.io/github/v/release/wintermi/bqwrite-test?include_prereleases)](https://github.com/wintermi/bqwrite-test/releases)

## Description

A command line application designed to provide a method to test the BigQuery Streaming API or BigQuery Storage Write API, allowing you to get a view of the potential throughput available via a given host.

```

USAGE:

    bqwrite-test -p PROJECT_ID -d DATASET -t TABLENAME -w WORKERS

ARGS:

  -b int

    	Batch Size, 1 to 50000 (default 1)

  -d string

    	BigQuery Dataset  (Required)

  -i int

    	Number of Records, 1 to 100000000 (default 100)

  -o	Overwrite BigQuery Table

  -p string

    	Google Cloud Project ID  (Required)

  -t string

    	BigQuery Table (default "bqwrite_test")

  -v	Output Verbose Detail

  -w int

    	Number of Parallel Workers, 1 to 100 (default 5)

```

## BigQuery Table

When you first execute the command line application it will verify if the target table exists, if not found then the table will be created.

If you wish to delete and recreate the existing table you can execute the command with the `-o` overwrite flag.

## Known Limitations

Because BigQuery's Streaming API is designed for high insertion rates, modifications to the underlying table metadata exhibit are eventually consistent when interacting with the streaming system.

Because of this, when overwriting or creating the table initially a 10 minute sleep is performed.

## License

**bqwrite-test** is released under the [Apache License 2.0](https://github.com/wintermi/bqwrite-test/blob/main/LICENSE) unless explicitly mentioned in the file header.