Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/wayfair-incubator/bigquery-buildkite-plugin
Buildkite Plugin to create/update structures on BigQuery
https://github.com/wayfair-incubator/bigquery-buildkite-plugin
bigquery buildkite buildkite-plugin gbq google google-bigquery google-cloud-platform hacktoberfest
Last synced: about 5 hours ago
JSON representation
Buildkite Plugin to create/update structures on BigQuery
- Host: GitHub
- URL: https://github.com/wayfair-incubator/bigquery-buildkite-plugin
- Owner: wayfair-incubator
- License: mit
- Created: 2021-01-13T13:07:02.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2024-10-23T17:10:19.000Z (15 days ago)
- Last Synced: 2024-10-24T00:38:49.580Z (15 days ago)
- Topics: bigquery, buildkite, buildkite-plugin, gbq, google, google-bigquery, google-cloud-platform, hacktoberfest
- Language: Python
- Homepage:
- Size: 270 KB
- Stars: 8
- Watchers: 6
- Forks: 1
- Open Issues: 24
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
README
[](https://github.com/wayfair-incubator/bigquery-buildkite-plugin/actions)
[](https://github.com/wayfair-incubator/bigquery-buildkite-plugin/actions)
# BigQuery Buildkite Plugin
This Buildkite plugin can be used to deploy tables/views schemas to BigQuery
A [Buildkite plugin](https://buildkite.com/docs/agent/v3/plugins) for deploying tables/views schemas to BigQuery.
## Using the plugin
If the version number is not provided then the most recent version of the plugin will be used. Do not use version number as `master` or any branch names.
### Simple
```yaml
steps:
- plugins:
- wayfair-incubator/bigquery#v1.2.2:
gcp_project: gcp-us-project
dataset_schema_directory: schemas/gcp-us-project/dataset
```
## Configuration
### Required
### `gcp_project` (required, string)
The full name of the GCP project you want to deploy.
Example: `gcp-us-project`
### `dataset_schema_directory` (required, string)
The directory in your repository where are you storing the schemas for your tables and views.
Example: `gcp-us-project/dataset_name`
### `prod_build_branch` (optional, string)
Default branch for the repo.
Example: `main`
Default: `master`
### `execute_only_changed_files` (optional, boolean)
Whether to execute just the files that changed in a directory or not.
Example: `true`
Default: `true`
### `fail_pipeline_on_first_exception` (optional, boolean)
Whether to fail pipeline as soon as there is one failure.
Example: `true`
Default: `true`
## Secret
This plugin expects `GCP_SERVICE_ACCOUNT` is placed as environment variable. Make sure to store it [securely](https://buildkite.com/docs/pipelines/secrets)!
```yaml
env:
gcp_service_account: '{"email": ""}'
```
## Example
### Basic
The following pipeline will deploy all the schemas to the `gcp-us-project` living under `directory/project/`
## Schemas
This plugin uses [GBQ](https://github.com/wayfair-incubator/gbq) to deploy to Google BigQuery.
[GBQ](https://github.com/wayfair-incubator/gbq) now supports specifying partitions with the schema as well.
To leverage this you need to nest your JSON table schema in a dictionary. An example for the same is given below. Library supports Time and Range based partitioning along with Clustering.
All the configuration options can be found [here](https://github.com/wayfair-incubator/gbq/blob/main/gbq/dto.py).
```json
{
"partition": {
"type": "range",
"definition": {
"field": "ID",
"range": {
"start": 1,
"end": 100000,
"interval": 10
}
}
},
"clustering": [
"ID"
],
"schema": [
{
"name": "ID",
"type": "INTEGER",
"mode": "REQUIRED"
}
]
}
```
## Contributing
See the [Contributing Guide](CONTRIBUTING.md) for additional information.
To execute tests locally (requires that `docker` and `docker-compose` are installed):
```bash
docker-compose run test
```
## Credits
This plugin was originally written by [Jash Parekh](https://github.com/jashparekh) for Wayfair.