Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/jashparekh/bigquery-action
This Github action can be used to deploy tables/views schemas to BigQuery.
https://github.com/jashparekh/bigquery-action
actions bigquery gbq github-actions google google-bigquery google-cloud-platform hacktoberfest
Last synced: 2 months ago
JSON representation
This Github action can be used to deploy tables/views schemas to BigQuery.
- Host: GitHub
- URL: https://github.com/jashparekh/bigquery-action
- Owner: jashparekh
- License: mit
- Created: 2021-09-12T18:46:30.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-03-07T19:59:58.000Z (almost 2 years ago)
- Last Synced: 2024-10-15T14:41:07.194Z (2 months ago)
- Topics: actions, bigquery, gbq, github-actions, google, google-bigquery, google-cloud-platform, hacktoberfest
- Language: Python
- Homepage:
- Size: 180 KB
- Stars: 9
- Watchers: 2
- Forks: 4
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
[](https://github.com/jashparekh/bigquery-action/actions)
[](https://github.com/jashparekh/bigquery-action/actions)
[](https://github.com/jashparekh/bigquery-action/actions)
# BigQuery Github Action
This Github action can be used to deploy tables/views schemas to BigQuery.
### Simple
```yaml
name: "Deploy to BigQuery"
on:
pull_request: {}
push:
branches: ["main"]
jobs:
deploy_schemas:
runs-on: ubuntu-latest
name: Deploy to BigQuery
steps:
# To use this repository's private action,
# you must check out the repository
- name: Checkout
uses: actions/[email protected]
- name: Deploy schemas to BigQuery
uses: jashparekh/bigquery-action@v3
env:
gcp_project: 'gcp-us-project'
dataset_schema_directory: 'gcp-us-project/dataset_name'
credentials: ${{ secrets.GCP_SERVICE_ACCOUNT }}
```
## Configuration
### Required
### `gcp_project` (required, string)
The full name of the GCP project you want to deploy.
Example: `gcp-us-project`
### `dataset_schema_directory` (required, string)
The directory in your repository where are you storing the schemas for your tables and views.
Example: `gcp-us-project/dataset_name`
### `credentials` (required, string)
Google Service Account with permission to create objects in the specified project. Can be stored as a [repository secret](https://docs.github.com/en/actions/reference/encrypted-secrets)
## Schemas
This action uses [GBQ](https://github.com/wayfair-incubator/gbq) to deploy to Google BigQuery.
[GBQ](https://github.com/wayfair-incubator/gbq) now supports specifying partitions with the schema as well.
To leverage this you need to nest your JSON table schema in a dictionary. An example for the same is given below. Library supports Time and Range based partitioning along with Clustering.
All the configuration options can be found [here](https://github.com/wayfair-incubator/gbq/blob/main/gbq/dto.py).
```json
{
"partition": {
"type": "range",
"definition": {
"field": "ID",
"range": {
"start": 1,
"end": 100000,
"interval": 10
}
}
},
"clustering": [
"ID"
],
"schema": [
{
"name": "ID",
"type": "INTEGER",
"mode": "REQUIRED"
}
]
}
```
## Contributing
See the [Contributing Guide](CONTRIBUTING.md) for additional information.
To execute tests locally (requires that `docker` and `docker-compose` are installed):
```bash
docker-compose run test
```
## Credits
This Github Action was originally written by [Jash Parekh](https://github.com/jashparekh).