Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/terraform-google-modules/terraform-google-bigquery
Creates opinionated BigQuery datasets and tables
https://github.com/terraform-google-modules/terraform-google-bigquery
cft-terraform data-analytics
Last synced: 28 days ago
JSON representation
Creates opinionated BigQuery datasets and tables
- Host: GitHub
- URL: https://github.com/terraform-google-modules/terraform-google-bigquery
- Owner: terraform-google-modules
- License: apache-2.0
- Created: 2018-10-17T15:20:30.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2024-12-13T20:18:40.000Z (about 1 month ago)
- Last Synced: 2024-12-13T21:23:33.599Z (about 1 month ago)
- Topics: cft-terraform, data-analytics
- Language: HCL
- Homepage: https://registry.terraform.io/modules/terraform-google-modules/bigquery/google
- Size: 1.64 MB
- Stars: 202
- Watchers: 29
- Forks: 170
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Codeowners: CODEOWNERS
- Security: SECURITY.md
Awesome Lists containing this project
README
# terraform-google-bigquery
This module allows you to create opinionated Google Cloud Platform BigQuery datasets and tables.
This will allow the user to programmatically create an empty table schema inside of a dataset, ready for loading.
Additional user accounts and permissions are necessary to begin querying the newly created table(s).
## Compatibility
This module is meant for use with Terraform 0.13+ and tested using Terraform 1.0+. If you find incompatibilities using Terraform >=0.13, please open an issue.
If you haven't
[upgraded](https://www.terraform.io/upgrade-guides/0-13.html) and need a Terraform
0.12.x-compatible version of this module, the last released version
intended for Terraform 0.12.x is [v4.5.0](https://registry.terraform.io/modules/terraform-google-modules/-bigquery/google/v4.5.0).
## Upgrading
The current version is 4.X. The following guides are available to assist with upgrades:
- [3.0 -> 4.0](./docs/upgrading_to_bigquery_v4.0.md)
- [2.0 -> 3.0](./docs/upgrading_to_bigquery_v3.0.md)
- [1.0 -> 2.0](./docs/upgrading_to_bigquery_v2.0.md)
- [0.1 -> 1.0](./docs/upgrading_to_bigquery_v1.0.md)
## Usage
Basic usage of this module is as follows:
```hcl
module "bigquery" {
source = "terraform-google-modules/bigquery/google"
version = "~> 9.0"
dataset_id = "foo"
dataset_name = "foo"
description = "some description"
project_id = ""
location = "US"
default_table_expiration_ms = 3600000
resource_tags = {"/":""}
tables = [
{
table_id = "foo",
schema = "",
time_partitioning = {
type = "DAY",
field = null,
require_partition_filter = false,
expiration_ms = null,
},
range_partitioning = null,
expiration_time = null,
clustering = ["fullVisitorId", "visitId"],
labels = {
env = "dev"
billable = "true"
owner = "joedoe"
},
},
{
table_id = "bar",
schema = "",
time_partitioning = null,
range_partitioning = {
field = "customer_id",
range = {
start = "1"
end = "100",
interval = "10",
},
},
expiration_time = 2524604400000, # 2050/01/01
clustering = [],
labels = {
env = "devops"
billable = "true"
owner = "joedoe"
}
}
],
views = [
{
view_id = "barview",
use_legacy_sql = false,
query = <
## Inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| access | An array of objects that define dataset access for one or more entities. | `any` |
[
{
"role": "roles/bigquery.dataOwner",
"special_group": "projectOwners"
}
]
| no |
| dataset\_id | Unique ID for the dataset being provisioned. | `string` | n/a | yes |
| dataset\_labels | Key value pairs in a map for dataset labels | `map(string)` | `{}` | no |
| dataset\_name | Friendly name for the dataset being provisioned. | `string` | `null` | no |
| default\_table\_expiration\_ms | TTL of tables using the dataset in MS | `number` | `null` | no |
| delete\_contents\_on\_destroy | (Optional) If set to true, delete all the tables in the dataset when destroying the resource; otherwise, destroying the resource will fail if tables are present. | `bool` | `null` | no |
| deletion\_protection | Whether or not to allow deletion of tables and external tables defined by this module. Can be overriden by table-level deletion\_protection configuration. | `bool` | `false` | no |
| description | Dataset description. | `string` | `null` | no |
| encryption\_key | Default encryption key to apply to the dataset. Defaults to null (Google-managed). | `string` | `null` | no |
| external\_tables | A list of objects which include table\_id, expiration\_time, external\_data\_configuration, and labels. | list(object({
table_id = string,
description = optional(string),
autodetect = bool,
compression = string,
ignore_unknown_values = bool,
max_bad_records = number,
schema = string,
source_format = string,
source_uris = list(string),
csv_options = object({
quote = string,
allow_jagged_rows = bool,
allow_quoted_newlines = bool,
encoding = string,
field_delimiter = string,
skip_leading_rows = number,
}),
google_sheets_options = object({
range = string,
skip_leading_rows = number,
}),
hive_partitioning_options = object({
mode = string,
source_uri_prefix = string,
}),
expiration_time = string,
max_staleness = optional(string),
deletion_protection = optional(bool),
labels = map(string),
})) | `[]` | no |
| location | The regional location for the dataset only US and EU are allowed in module | `string` | `"US"` | no |
| materialized\_views | A list of objects which includes view\_id, view\_query, clustering, time\_partitioning, range\_partitioning, expiration\_time and labels | list(object({
view_id = string,
description = optional(string),
query = string,
enable_refresh = bool,
refresh_interval_ms = string,
clustering = list(string),
time_partitioning = object({
expiration_ms = string,
field = string,
type = string,
require_partition_filter = bool,
}),
range_partitioning = object({
field = string,
range = object({
start = string,
end = string,
interval = string,
}),
}),
expiration_time = string,
max_staleness = optional(string),
labels = map(string),
})) | `[]` | no |
| max\_time\_travel\_hours | Defines the time travel window in hours | `number` | `null` | no |
| project\_id | Project where the dataset and table are created | `string` | n/a | yes |
| resource\_tags | A map of resource tags to add to the dataset | `map(string)` | `{}` | no |
| routines | A list of objects which include routine\_id, routine\_type, routine\_language, definition\_body, return\_type, routine\_description and arguments. | list(object({
routine_id = string,
routine_type = string,
language = string,
definition_body = string,
return_type = string,
description = string,
arguments = list(object({
name = string,
data_type = string,
argument_kind = string,
mode = string,
})),
})) | `[]` | no |
| tables | A list of objects which include table\_id, table\_name, schema, clustering, time\_partitioning, range\_partitioning, expiration\_time and labels. | list(object({
table_id = string,
description = optional(string),
table_name = optional(string),
schema = string,
clustering = list(string),
require_partition_filter = optional(bool),
time_partitioning = object({
expiration_ms = string,
field = string,
type = string,
}),
range_partitioning = object({
field = string,
range = object({
start = string,
end = string,
interval = string,
}),
}),
expiration_time = string,
deletion_protection = optional(bool),
labels = map(string),
})) | `[]` | no |
| views | A list of objects which include view\_id and view query | list(object({
view_id = string,
description = optional(string),
query = string,
use_legacy_sql = bool,
labels = map(string),
})) | `[]` | no |
## Outputs
| Name | Description |
|------|-------------|
| bigquery\_dataset | Bigquery dataset resource. |
| bigquery\_external\_tables | Map of BigQuery external table resources being provisioned. |
| bigquery\_tables | Map of bigquery table resources being provisioned. |
| bigquery\_views | Map of bigquery view resources being provisioned. |
| external\_table\_ids | Unique IDs for any external tables being provisioned |
| external\_table\_names | Friendly names for any external tables being provisioned |
| project | Project where the dataset and tables are created |
| routine\_ids | Unique IDs for any routine being provisioned |
| table\_ids | Unique id for the table being provisioned |
| table\_names | Friendly name for the table being provisioned |
| view\_ids | Unique id for the view being provisioned |
| view\_names | friendlyname for the view being provisioned |
## Requirements
These sections describe requirements for using this module.
### Software
The following dependencies must be available:
- [Terraform](https://www.terraform.io/downloads.html) >= 0.13.0
- [Terraform Provider for GCP][terraform-provider-gcp] plugin v3
### Service Account
A service account with the following roles must be used to provision
the resources of this module:
- BigQuery Data Owner: `roles/bigquery.dataOwner`
The [Project Factory module][project-factory-module] and the
[IAM module][iam-module] may be used in combination to provision a
service account with the necessary roles applied.
#### Script Helper
A helper script for configuring a Service Account is located at (./helpers/setup-sa.sh).
### APIs
A project with the following APIs enabled must be used to host the
resources of this module:
- BigQuery JSON API: `bigquery-json.googleapis.com`
The [Project Factory module][project-factory-module] can be used to
provision a project with the necessary APIs enabled.
## Contributing
Refer to the [contribution guidelines](./CONTRIBUTING.md) for
information on contributing to this module.
[iam-module]: https://registry.terraform.io/modules/terraform-google-modules/iam/google
[project-factory-module]: https://registry.terraform.io/modules/terraform-google-modules/project-factory/google
[terraform-provider-gcp]: https://www.terraform.io/docs/providers/google/index.html
[terraform]: https://www.terraform.io/downloads.html