https://github.com/terraform-google-modules/terraform-google-bigquery

Creates opinionated BigQuery datasets and tables
https://github.com/terraform-google-modules/terraform-google-bigquery
cft-terraform data-analytics
Last synced: 7 months ago
JSON representation
Creates opinionated BigQuery datasets and tables
Host: GitHub
URL: https://github.com/terraform-google-modules/terraform-google-bigquery
Owner: terraform-google-modules
License: apache-2.0
Created: 2018-10-17T15:20:30.000Z (over 7 years ago)
Default Branch: main
Last Pushed: 2025-02-11T01:08:32.000Z (over 1 year ago)
Last Synced: 2025-02-11T02:22:40.425Z (over 1 year ago)
Topics: cft-terraform, data-analytics
Language: HCL
Homepage: https://registry.terraform.io/modules/terraform-google-modules/bigquery/google
Size: 1.71 MB
Stars: 205
Watchers: 39
Forks: 177
Open Issues: 8
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Codeowners: CODEOWNERS
- Security: SECURITY.md
Awesome Lists containing this project

README

          # terraform-google-bigquery

This module allows you to create opinionated Google Cloud Platform BigQuery datasets and tables.

This will allow the user to programmatically create an empty table schema inside of a dataset, ready for loading.

Additional user accounts and permissions are necessary to begin querying the newly created table(s).

## Compatibility

This module is meant for use with Terraform 0.13+ and tested using Terraform 1.0+. If you find incompatibilities using Terraform >=0.13, please open an issue.

 If you haven't

[upgraded](https://www.terraform.io/upgrade-guides/0-13.html) and need a Terraform

0.12.x-compatible version of this module, the last released version

intended for Terraform 0.12.x is [v4.5.0](https://registry.terraform.io/modules/terraform-google-modules/-bigquery/google/v4.5.0).

## Upgrading

The current version is 4.X. The following guides are available to assist with upgrades:

- [3.0 -> 4.0](./docs/upgrading_to_bigquery_v4.0.md)

- [2.0 -> 3.0](./docs/upgrading_to_bigquery_v3.0.md)

- [1.0 -> 2.0](./docs/upgrading_to_bigquery_v2.0.md)

- [0.1 -> 1.0](./docs/upgrading_to_bigquery_v1.0.md)

## Usage

Basic usage of this module is as follows:

```hcl

module "bigquery" {

  source  = "terraform-google-modules/bigquery/google"

  version = "~> 9.0"

  dataset_id                  = "foo"

  dataset_name                = "foo"

  description                 = "some description"

  project_id                  = ""

  location                    = "US"

  default_table_expiration_ms = 3600000

  resource_tags               = {"/":""}

  tables = [

  {

    table_id           = "foo",

    schema             =  "",

    time_partitioning  = {

      type                     = "DAY",

      field                    = null,

      require_partition_filter = false,

      expiration_ms            = null,

    },

    range_partitioning = null,

    expiration_time = null,

    clustering      = ["fullVisitorId", "visitId"],

    labels          = {

      env      = "dev"

      billable = "true"

      owner    = "joedoe"

    },

  },

  {

    table_id           = "bar",

    schema             =  "",

    time_partitioning  = null,

    range_partitioning = {

      field = "customer_id",

      range = {

        start    = "1"

        end      = "100",

        interval = "10",

      },

    },

    expiration_time    = 2524604400000, # 2050/01/01

    clustering         = [],

    labels = {

      env      = "devops"

      billable = "true"

      owner    = "joedoe"

    }

  }

  ],

  views = [

    {

      view_id    = "barview",

      use_legacy_sql = false,

      query          = <

## Inputs

| Name | Description | Type | Default | Required |

|------|-------------|------|---------|:--------:|

| access | An array of objects that define dataset access for one or more entities. | `any` | 
[
  {
    "role": "roles/bigquery.dataOwner",
    "special_group": "projectOwners"
  }
] | no |

| dataset\_id | Unique ID for the dataset being provisioned. | `string` | n/a | yes |

| dataset\_labels | Key value pairs in a map for dataset labels | `map(string)` | `{}` | no |

| dataset\_name | Friendly name for the dataset being provisioned. | `string` | `null` | no |

| default\_partition\_expiration\_ms | The default partition expiration for all partitioned tables in the dataset, in MS | `number` | `null` | no |

| default\_table\_expiration\_ms | TTL of tables using the dataset in MS | `number` | `null` | no |

| delete\_contents\_on\_destroy | (Optional) If set to true, delete all the tables in the dataset when destroying the resource; otherwise, destroying the resource will fail if tables are present. | `bool` | `null` | no |

| deletion\_protection | Whether or not to allow deletion of tables and external tables defined by this module. Can be overriden by table-level deletion\_protection configuration. | `bool` | `false` | no |

| description | Dataset description. | `string` | `null` | no |

| encryption\_key | Default encryption key to apply to the dataset. Defaults to null (Google-managed). | `string` | `null` | no |

| external\_tables | A list of objects which include table\_id, expiration\_time, external\_data\_configuration, and labels. | list(object({
    table_id              = string,
    description           = optional(string),
    autodetect            = bool,
    compression           = string,
    ignore_unknown_values = bool,
    max_bad_records       = number,
    schema                = string,
    source_format         = string,
    source_uris           = list(string),
    csv_options = object({
      quote                 = string,
      allow_jagged_rows     = bool,
      allow_quoted_newlines = bool,
      encoding              = string,
      field_delimiter       = string,
      skip_leading_rows     = number,
    }),
    google_sheets_options = object({
      range             = string,
      skip_leading_rows = number,
    }),
    hive_partitioning_options = object({
      mode              = string,
      source_uri_prefix = string,
    }),
    expiration_time     = string,
    max_staleness       = optional(string),
    deletion_protection = optional(bool),
    labels              = map(string),
  })) | `[]` | no |

| location | The regional location for the dataset only US and EU are allowed in module | `string` | `"US"` | no |

| materialized\_views | A list of objects which includes view\_id, view\_query, clustering, time\_partitioning, range\_partitioning, expiration\_time and labels | list(object({
    view_id             = string,
    description         = optional(string),
    query               = string,
    enable_refresh      = bool,
    refresh_interval_ms = string,
    clustering          = list(string),
    time_partitioning = object({
      expiration_ms            = string,
      field                    = string,
      type                     = string,
      require_partition_filter = bool,
    }),
    range_partitioning = object({
      field = string,
      range = object({
        start    = string,
        end      = string,
        interval = string,
      }),
    }),
    expiration_time = string,
    max_staleness   = optional(string),
    labels          = map(string),
  })) | `[]` | no |

| max\_time\_travel\_hours | Defines the time travel window in hours | `number` | `null` | no |

| project\_id | Project where the dataset and table are created | `string` | n/a | yes |

| resource\_tags | A map of resource tags to add to the dataset | `map(string)` | `{}` | no |

| routines | A list of objects which include routine\_id, routine\_type, routine\_language, definition\_body, return\_type, routine\_description and arguments. | list(object({
    routine_id      = string,
    routine_type    = string,
    language        = string,
    definition_body = string,
    return_type     = string,
    description     = string,
    arguments = list(object({
      name          = string,
      data_type     = string,
      argument_kind = string,
      mode          = string,
    })),
  })) | `[]` | no |

| storage\_billing\_model | Specifies the storage billing model for the dataset. Set this flag value to LOGICAL to use logical bytes for storage billing, or to PHYSICAL to use physical bytes instead. LOGICAL is the default if this flag isn't specified. | `string` | `null` | no |

| tables | A list of objects which include table\_id, table\_name, schema, clustering, time\_partitioning, range\_partitioning, expiration\_time and labels. | list(object({
    table_id                 = string,
    description              = optional(string),
    table_name               = optional(string),
    schema                   = string,
    clustering               = list(string),
    require_partition_filter = optional(bool),
    time_partitioning = object({
      expiration_ms = string,
      field         = string,
      type          = string,
    }),
    range_partitioning = object({
      field = string,
      range = object({
        start    = string,
        end      = string,
        interval = string,
      }),
    }),
    expiration_time     = string,
    deletion_protection = optional(bool),
    labels              = map(string),
  })) | `[]` | no |

| views | A list of objects which include view\_id and view query | list(object({
    view_id        = string,
    description    = optional(string),
    query          = string,
    use_legacy_sql = bool,
    labels         = map(string),
  }))
 | `[]` | no |

## Outputs

| Name | Description |

|------|-------------|

| bigquery\_dataset | Bigquery dataset resource. |

| bigquery\_external\_tables | Map of BigQuery external table resources being provisioned. |

| bigquery\_tables | Map of bigquery table resources being provisioned. |

| bigquery\_views | Map of bigquery view resources being provisioned. |

| external\_table\_ids | Unique IDs for any external tables being provisioned |

| external\_table\_names | Friendly names for any external tables being provisioned |

| project | Project where the dataset and tables are created |

| routine\_ids | Unique IDs for any routine being provisioned |

| table\_ids | Unique id for the table being provisioned |

| table\_names | Friendly name for the table being provisioned |

| view\_ids | Unique id for the view being provisioned |

| view\_names | friendlyname for the view being provisioned |

## Requirements

These sections describe requirements for using this module.

### Software

The following dependencies must be available:

- [Terraform](https://www.terraform.io/downloads.html) >= 0.13.0

- [Terraform Provider for GCP][terraform-provider-gcp] plugin v3

### Service Account

A service account with the following roles must be used to provision

the resources of this module:

- BigQuery Data Owner: `roles/bigquery.dataOwner`

The [Project Factory module][project-factory-module] and the

[IAM module][iam-module] may be used in combination to provision a

service account with the necessary roles applied.

#### Script Helper

A helper script for configuring a Service Account is located at (./helpers/setup-sa.sh).

### APIs

A project with the following APIs enabled must be used to host the

resources of this module:

- BigQuery JSON API: `bigquery-json.googleapis.com`

The [Project Factory module][project-factory-module] can be used to

provision a project with the necessary APIs enabled.

## Contributing

Refer to the [contribution guidelines](./CONTRIBUTING.md) for

information on contributing to this module.

[iam-module]: https://registry.terraform.io/modules/terraform-google-modules/iam/google

[project-factory-module]: https://registry.terraform.io/modules/terraform-google-modules/project-factory/google

[terraform-provider-gcp]: https://www.terraform.io/docs/providers/google/index.html

[terraform]: https://www.terraform.io/downloads.html
ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/terraform-google-modules/terraform-google-bigquery

Awesome Lists containing this project

README
