{"id":18706990,"url":"https://github.com/terraform-google-modules/terraform-google-bigquery","last_synced_at":"2025-11-09T08:30:32.629Z","repository":{"id":38445130,"uuid":"153479195","full_name":"terraform-google-modules/terraform-google-bigquery","owner":"terraform-google-modules","description":"Creates opinionated BigQuery datasets and tables","archived":false,"fork":false,"pushed_at":"2025-02-11T01:08:32.000Z","size":1793,"stargazers_count":205,"open_issues_count":8,"forks_count":177,"subscribers_count":39,"default_branch":"main","last_synced_at":"2025-02-11T02:22:40.425Z","etag":null,"topics":["cft-terraform","data-analytics"],"latest_commit_sha":null,"homepage":"https://registry.terraform.io/modules/terraform-google-modules/bigquery/google","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/terraform-google-modules.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-17T15:20:30.000Z","updated_at":"2025-02-04T04:25:53.000Z","dependencies_parsed_at":"2023-12-19T21:04:35.582Z","dependency_job_id":"b8f19f17-0249-4fd1-b799-5be92ccbb721","html_url":"https://github.com/terraform-google-modules/terraform-google-bigquery","commit_stats":null,"previous_names":[],"tags_count":31,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terraform-google-modules%2Fterraform-google-bigquery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terraform-google-modules%2Fterraform-google-bigquery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terraform-google-modules%2Fterraform-google-bigquery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terraform-google-modules%2Fterraform-google-bigquery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/terraform-google-modules","download_url":"https://codeload.github.com/terraform-google-modules/terraform-google-bigquery/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239571542,"owners_count":19661164,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cft-terraform","data-analytics"],"created_at":"2024-11-07T12:16:19.511Z","updated_at":"2025-11-09T08:30:32.597Z","avatar_url":"https://github.com/terraform-google-modules.png","language":"HCL","funding_links":[],"categories":["HCL"],"sub_categories":[],"readme":"# terraform-google-bigquery\n\nThis module allows you to create opinionated Google Cloud Platform BigQuery datasets and tables.\nThis will allow the user to programmatically create an empty table schema inside of a dataset, ready for loading.\nAdditional user accounts and permissions are necessary to begin querying the newly created table(s).\n\n## Compatibility\nThis module is meant for use with Terraform 0.13+ and tested using Terraform 1.0+. If you find incompatibilities using Terraform \u003e=0.13, please open an issue.\n If you haven't\n[upgraded](https://www.terraform.io/upgrade-guides/0-13.html) and need a Terraform\n0.12.x-compatible version of this module, the last released version\nintended for Terraform 0.12.x is [v4.5.0](https://registry.terraform.io/modules/terraform-google-modules/-bigquery/google/v4.5.0).\n\n## Upgrading\n\nThe current version is 4.X. The following guides are available to assist with upgrades:\n\n- [3.0 -\u003e 4.0](./docs/upgrading_to_bigquery_v4.0.md)\n- [2.0 -\u003e 3.0](./docs/upgrading_to_bigquery_v3.0.md)\n- [1.0 -\u003e 2.0](./docs/upgrading_to_bigquery_v2.0.md)\n- [0.1 -\u003e 1.0](./docs/upgrading_to_bigquery_v1.0.md)\n\n## Usage\n\nBasic usage of this module is as follows:\n\n```hcl\nmodule \"bigquery\" {\n  source  = \"terraform-google-modules/bigquery/google\"\n  version = \"~\u003e 9.0\"\n\n  dataset_id                  = \"foo\"\n  dataset_name                = \"foo\"\n  description                 = \"some description\"\n  project_id                  = \"\u003cPROJECT ID\u003e\"\n  location                    = \"US\"\n  default_table_expiration_ms = 3600000\n  resource_tags               = {\"\u003cPROJECT\u003e/\u003cTAG KEY\u003e\":\"\u003cTAG VALUE\u003e\"}\n\n  tables = [\n  {\n    table_id           = \"foo\",\n    schema             =  \"\u003cSCHEMA JSON DATA\u003e\",\n    time_partitioning  = {\n      type                     = \"DAY\",\n      field                    = null,\n      require_partition_filter = false,\n      expiration_ms            = null,\n    },\n    range_partitioning = null,\n    expiration_time = null,\n    clustering      = [\"fullVisitorId\", \"visitId\"],\n    labels          = {\n      env      = \"dev\"\n      billable = \"true\"\n      owner    = \"joedoe\"\n    },\n  },\n  {\n    table_id           = \"bar\",\n    schema             =  \"\u003cSCHEMA JSON DATA\u003e\",\n    time_partitioning  = null,\n    range_partitioning = {\n      field = \"customer_id\",\n      range = {\n        start    = \"1\"\n        end      = \"100\",\n        interval = \"10\",\n      },\n    },\n    expiration_time    = 2524604400000, # 2050/01/01\n    clustering         = [],\n    labels = {\n      env      = \"devops\"\n      billable = \"true\"\n      owner    = \"joedoe\"\n    }\n  }\n  ],\n\n  views = [\n    {\n      view_id    = \"barview\",\n      use_legacy_sql = false,\n      query          = \u003c\u003cEOF\n      SELECT\n       column_a,\n       column_b,\n      FROM\n        `project_id.dataset_id.table_id`\n      WHERE\n        approved_user = SESSION_USER\n      EOF,\n      labels = {\n        env      = \"devops\"\n        billable = \"true\"\n        owner    = \"joedoe\"\n      }\n    }\n  ]\n  dataset_labels = {\n    env      = \"dev\"\n    billable = \"true\"\n  }\n}\n```\n\nFunctional examples are included in the\n[examples](./examples/) directory.\n\n### Variable `tables` detailed description\n\nThe `tables` variable should be provided as a list of object with the following keys:\n```hcl\n{\n  table_id = \"some_id\"                        # Unique table id (will be used as ID for table).\n  table_name = \"Friendly Name\"                # Optional friendly name for table. If not set, the \"table_id\" will be used by default.\n  schema = file(\"path/to/schema.json\")        # Schema as JSON string.\n  time_partitioning = {                       # Set it to `null` to omit partitioning configuration for the table.\n        type                     = \"DAY\",     # The only type supported is DAY, which will generate one partition per day based on data loading time.\n        field                    = null,      # The field used to determine how to create a time-based partition. If time-based partitioning is enabled without this value, the table is partitioned based on the load time. Set it to `null` to omit configuration.\n        require_partition_filter = false,     # If set to true, queries over this table require a partition filter that can be used for partition elimination to be specified. Set it to `null` to omit configuration.\n        expiration_ms            = null,      # Number of milliseconds for which to keep the storage for a partition.\n      },\n  range_partitioning = {                      # Set it to `null` to omit partitioning configuration for the table.\n    field = \"integer_column\",                 # The column used to create the integer range partitions.\n    range = {\n      start    = \"1\"                          # The start of range partitioning, inclusive.\n      end      = \"100\",                       # The end of range partitioning, exclusive.\n      interval = \"10\",                        # The width of each range within the partition.\n        },\n      },\n  clustering = [\"fullVisitorId\", \"visitId\"]   # Specifies column names to use for data clustering. Up to four top-level columns are allowed, and should be specified in descending priority order. Partitioning should be configured in order to use clustering.\n  expiration_time = 2524604400000             # The time when this table expires, in milliseconds since the epoch. If set to `null`, the table will persist indefinitely.\n  deletion_protection = true                  # Optional. Configures deletion_protection for the table. If unset, module-level deletion_protection setting will be used.\n  labels = {                                  # A mapping of labels to assign to the table.\n      env      = \"dev\"\n      billable = \"true\"\n    }\n}\n```\n\n### Variable `views` detailed description\n\nThe `views` variable should be provided as a list of object with the following keys:\n```hcl\n{\n  view_id = \"some_id\"                                                # Unique view id. it will be set to friendly name as well\n  query = \"Select user_id, name from `project_id.dataset_id.table`\"  # the Select query that will create the view. Tables should be created before.\n  use_legacy_sql = false                                             # whether to use legacy sql or standard sql\n  labels = {                                                         # A mapping of labels to assign to the view.\n      env      = \"dev\"\n      billable = \"true\"\n  }\n}\n```\n\n### Variable `routines` detailed description\n\nThe `routines` variable should be provided as a list of object with the following keys:\n```hcl\n{\n  routine_id = \"some_id\"                     # The ID of the routine. The ID must contain only letters, numbers, or underscores. The maximum length is 256 characters.\n  routine_type = \"PROCEDURE\"                 # The type of routine. Possible values are SCALAR_FUNCTION and PROCEDURE.\n  language = \"SQL\"                           # The language of the routine. Possible values are SQL and JAVASCRIPT.\n  definition_body = \"CREATE FUNCTION test return x*y;\"  # The body of the routine. For functions, this is the expression in the AS clause. If language=SQL, it is the substring inside (but excluding) the parentheses.\n  return_type     = null                     # A JSON schema for the return type. Optional if language = \"SQL\"; required otherwise. If absent, the return type is inferred from definitionBody at query time in each query that references this routine. If present, then the evaluated result will be cast to the specified returned type at query time.\n  description = \"Description\"               # The description of the routine if defined.\n  arguments = [                             # Set it to `null` to omit arguments block configuration for the routine.\n    {\n      name      = \"x\",                      # The name of this argument. Can be absent for function return argument.\n      data_type = null,                     # A JSON schema for the data type. Required unless argumentKind = ANY_TYPE.\n      argument_kind = \"ANY_TYPE\"            # Defaults to FIXED_TYPE. Default value is FIXED_TYPE. Possible values are FIXED_TYPE and ANY_TYPE.\n      mode = null                           # Specifies whether the argument is input or output. Can be set for procedures only. Possible values are IN, OUT, and INOUT.\n    }\n  ]\n}\n```\nA detailed example with authorized views can be found [here](./examples/basic_view/main.tf).\n\n## Features\nThis module provisions a dataset and a list of tables with associated JSON schemas and views from queries.\n\n\u003c!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK --\u003e\n## Inputs\n\n| Name | Description | Type | Default | Required |\n|------|-------------|------|---------|:--------:|\n| access | An array of objects that define dataset access for one or more entities. | `any` | \u003cpre\u003e[\u003cbr\u003e  {\u003cbr\u003e    \"role\": \"roles/bigquery.dataOwner\",\u003cbr\u003e    \"special_group\": \"projectOwners\"\u003cbr\u003e  }\u003cbr\u003e]\u003c/pre\u003e | no |\n| dataset\\_id | Unique ID for the dataset being provisioned. | `string` | n/a | yes |\n| dataset\\_labels | Key value pairs in a map for dataset labels | `map(string)` | `{}` | no |\n| dataset\\_name | Friendly name for the dataset being provisioned. | `string` | `null` | no |\n| default\\_partition\\_expiration\\_ms | The default partition expiration for all partitioned tables in the dataset, in MS | `number` | `null` | no |\n| default\\_table\\_expiration\\_ms | TTL of tables using the dataset in MS | `number` | `null` | no |\n| delete\\_contents\\_on\\_destroy | (Optional) If set to true, delete all the tables in the dataset when destroying the resource; otherwise, destroying the resource will fail if tables are present. | `bool` | `null` | no |\n| deletion\\_protection | Whether or not to allow deletion of tables and external tables defined by this module. Can be overriden by table-level deletion\\_protection configuration. | `bool` | `false` | no |\n| description | Dataset description. | `string` | `null` | no |\n| encryption\\_key | Default encryption key to apply to the dataset. Defaults to null (Google-managed). | `string` | `null` | no |\n| external\\_tables | A list of objects which include table\\_id, expiration\\_time, external\\_data\\_configuration, and labels. | \u003cpre\u003elist(object({\u003cbr\u003e    table_id              = string,\u003cbr\u003e    description           = optional(string),\u003cbr\u003e    autodetect            = bool,\u003cbr\u003e    compression           = string,\u003cbr\u003e    ignore_unknown_values = bool,\u003cbr\u003e    max_bad_records       = number,\u003cbr\u003e    schema                = string,\u003cbr\u003e    source_format         = string,\u003cbr\u003e    source_uris           = list(string),\u003cbr\u003e    csv_options = object({\u003cbr\u003e      quote                 = string,\u003cbr\u003e      allow_jagged_rows     = bool,\u003cbr\u003e      allow_quoted_newlines = bool,\u003cbr\u003e      encoding              = string,\u003cbr\u003e      field_delimiter       = string,\u003cbr\u003e      skip_leading_rows     = number,\u003cbr\u003e    }),\u003cbr\u003e    google_sheets_options = object({\u003cbr\u003e      range             = string,\u003cbr\u003e      skip_leading_rows = number,\u003cbr\u003e    }),\u003cbr\u003e    hive_partitioning_options = object({\u003cbr\u003e      mode              = string,\u003cbr\u003e      source_uri_prefix = string,\u003cbr\u003e    }),\u003cbr\u003e    expiration_time     = string,\u003cbr\u003e    max_staleness       = optional(string),\u003cbr\u003e    deletion_protection = optional(bool),\u003cbr\u003e    labels              = map(string),\u003cbr\u003e  }))\u003c/pre\u003e | `[]` | no |\n| location | The regional location for the dataset only US and EU are allowed in module | `string` | `\"US\"` | no |\n| materialized\\_views | A list of objects which includes view\\_id, view\\_query, clustering, time\\_partitioning, range\\_partitioning, expiration\\_time and labels | \u003cpre\u003elist(object({\u003cbr\u003e    view_id             = string,\u003cbr\u003e    description         = optional(string),\u003cbr\u003e    query               = string,\u003cbr\u003e    enable_refresh      = bool,\u003cbr\u003e    refresh_interval_ms = string,\u003cbr\u003e    clustering          = list(string),\u003cbr\u003e    time_partitioning = object({\u003cbr\u003e      expiration_ms            = string,\u003cbr\u003e      field                    = string,\u003cbr\u003e      type                     = string,\u003cbr\u003e      require_partition_filter = bool,\u003cbr\u003e    }),\u003cbr\u003e    range_partitioning = object({\u003cbr\u003e      field = string,\u003cbr\u003e      range = object({\u003cbr\u003e        start    = string,\u003cbr\u003e        end      = string,\u003cbr\u003e        interval = string,\u003cbr\u003e      }),\u003cbr\u003e    }),\u003cbr\u003e    expiration_time = string,\u003cbr\u003e    max_staleness   = optional(string),\u003cbr\u003e    labels          = map(string),\u003cbr\u003e  }))\u003c/pre\u003e | `[]` | no |\n| max\\_time\\_travel\\_hours | Defines the time travel window in hours | `number` | `null` | no |\n| project\\_id | Project where the dataset and table are created | `string` | n/a | yes |\n| resource\\_tags | A map of resource tags to add to the dataset | `map(string)` | `{}` | no |\n| routines | A list of objects which include routine\\_id, routine\\_type, routine\\_language, definition\\_body, return\\_type, routine\\_description and arguments. | \u003cpre\u003elist(object({\u003cbr\u003e    routine_id      = string,\u003cbr\u003e    routine_type    = string,\u003cbr\u003e    language        = string,\u003cbr\u003e    definition_body = string,\u003cbr\u003e    return_type     = string,\u003cbr\u003e    description     = string,\u003cbr\u003e    arguments = list(object({\u003cbr\u003e      name          = string,\u003cbr\u003e      data_type     = string,\u003cbr\u003e      argument_kind = string,\u003cbr\u003e      mode          = string,\u003cbr\u003e    })),\u003cbr\u003e  }))\u003c/pre\u003e | `[]` | no |\n| storage\\_billing\\_model | Specifies the storage billing model for the dataset. Set this flag value to LOGICAL to use logical bytes for storage billing, or to PHYSICAL to use physical bytes instead. LOGICAL is the default if this flag isn't specified. | `string` | `null` | no |\n| tables | A list of objects which include table\\_id, table\\_name, schema, clustering, time\\_partitioning, range\\_partitioning, expiration\\_time and labels. | \u003cpre\u003elist(object({\u003cbr\u003e    table_id                 = string,\u003cbr\u003e    description              = optional(string),\u003cbr\u003e    table_name               = optional(string),\u003cbr\u003e    schema                   = string,\u003cbr\u003e    clustering               = list(string),\u003cbr\u003e    require_partition_filter = optional(bool),\u003cbr\u003e    time_partitioning = object({\u003cbr\u003e      expiration_ms = string,\u003cbr\u003e      field         = string,\u003cbr\u003e      type          = string,\u003cbr\u003e    }),\u003cbr\u003e    range_partitioning = object({\u003cbr\u003e      field = string,\u003cbr\u003e      range = object({\u003cbr\u003e        start    = string,\u003cbr\u003e        end      = string,\u003cbr\u003e        interval = string,\u003cbr\u003e      }),\u003cbr\u003e    }),\u003cbr\u003e    expiration_time     = string,\u003cbr\u003e    deletion_protection = optional(bool),\u003cbr\u003e    labels              = map(string),\u003cbr\u003e  }))\u003c/pre\u003e | `[]` | no |\n| views | A list of objects which include view\\_id and view query | \u003cpre\u003elist(object({\u003cbr\u003e    view_id        = string,\u003cbr\u003e    description    = optional(string),\u003cbr\u003e    query          = string,\u003cbr\u003e    use_legacy_sql = bool,\u003cbr\u003e    labels         = map(string),\u003cbr\u003e  }))\u003c/pre\u003e | `[]` | no |\n\n## Outputs\n\n| Name | Description |\n|------|-------------|\n| bigquery\\_dataset | Bigquery dataset resource. |\n| bigquery\\_external\\_tables | Map of BigQuery external table resources being provisioned. |\n| bigquery\\_tables | Map of bigquery table resources being provisioned. |\n| bigquery\\_views | Map of bigquery view resources being provisioned. |\n| external\\_table\\_ids | Unique IDs for any external tables being provisioned |\n| external\\_table\\_names | Friendly names for any external tables being provisioned |\n| project | Project where the dataset and tables are created |\n| routine\\_ids | Unique IDs for any routine being provisioned |\n| table\\_ids | Unique id for the table being provisioned |\n| table\\_names | Friendly name for the table being provisioned |\n| view\\_ids | Unique id for the view being provisioned |\n| view\\_names | friendlyname for the view being provisioned |\n\n\u003c!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK --\u003e\n\n## Requirements\n\nThese sections describe requirements for using this module.\n\n### Software\n\nThe following dependencies must be available:\n\n- [Terraform](https://www.terraform.io/downloads.html) \u003e= 0.13.0\n- [Terraform Provider for GCP][terraform-provider-gcp] plugin v3\n\n### Service Account\n\nA service account with the following roles must be used to provision\nthe resources of this module:\n\n- BigQuery Data Owner: `roles/bigquery.dataOwner`\n\nThe [Project Factory module][project-factory-module] and the\n[IAM module][iam-module] may be used in combination to provision a\nservice account with the necessary roles applied.\n\n#### Script Helper\nA helper script for configuring a Service Account is located at (./helpers/setup-sa.sh).\n\n### APIs\n\nA project with the following APIs enabled must be used to host the\nresources of this module:\n\n- BigQuery JSON API: `bigquery-json.googleapis.com`\n\nThe [Project Factory module][project-factory-module] can be used to\nprovision a project with the necessary APIs enabled.\n\n## Contributing\n\nRefer to the [contribution guidelines](./CONTRIBUTING.md) for\ninformation on contributing to this module.\n\n[iam-module]: https://registry.terraform.io/modules/terraform-google-modules/iam/google\n[project-factory-module]: https://registry.terraform.io/modules/terraform-google-modules/project-factory/google\n[terraform-provider-gcp]: https://www.terraform.io/docs/providers/google/index.html\n[terraform]: https://www.terraform.io/downloads.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fterraform-google-modules%2Fterraform-google-bigquery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fterraform-google-modules%2Fterraform-google-bigquery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fterraform-google-modules%2Fterraform-google-bigquery/lists"}