{"id":18304037,"url":"https://github.com/googleclouddataproc/hive-bigquery-connector","last_synced_at":"2025-04-05T15:31:05.991Z","repository":{"id":37798890,"uuid":"493492444","full_name":"GoogleCloudDataproc/hive-bigquery-connector","owner":"GoogleCloudDataproc","description":"A library enabling BigQuery as Hive storage handler","archived":false,"fork":false,"pushed_at":"2025-01-29T19:22:44.000Z","size":913,"stargazers_count":10,"open_issues_count":20,"forks_count":15,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-21T06:33:26.571Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GoogleCloudDataproc.png","metadata":{"files":{"readme":"README-template.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-18T03:17:16.000Z","updated_at":"2025-02-18T16:35:00.000Z","dependencies_parsed_at":"2024-01-05T18:29:00.568Z","dependency_job_id":"79097f0f-a5a9-4970-ba25-df55a8c79a94","html_url":"https://github.com/GoogleCloudDataproc/hive-bigquery-connector","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fhive-bigquery-connector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fhive-bigquery-connector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fhive-bigquery-connector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fhive-bigquery-connector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GoogleCloudDataproc","download_url":"https://codeload.github.com/GoogleCloudDataproc/hive-bigquery-connector/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247358523,"owners_count":20926235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T15:27:36.718Z","updated_at":"2025-04-05T15:31:04.961Z","avatar_url":"https://github.com/GoogleCloudDataproc.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Hive-BigQuery Connector\n\nThe Hive-BigQuery Connector is a Hive storage handler that enables Hive to interact with BigQuery's\nstorage layer. It allows you to run queries in Hive using the HiveQL dialect to read from and write\nto BigQuery.\n\n## Release notes\n\nSee the details in [CHANGES.md](CHANGES.md).\n\n## Version support\n\nThis connector supports [Dataproc](https://cloud.google.com/dataproc) 2.0, 2.1, and 2.2.\n\nFor Hadoop clusters other than Dataproc, the connector has been tested with the following\nsoftware versions:\n\n* Hive 1.2.1, 2.3.6, 2.3.9, 3.1.2, and 3.1.3.\n* Hadoop 2.6.4, 2.7.0, 2.10.2, 3.2.3, and 3.3.3.\n* Tez 0.9.2 on Hadoop 2, and Tez 0.10.1 on Hadoop 3.\n* Pig 0.16.0, 0.17.0.\n\n## Installation\n\n### Prerequisite\n\nMake sure you have the BigQuery Storage API enabled in your GCP project. Follow [these instructions](https://cloud.google.com/bigquery/docs/reference/storage/#enabling_the_api).\n\n### Option 1: connectors init action\n\nFor Dataproc clusters, the most convenient way to install the Hive-BigQuery\nconnector is to use the [connectors init action](https://github.com/GoogleCloudDataproc/initialization-actions/tree/master/connectors).\n\n### Option 2: manual installation\n\nYou can also download an official release JAR from [Maven Central](https://mvnrepository.com/artifact/com.google.cloud.hive/hive-bigquery-connector).\n\nAlternately, you can build a JAR from source:\n\n1. Clone this repository:\n   ```sh\n   git clone https://github.com/GoogleCloudPlatform/hive-bigquery-connector\n   cd hive-bigquery-connector\n   ```\n\n2. Compile and package the jar:\n\n  * For Hive 1:\n\n    ``` sh\n    ./mvnw package -DskipTests -P hive1-generic\n    ```\n\n  * For Hive 2:\n\n    ``` sh\n    ./mvnw package -DskipTests -P hive2-generic\n    ```\n\n    The packaged jar is now available at: `hive-2-bigquery-connector/target/hive-2-bigquery-connector-\u003cversion\u003e.jar`\n\n  * For Hive 3:\n\n    ``` sh\n    ./mvnw package -DskipTests -P hive3-generic\n    ```\n\n    The packaged jar is now available at: `hive-3-bigquery-connector/target/hive-3-bigquery-connector-\u003cversion\u003e.jar`\n\nOnce you have the connector JAR, deploy the JAR to the classpath of all nodes in your Hive cluster.\n\nYou can also provide the JAR as a parameter when starting a Hive or Beeline session:\n\n   ```sh\n   hive --auxpath \u003cjar path\u003e/hive-bigquery-connector-\u003cversion\u003e.jar\n   ```\n   ```sh\n   beeline \u003e add jar \u003cjar path\u003e/hive-bigquery-connector-\u003cversion\u003e.jar;\n   ```\n\n## Managed tables vs external tables\n\nHive can have [two types](https://cwiki.apache.org/confluence/display/Hive/Managed+vs.+External+Tables)\nof tables:\n\n- Managed tables, sometimes referred to as internal tables.\n- External tables.\n\nThe Hive BigQuery connector supports both types in the following ways.\n\n### Managed tables\n\nWhen you create a managed table using the `CREATE TABLE` statement, the connector creates both\nthe table metadata in the Hive Metastore and a new table in BigQuery with the same schema.\n\nHere's an example:\n\n```sql\nCREATE TABLE mytable (word_count BIGINT, word STRING)\nSTORED BY 'com.google.cloud.hive.bigquery.connector.BigQueryStorageHandler'\nTBLPROPERTIES (\n    'bq.table'='myproject.mydataset.mytable'\n);\n```\n\nWhen you drop a managed table using the `DROP TABLE` statement, the connector drops both the table\nmetadata from the Hive Metastore and the BigQuery table (including all of its data).\n\nFor Hive-3.x, create a managed table with `NOT NULL` column restraint will not create the BigQuery table\nwith corresponding `NOT NULL` restraint. The `NOT NULL` restraint is still enforced by Hive at runtime.\nIt is recommended to use external table if user needs to have the `NOT NULL` restraint on the BigQuery table.\n\n### External tables\n\nWhen you create an external table using the `CREATE EXTERNAL TABLE` statement, the connector only\ncreates the table metadata in the Hive Metastore. It assumes that the corresponding table in\nBigQuery already exists.\n\nHere's an example:\n\n```sql\nCREATE EXTERNAL TABLE mytable (word_count BIGINT, word STRING)\nSTORED BY 'com.google.cloud.hive.bigquery.connector.BigQueryStorageHandler'\nTBLPROPERTIES (\n    'bq.table'='myproject.mydataset.mytable'\n);\n```\n\nWhen you drop an external table using the `DROP TABLE` statement, the connector only drops the table\nmetadata from the Hive Metastore. The corresponding BigQuery table remains unaffected.\n\n### Statistics For Hive Query Planning\n\nIt is recommended to collect [statistics](https://cwiki.apache.org/confluence/display/hive/statsdev)\n(e.g. the number of rows, raw data size, etc) to help Hive to optimize query plan,\ntherefore improving performance. Follow these steps to collect statistics for a table:\n(replace `\u003ctable_name\u003e` with your table name)\n\nIf your Hive has [HIVE-24928](https://issues.apache.org/jira/browse/HIVE-24928) (this is applied on Dataproc),\nstatistics can be collected by a quick metadata collection operation through\n```sql\nANALYZE TABLE \u003ctable_name\u003e COMPUTE STATISTICS;\n```\nTo verify if Hive has reasonable statistics on `numRows` and `rawDataSize` for your table.\n   ```sql\n   DESCRIBE FORMATTED \u003ctable_name\u003e;\n   ```\nExample output:\n   ```\n   Table Parameters:\n       COLUMN_STATS_ACCURATE\t{\\\"BASIC_STATS\\\":\\\"true\\\"}\n       numFiles            \t0\n       numRows             \t12345\n       rawDataSize         \t67890\n       totalSize           \t34567\n   ```\n\nWorkaround for without [HIVE-24928](https://issues.apache.org/jira/browse/HIVE-24928):\n\nUser can run\n```sql\nANALYZE TABLE \u003ctable_name\u003e COMPUTE STATISTICS FOR COLUMNS;\n```\nto collect column stats and run query with the assistance of column stats.\n```sql\nSET hive.stats.fetch.column.stats=true;\n```\nColumn stats collection is more expensive operation, so it is recommended to have [HIVE-24928](https://issues.apache.org/jira/browse/HIVE-24928) applied.\n\n## Partitioning\n\nAs Hive's partitioning and BigQuery's partitioning inherently work in different ways, the Hive\n`PARTITIONED BY` clause is not supported. However, you can still leverage BigQuery's native\npartitioning by specifying some table properties. Two types of BigQuery native partitioning are\ncurrently supported: time-unit column partitioning and ingestion time partitioning.\n\nNote: This section is about BigQuery native partitioning. To learn about integrating with\npartitioned tables in open formats like Parquet or ORC, refer to the section on\n[BigLake integration](#bigLake-integration).\n\n### Time-unit column partitioning\n\nYou can partition a BigQuery table on a column of BigQuery types `DATE`, `DATETIME`, or\n`TIMESTAMP`, which respectively correspond to the Hive types `DATE`, `TIMESTAMP`, and\n`TIMESTAMPLOCALTZ`. When you write data to the table, BigQuery automatically puts the data into the\ncorrect partition based on the values in the column.\n\nFor the `DATETIME` and `TIMESTAMP` BigQuery types, the partitions can have either hourly, daily,\nmonthly, or yearly granularity. For the `DATE` type, the partitions can have daily, monthly, or\nyearly granularity. Partition boundaries are based on UTC time.\n\nTo create a table partitioned by a time-unit column, you must set the `bq.time.partition.field`\ntable property to the column's name.\n\nHere's an example:\n\n```sql\nCREATE TABLE mytable (int_val BIGINT, ts TIMESTAMP)\nSTORED BY 'com.google.cloud.hive.bigquery.connector.BigQueryStorageHandler'\nTBLPROPERTIES (\n    'bq.table'='myproject.mydataset.mytable',\n    'bq.time.partition.field'='ts',\n    'bq.time.partition.type'='MONTH'\n);\n```\n\nCheck out the official BigQuery documentation about [Time-unit column partitioning](https://cloud.google.com/bigquery/docs/partitioned-tables#date_timestamp_partitioned_tables)\nto learn more.\n\n### Ingestion time partitioning\n\nWhen you create a table partitioned by ingestion time, BigQuery automatically assigns rows to\npartitions based on the time when BigQuery ingests the data. You can choose hourly, daily, monthly,\nor yearly boundaries for the partitions. Partitions boundaries are based on UTC time.\n\nAn ingestion time partitioned table also has two pseudo columns:\n\n- `_PARTITIONTIME`: ingestion time for each row, truncated to the partition boundary (such as hourly\n  or daily). This column has the `DATE` Hive type, which corresponds to the BigQuery `DATE` type.\n- `_PARTITIONDATE`: UTC date corresponding to the value in the `_PARTITIONTIME` pseudo column. This\n  column has the `TIMESTAMPLOCALTZ` Hive type, which corresponds to the BigQuery `TIMESTAMP` type.\n\nTo create a table partitioned by ingestion time, you must set the `bq.time.partition.type` table\nproperty to the partition boundary of your choice (`HOUR`, `DAY`, `MONTH`, or `YEAR`).\n\nHere's an example:\n\n```sql\nCREATE TABLE mytable (int_val BIGINT)\nSTORED BY 'com.google.cloud.hive.bigquery.connector.BigQueryStorageHandler'\nTBLPROPERTIES (\n    'bq.table'='myproject.mydataset.mytable',\n    'bq.time.partition.type'='DAY'\n);\n```\n\nNote: Ingestion time partitioning is currently supported only for read operations.\n\nCheck out the official BigQuery documentation about [Ingestion time partitioning](https://cloud.google.com/bigquery/docs/partitioned-tables#ingestion_time)\nto learn more.\n\n## Clustering\n\nAs Hive's clustering and BigQuery's clustering inherently work in different ways, the Hive\n`CLUSTERED BY` clause is not supported. However, you can still leverage BigQuery's native clustering\nby setting the `bq.clustered.fields` table property to a comma-separated list of the columns to\ncluster the table by.\n\nHere's an example:\n\n```sql\nCREATE TABLE mytable (int_val BIGINT, text STRING, purchase_date DATE)\nSTORED BY 'com.google.cloud.hive.bigquery.connector.BigQueryStorageHandler'\nTBLPROPERTIES (\n    'bq.table'='myproject.mydataset.mytable',\n    'bq.clustered.fields'='int_val,text'\n);\n```\n\nCheck out the official BigQuery documentation about [Clustering](https://cloud.google.com/bigquery/docs/clustered-tables)\nto learn more.\n\n## Table properties\n\nYou can use the following properties in the `TBLPROPERTIES` clause when you create a new table:\n\n| Property                           | Description                                                                                                                                                                                                                                                       |\n|------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `bq.table`                         | Always required. BigQuery table name in the format of project.dataset.table                                                                                                                                                                                       |\n| `bq.time.partition.type`           | Time partitioning granularity. Possible values: `HOUR`, `DAY`, `MONTH`, `YEAR`                                                                                                                                                                                    |\n| `bq.time.partition.field`          | Name of a `DATE` or `TIMESTAMP` column to partition the table by                                                                                                                                                                                                  |\n| `bq.time.partition.expiration.ms`  | Partition [expiration time](https://cloud.google.com/bigquery/docs/managing-partitioned-tables#partition-expiration) in milliseconds                                                                                                                              |\n| `bq.time.partition.require.filter` | Set it to `true` to require that all queries on the table [must include a predicate filter]((https://cloud.google.com/bigquery/docs/managing-partitioned-tables#require-filter)) (a `WHERE` clause) that filters on the partitioning column. Defaults to `false`. |\n| `bq.clustered.fields`              | Comma-separated list of fields to cluster the table by                                                                                                                                                                                                            |                                                                                                                                                                                                 |\n\n## Job configuration properties\n\nYou can set the following Hive/Hadoop configuration properties in your environment:\n\n| Property                            | Default value       | Description                                                                                                                                                                                         |\n|-------------------------------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `bq.read.data.format`               | `arrow`             | Data format used for reads from BigQuery. Possible values: `arrow`, `avro`.                                                                                                                         |\n| `bq.temp.gcs.path`                  |                     | GCS location for storing temporary Avro files when using the `indirect` write method                                                                                                                |\n| `bq.write.method`                   | `direct`            | Indicates how to write data to BigQuery. Possible values: `direct` (to directly write to the BigQuery storage API), `indirect` (to stage temporary Avro files to GCS before loading into BigQuery). |\n| `bq.work.dir.parent.path`           | `${hadoop.tmp.dir}` | Parent path on HDFS where each job creates its temporary work directory                                                                                                                             |\n| `bq.work.dir.name.prefix`           | `hive-bq-`          | Prefix used for naming the jobs' temporary directories.                                                                                                                                             |\n| `bq.destination.table.kms.key.name` |                     | Cloud KMS encryption key used to protect the job's destination BigQuery table. Read more in the section on [customer-managed encryption keys](#customer-managed-encryption-keys)                    |\n| `materializationProject`            |                     | Project used to temporarily materialize data when reading views. Defaults to the same project as the read view.                                                                                     |\n| `materializationDataset`            |                     | Dataset used to temporarily materialize data when reading views. Defaults to the same dataset as the read view.                                                                                     |\n| `maxParallelism`                    |                     | Maximum initial number of read streams                                                                                                                                                              |\n| `viewsEnabled`                      | `false`             | Set it to `true` to enable reading views.                                                                                                                                                           |\n\n## Data Type Mapping\n\nAdd links to Hive \u0026 BQ types doc.\n\n| Hive               | Hive type description                                                                             | BigQuery        | BigQuery type description                                                                                                                        |\n|--------------------|---------------------------------------------------------------------------------------------------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------------|\n| `TINYINT`          | 1-byte signed integer                                                                             | `INT64`         |                                                                                                                                                  |\n| `SMALLINT`         | 2-byte signed integer                                                                             | `INT64`         |                                                                                                                                                  |\n| `INT`              | 4-byte signed integer                                                                             | `INT64`         |                                                                                                                                                  |\n| `BIGINT`           | 8-byte signed integer                                                                             | `INT64`         |                                                                                                                                                  |\n| `FLOAT`            | 4-byte single precision floating point number                                                     | `FLOAT64`       |                                                                                                                                                  |\n| `DOUBLE`           | 8-byte double precision floating point number                                                     | `FLOAT64`       |                                                                                                                                                  |\n| `DECIMAL`          | Alias of `NUMERIC`. Precision: 38. Scale: 38                                                      | `DECIMAL`       | Alias of `NUMERIC`. Precision: 38. Scale: 9.                                                                                                     |\n| `DATE`             | Format: `YYYY-MM-DD`                                                                              | `DATE`          | Format: `YYYY-[M]M-[D]D`. Supported range: 0001-01-01 to 9999-12-31                                                                              |\n| `TIMESTAMP`        | Timezone-less timestamp stored as an offset from the UNIX epoch                                   | `DATETIME`      | A date and time, as they might be displayed on a watch, independent of time zone.                                                                |\n| `TIMESTAMPLOCALTZ` | Timezoned timestamp stored as an offset from the UNIX epoch                                       | `TIMESTAMP`     | Absolute point in time, independent of any time zone or convention such as Daylight Savings Time                                                 |\n| `BOOLEAN`          | Boolean values are represented by the keywords TRUE and FALSE                                     | `BOOLEAN`       |                                                                                                                                                  |\n| `CHAR`             | Variable-length character data                                                                    | `STRING`        |                                                                                                                                                  |\n| `VARCHAR`          | Variable-length character data                                                                    | `STRING`        |                                                                                                                                                  |\n| `STRING`           | Variable-length character data                                                                    | `STRING`        |                                                                                                                                                  |\n| `BINARY`           | Variable-length binary data                                                                       | `BYTES`         |                                                                                                                                                  |\n| `ARRAY`            | Represents repeated values                                                                        | `ARRAY`         |                                                                                                                                                  |\n| `STRUCT`           | Represents nested structures                                                                      | `STRUCT`        |                                                                                                                                                  |\n| `MAP`              | Dictionary of keys and values. Keys must be of primitive type, whereas values can be of any type. | `ARRAY\u003cSTRUCT\u003e` | BigQuery doesn't support Maps natively. The connector implements it as a list of structs, where each struct has two columns: `name` and `value`. |\n\n## Execution engines\n\nThe BigQuery storage handler supports both the MapReduce and Tez execution engines. Tez is recommended for better\nperformance -- you can use it by setting the `hive.execution.engine=tez` configuration property.\n\n## Column Pruning\n\nSince BigQuery is [backed by a columnar datastore](https://cloud.google.com/blog/big-data/2016/04/inside-capacitor-bigquerys-next-generation-columnar-storage-format),\nit can efficiently stream data without reading all columns.\n\n## Predicate pushdowns\n\nThe connector supports predicate pushdowns to the BigQuery Storage Read API. This allows to filter\ndata at the BigQuery storage layer, which reduces the amount of data traversing the network and\nimproves overall performance.\n\nMany built-in Hive UDFs and operators (e.g. `AND`, `OR`, `ABS`, `TRIM`, `BETWEEN`...) are identical\nin BigQuery, so the connector pushes those as-is to BigQuery.\n\nHowever, some Hive UDFs and operators are different in BigQuery. So the connector automatically\nconverts those to the equivalent functions in BigQuery. Below is the list of UDFs and operators that\nare automatically converted:\n\n| Hive generic UDF | BigQuery function             | Notes                                                                                     |\n|------------------|-------------------------------|-------------------------------------------------------------------------------------------|\n| `%`              | `MOD`                         | BigQuery currently supports `MOD` only for the `INT64`, `NUMERIC`, and `BIGNUMERIC` types |\n| `TO_DATE`        | `DATE`                        |                                                                                           |\n| `DATE_ADD`       | `DATE_ADD`                    |                                                                                           |\n| `DATE_SUB`       | `DATE_SUB`                    |                                                                                           |\n| `DATEDIFF`       | `DATE_DIFF`                   |                                                                                           |\n| `DATEDIFF`       | `DATE_DIFF`                   |                                                                                           |\n| `HEX`            | `TO_HEX`                      |                                                                                           |\n| `UNHEX`          | `FROM_HEX`                    |                                                                                           |\n| `NVL`            | `IFNULL`                      |                                                                                           |\n| `RLIKE`          | `REGEXP_CONTAINS`             |                                                                                           |\n| `SHIFTLEFT`      | `\u003c\u003c`                          |                                                                                           |\n| `SHIFTRIGHT`     | `\u003e\u003e`                          |                                                                                           |\n| `YEAR`           | `EXTRACT(YEAR FROM ...)`      |                                                                                           |\n| `MONTH`          | `EXTRACT(MONTH FROM ...)`     |                                                                                           |\n| `DAY`            | `EXTRACT(DAY FROM ...)`       |                                                                                           |\n| `HOUR`           | `EXTRACT(HOUR FROM ...)`      |                                                                                           |\n| `MINUTE`         | `EXTRACT(MINUTE FROM ...)`    |                                                                                           |\n| `SECOND`         | `EXTRACT(SECOND FROM ...)`    |                                                                                           |\n| `DAYOFWEEK`      | `EXTRACT(DAYOFWEEK FROM ...)` |                                                                                           |\n| `WEEKOFYEAR`     | `EXTRACT(WEEK FROM ...)`      |                                                                                           |\n| `QUARTER`        | `EXTRACT(QUARTER FROM ...)`   |                                                                                           |\n\nNote: [Custom Hive UDFs](https://cwiki.apache.org/confluence/display/hive/hiveplugins) (aka Hive\nplugins) are currently not supported in predicate pushdowns. If a query contains a custom Hive UDF\nin a `WHERE` filter, then the custom UDF will not be pushed down to BigQuery and will instead be\nprocessed by the Hive query engine.\n\n## Parallelism\n\n### Parallel reads\n\nThe connector allows parallel reads from BigQuery by using the\n[BigQuery Storage API](https://cloud.google.com/bigquery/docs/reference/storage).\n\nYou can set the `preferredMinParallelism` configuration property, which the connector passes to\n[`CreateReadSessionRequest.setPreferredMinStreamCount()`](https://cloud.google.com/java/docs/reference/google-cloud-bigquerystorage/latest/com.google.cloud.bigquery.storage.v1.CreateReadSessionRequest.Builder#com_google_cloud_bigquery_storage_v1_CreateReadSessionRequest_Builder_setPreferredMinStreamCount_int_)\nwhen it creates the BigQuery read session. This parameter can be used to inform the BigQuery service that there is a\ndesired lower bound on the number of streams. This is typically the target parallelism of the client (e.g. a Hive\ncluster with N-workers would set this to a low multiple of N to ensure good cluster utilization). The BigQuery backend\nmakes a best effort to provide at least this number of streams, but in some cases might provide less.\n\nAdditionally, you can set the `maxParallelism` configuration property, which the connector passes to\n[CreateReadSessionRequest.setMaxStreamCount()](https://cloud.google.com/java/docs/reference/google-cloud-bigquerystorage/latest/com.google.cloud.bigquery.storage.v1.CreateReadSessionRequest.Builder#com_google_cloud_bigquery_storage_v1_CreateReadSessionRequest_Builder_setMaxStreamCount_int_).\nIf unset or zero, the BigQuery backend server will provide a value of streams to produce reasonable throughput. The\nnumber of streams may be lower than the requested number, depending on the amount parallelism that is reasonable for\nthe table. There is a default system max limit of 1,000. This must be greater than or equal to the number of MapReduce\nsplits. Typically, a client should either leave this unset to let the system determine an upper bound or set this as the\nmaximum \"units of work\" that the client can gracefully handle.\n\nThe connector supports both the [Arrow](https://cloud.google.com/bigquery/docs/reference/storage#arrow_schema_details)\nand [Avro](https://cloud.google.com/bigquery/docs/reference/storage#avro_schema_details) read formats. You can specify\nwhich format the connector should use by setting the `bq.read.data.format` configuration property to either `arrow` or\n`avro`. The connector uses Arrow by default as Arrow generally performs better than Avro.\n\n### Parallel writes\n\nThe connector supports two methods for writing to BigQuery: direct writes and indirect writes.\n\n#### Direct write method\n\nThe direct write method consists of writing directly to BigQuery by using the [BigQuery Write API in \"pending\" mode](https://cloud.google.com/bigquery/docs/write-api-batch).\n\nThe indirect write method consists of the following steps:\n\n- During the execution of a write job, each mapper task creates its own BigQuery write stream and writes directly to\n  BigQuery in parallel.\n- At the end of the job, the connector commits all the streams together atomically. If the commit succeeds, then the\n  newly written data instantly becomes available for reading.\n\nIf for some reason the job fails, all the writes that may have been done through the various open streams are\nautomatically garbage-collected by the BigQuery backend and none of the data ends up persisting in the target table.\n\nThe direct write method is generally faster than the indirect write method but incurs [costs](https://cloud.google.com/bigquery/pricing#data_ingestion_pricing)\nassociated with usage of the BigQuery Storage Write API.\n\nThe connector uses this method by default.\n\n#### Indirect write method\n\nThe indirect write method consists of the following steps:\n\n- During the execution of a write job, each mapper task creates its own temporary output Avro file and stages it to GCS.\n- At the end of the job, the connector commits the writes by executing a [BigQuery load job](https://cloud.google.com/bigquery/docs/batch-loading-data)\n  with all the Avro files.\n- Once the job is complete, the connector deletes the temporary Avro files from GCS.\n\nThis method is generally costs less than the direct write method as [BigQuery load jobs are free](https://cloud.google.com/bigquery/pricing#data_ingestion_pricing)\nand this method only incur [costs related to GCS write operations and storage](https://cloud.google.com/storage/pricing).\nHowever, this method is also generally much slower due to its multi-stage nature and data being routed through GCS.\nLearn more about other [limitations](https://cloud.google.com/bigquery/docs/batch-loading-data#limitations).\n\nThe connector uses the direct write method by default. To let it use the indirect method instead,\nset the `bq.write.method` configuration property to `indirect`, and set the `bq.temp.gcs.path`\nproperty to indicate where to store the temporary Avro files in GCS.\n\n## Reading from BigQuery views and materialized views\n\nThe connector has preliminary support for reading from [BigQuery logical views](https://cloud.google.com/bigquery/docs/views-intro)\nand [BigQuery materialized views](https://cloud.google.com/bigquery/docs/materialized-views-intro).\n\nPlease note there are a few caveats:\n\n* The Storage Read API operates on storage directly, so the API cannot be used to read logical or materialized views. To\n  get around this limitation, the connector materializes the views into temporary tables before it can read them. This\n  materialization process can affect overall read performance and incur additional costs to your BigQuery bill.\n* By default, the materialized views are created in the same project and dataset. Those can be configured by the\n  optional `materializationProject` and `materializationDataset` Hive configuration properties or\n  table properties, respectively.\n* As mentioned in the [BigQuery documentation](https://cloud.google.com/bigquery/docs/writing-results#temporary_and_permanent_tables),\n  the `materializationDataset` should be in same location as the view.\n* Reading from views is **disabled** by default. In order to enable it, set the `viewsEnabled` configuration\n  property to `true`.\n\n## Reading from BigQuery table snapshots\n\nThe connector supports reading from [BigQuery table snapshots](https://cloud.google.com/bigquery/docs/table-snapshots-intro).\n\nA BigQuery table snapshot preserves the contents of a table (called the base table) at a particular\ntime. You can save a snapshot of a current table, or create a snapshot of a table as it was at any\ntime in the past seven days.\n\nTo link a Hive table to a BigQuery table snapshot, simply specify the snapshot's name in the\n`bq.table` table property, for example:\n\n```sql\nCREATE TABLE mytable (abc BIGINT, xyz STRING)\nSTORED BY 'com.google.cloud.hive.bigquery.connector.BigQueryStorageHandler'\nTBLPROPERTIES (\n    'bq.table'='myproject.mydataset.mysnapshot'\n);\n```\n\n## Read consistency\n\nAs stated in the [BigQuery Storage API documentation](https://cloud.google.com/bigquery/docs/reference/storage),\nread sessions are based on a snapshot isolation model (Note: this is unrelated to\n[BigQuery table snapshots](https://cloud.google.com/bigquery/docs/table-snapshots-intro)). All\nconsumers read based on a specific point in time. The snapshot time is based on the read\nsession creation time (i.e. when the `SELECT` query is initiated).\n\nNote that this consistency model currently only applies to the table data, not its metadata.\n\n## Spark SQL integration\n\nDataproc uses a patched version of Spark that automatically detects a table that has the `bq.table`\ntable property, in which case Spark will use the [`Spark-BigQuery Connector`](https://github.com/GoogleCloudDataproc/spark-bigquery-connector)\nto access the table's data. This means that on Dataproc you actually do not need to use the\nHive-BigQuery Connector for Spark SQL.\n\n### Code samples\n\nJava example:\n\n```java\nSparkConf sparkConf = new SparkConf().setMaster(\"local\");\nSparkSession spark =\n    SparkSession.builder()\n    .appName(\"example\")\n    .config(sparkConf)\n    .enableHiveSupport()\n    .getOrCreate();\nDataset\u003cRow\u003e ds = spark.sql(\"SELECT * FROM mytable\");\nRow[] rows = ds.collect();\n```\n\nPython example:\n\n```python\nspark = SparkSession.builder \\\n    .appName(\"example\") \\\n    .config(\"spark.master\", \"local\") \\\n    .enableHiveSupport() \\\n    .getOrCreate()\ndf = spark.sql(\"SELECT * FROM mytable\")\nrows = df.collect()\n```\n\n## Apache Pig integration\n\nThe connector supports Apache Pig via HCatalog.\n\nHere's an example reading from a BigQuery table and writing to another, assuming that\n`my-database.my-table` and `my-database.my-other-table` have been registered as BigQuery tables:\n\n```pig\nsome_data = LOAD 'my-database.my-table' USING org.apache.hive.hcatalog.pig.HCatLoader();\nSTORE some_data INTO 'my-database.my-other-table' USING org.apache.hive.hcatalog.pig.HCatStorer();\n```\n\nNotes:\n\n* Pig only supports `datetime` types with milliseconds precision, so you may encounter precision\n  loss if you have values with nanoseconds in Hive or BigQuery. Learn more in the HCatalog\n  documentation on [data type mappings](https://cwiki.apache.org/confluence/display/hive/hcatalog+loadstore#HCatalogLoadStore-DataTypeMappings).\n\n## BigLake integration\n\n[BigLake](https://cloud.google.com/biglake) allows you to store your data in open formats\n(e.g Parquet, ORC) in an object store like [GCS](https://cloud.google.com/storage) instead\nof in [BigQuery's native storage called Capacitor](https://cloud.google.com/blog/topics/developers-practitioners/bigquery-admin-reference-guide-storage),\nbut still leverage advanced BigQuery features like [metadata caching](https://cloud.google.com/bigquery/docs/biglake-intro#metadata_caching_for_performance)\nfor query performance, or [column-level access control](https://cloud.google.com/bigquery/docs/column-level-security-intro)\nand [dynamic data masking](https://cloud.google.com/bigquery/docs/column-data-masking-intro) for\nsecurity and governance at scale.\n\nTo integrate Hive with BigLake tables backed by GCS, you need to:\n\n1. Create a [Cloud resource connection](https://cloud.google.com/bigquery/docs/create-cloud-resource-connection).\n2. Upload your data files to a GCS bucket.\n3. Create an external table definition in BigQuery that points to the data file URIs in GCS.\n4. Create an external table definition in Hive that points to the BigQuery table definition.\n\n### Example: CSV table\n\nIn this example, we create simple table backed by some CSV files.\n\nThe files have the following layout in a GCS bucket:\n\n```text\ngs://mybucket/warehouse/my_csv_table/file1.csv\ngs://mybucket/warehouse/my_csv_table/file2.csv\n```\n\nExternal BigQuery table definition:\n\n```sql\nCREATE EXTERNAL TABLE `myproject.mydataset.my_csv_table`\nWITH CONNECTION `myproject.us.myconnection`\nOPTIONS (\n  format = 'CSV',\n  uris = ['gs://mybucket/warehouse/my_csv_table/*.csv']\n)\n```\n\nExternal Hive table definition:\n\n```sql\nCREATE EXTERNAL TABLE mydb.my_csv_table (\n    abc BIGINT,\n    xyz STRING\n)\nSTORED BY 'com.google.cloud.hive.bigquery.connector.BigQueryStorageHandler'\nTBLPROPERTIES (\n  'bq.table' = 'myproject.mydataset.my_csv_table'\n)\n```\n\nSample Hive query to read data:\n\n```sql\nSELECT abc\nFROM mydb.my_csv_table\nWHERE xyz = \"some text\"\n```\n\n### Example: Hive-partitioned Parquet table\n\nIn this example, we integrate a hive-partitioned Parquet table with BigLake.\n\nThe Parquet files have two columns:\n* `id`: 64-bit integer\n* `name`: string\n\nThe Parquet files are partitioned by `city` and have the following layout:\n\n```text\ngs://mybucket/warehouse/my_parquet_table/city=Paris/000000_0.parquet\ngs://mybucket/warehouse/my_parquet_table/city=Paris/000001_0.parquet\n\ngs://mybucket/warehouse/my_parquet_table/city=London/000000_0.parquet\ngs://mybucket/warehouse/my_parquet_table/city=London/000001_0.parquet\ngs://mybucket/warehouse/my_parquet_table/city=London/000002_0.parquet\n```\n\nExternal BigQuery table definition:\n\n```sql\nCREATE EXTERNAL TABLE `myproject.mydataset.my_parquet_table`\nWITH PARTITION COLUMNS\nWITH CONNECTION `myproject.us.myconnection`\nOPTIONS (\n  format = 'PARQUET',\n  uris = ['gs://mybucket/warehouse/my_parquet_table/*'],\n  hive_partition_uri_prefix = 'gs://mybucket/warehouse/my_parquet_table',\n  require_hive_partition_filter = true\n)\n```\n\nNotes about the above statement:\n* The `WITH PARTITION COLUMNS` clause exposes the `city` partition as a column in the BigQuery\n  table schema.\n* The `require_hive_partition_filter=true` option means that all queries over this table require a\n  partition filter that can be used to eliminate partitions when reading data.\n\nExternal Hive table definition that matches the BigQuery table's schema:\n\n```sql\nCREATE EXTERNAL TABLE mydb.my_parquet_table (\n    id BIGINT,\n    name STRING\n    city STRING\n)\nSTORED BY 'com.google.cloud.hive.bigquery.connector.BigQueryStorageHandler'\nTBLPROPERTIES (\n  'bq.table' = 'myproject.mydataset.my_parquet_table'\n)\n```\n\nSample Hive query to read data:\n\n```sql\nSELECT *\nFROM mydb.my_parquet_table\nWHERE city = \"Paris\"\n```\n\nFor more information, refer to the official BigQuery documentation on\n[Creating external tables on partitioned data](https://cloud.google.com/bigquery/docs/external-data-cloud-storage#create-external-table-partitioned)\nand [Creating Cloud Storage BigLake tables](https://cloud.google.com/bigquery/docs/create-cloud-storage-table-biglake).\n\n## Authentication\n\nThe connector needs an instance of a `GoogleCredentials` in order to connect to the BigQuery APIs.\n\nBy default, on Dataproc, the connector automatically uses the cluster service account's credentials.\n\nThere are multiple options to override the default behavior and to provide custom credentials:\n\n* Set the path to a service account's JSON private key in the `GOOGLE_APPLICATION_CREDENTIALS`\n  environment variable.\n* Set the path to a service account's JSON private key in the `bq.credentials.file` configuration\n  property.\n* Set a base64-encoded service account JSON private key in the `bq.credentials.key` configuration\n  property.\n* Set the fully qualified class name of a custom [access token provider]((https://github.com/GoogleCloudDataproc/spark-bigquery-connector/tree/master/bigquery-connector-common/src/main/java/com/google/cloud/bigquery/connector/common/AccessTokenProvider.java))\n  implementation in the `bq.access.token.provider.fqcn` configuration property. The class must be\n  implemented in Java or other JVM languages such as Scala or Kotlin. It must either have a no-arg\n  constructor or a constructor accepting a single `java.util.String` parameter. This parameter can\n  be supplied using the `bq.access.token.provider.config` configuration property. If the property is\n  not set then the no-arg constructor will be called. The JAR containing the implementation class\n  should be on the cluster's classpath.\n* Define [service account impersonation](https://cloud.google.com/iam/docs/service-account-permissions#directly-impersonate)\n  for specific users, specific groups, or for all users that run the Hive query by default using\n  the below properties:\n\n  - `bq.impersonation.service.account.for.user.\u003cUSER_NAME\u003e` (not set by default)\n\n    The service account to be impersonated for a specific user. You can specify multiple\n    properties using that pattern for multiple users.\n\n  - `bq.impersonation.service.account.for.group.\u003cGROUP_NAME\u003e` (not set by default)\n\n    The service account to be impersonated for a specific group. You can specify multiple\n    properties using that pattern for multiple groups.\n\n  - `bq.impersonation.service.account` (not set by default)\n\n    Default service account to be impersonated for all users.\n\n  If any of the above properties are set then the service account specified will be impersonated by\n  generating a short-lived credentials when accessing BigQuery.\n\n  If more than one property is set then the service account associated with the username will take\n  precedence over the service account associated with the group name for a matching user and group,\n  which in turn will take precedence over default service account impersonation.\n\n  The impersonator service account must have the `serviceAccountTokenCreator` IAM role set on the\n  impersonated service account.\n\n* For simpler applications where access token refresh is not required, pass the access token itself\n  with the `bq.access.token` configuration property. You can generate an access token by running\n  `gcloud auth application-default print-access-token`.\n\n## Customer-managed encryption key (CMEK)\n\nYou can provide a Cloud KMS key to be used to encrypt the destination table, for example when you\nrun a `CREATE TABLE` statement for a managed table, or when you insert data into a table that\ndoesn't exist yet. To do so, set the `bq.destination.table.kms.key.name` property with the\nfully-qualified named of the desired Cloud KMS key in the form:\n\n```\nprojects/\u003cKMS_PROJECT_ID\u003e/locations/\u003cLOCATION\u003e/keyRings/\u003cKEY_RING\u003e/cryptoKeys/\u003cKEY\u003e\n```\n\nThe BigQuery service account associated with your project requires access to this encryption key.\n\nThe table will be encrypted by the key only if it created by the connector. A pre-existing\nunencrypted table won't be encrypted just by setting this option.\n\nFor further information about using customer-managed encryption keys (CMEK) with BigQuery, see [here](https://cloud.google.com/bigquery/docs/customer-managed-encryption#key_resource_id).\n\n## Known issues and limitations\n\n* The `UPDATE`, `MERGE`, and `DELETE`, and `ALTER TABLE` statements are currently not supported.\n* The `PARTITIONED BY`, `CLUSTERED BY`, `INSERT INTO PARTITION`, and\n  `INSERT INTO PARTITION OVERWRITE` statements are currently not supported. Note, however, that\n  partitioning and clustering in BigQuery are supported via `TBLPROPERTIES`. See the corresponding\n  sections on [partitioning](#partitioning) and [clustering](#clustering).\n* CTAS (aka `CREATE TABLE AS SELECT`) and CTLT (`CREATE TABLE LIKE TABLE`) statements are currently\n  not supported.\n* If you use the Hive `MAP` type, then the map's key must be of `STRING` type if you use the Avro\n  format for reading or the indirect method for writing. This is because Avro requires keys to be\n  strings. If you use the Arrow format for reading (default) and the direct method for writing (also\n  default), then there are no type limitations for the keys.\n* Hive `DECIMAL` data type has precision of 38 and scale of 38, while BigQuery's `NUMERIC`/`BIGNUMERIC` have different precisions and scales.\n  (https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#decimal_types),\n  If the BigQuery data of `BIGNUMERIC`'s precision and scale out of Hive's `DECIMAL` range, data can show up as NULL.\n* [Custom Hive UDFs](https://cwiki.apache.org/confluence/display/hive/hiveplugins) (aka Hive plugins) are currently not supported.\n* BigQuery [ingestion time partitioning](https://cloud.google.com/bigquery/docs/partitioned-tables#ingestion_time) is currently supported only for read operations.\n* BigQuery [integer range partitioning](https://cloud.google.com/bigquery/docs/partitioned-tables#integer_range) is currently not supported.\n\n## Development\n\n### Code formatting\n\nTo standardize the code's format, use [Spotless](https://github.com/diffplug/spotless):\n\n```sh\n./mvnw spotless:apply\n```\n\n### Unit/integration tests\n\n#### Set up IAM permissions\n\nCreate a service account and give the following roles in your project:\n\n- BigQuery Admin\n- Storage Admin\n\nDownload a JSON private key for the service account, and set the `GOOGLE_APPLICATION_CREDENTIALS` environment\nvariable:\n\n```sh\nexport GOOGLE_APPLICATION_CREDENTIALS=\u003cpath/to/your/key.json\u003e\n```\n\n#### Enable APIs\n\nEnable the following APIs:\n\n```sh\ngcloud services enable \\\n  bigquerystorage.googleapis.com \\\n  bigqueryconnection.googleapis.com \\\n  cloudkms.googleapis.com\n```\n\n#### BigLake setup\n\nDefine environment variables:\n\n```sh\nexport PROJECT=\u003cmy-gcp-project\u003e\nexport BIGLAKE_LOCATION=us\nexport BIGLAKE_REGION=us-central1\nexport BIGLAKE_CONNECTION=hive-integration-tests\nexport BIGLAKE_BUCKET=${PROJECT}-biglake-tests\n```\n\nCreate the test BigLake connection:\n\n```sh\nbq mk \\\n  --connection \\\n  --project_id=\"${PROJECT}\" \\\n  --location=\"${BIGLAKE_LOCATION}\" \\\n  --connection_type=CLOUD_RESOURCE \\\n  \"${BIGLAKE_CONNECTION}\"\n```\n\nCreate the bucket to host BigLake datasets:\n\n```sh\ngsutil mb -l \"${BIGLAKE_REGION}\" \"gs://${BIGLAKE_BUCKET}\"\n```\n\nGive the BigLake connection's service account access to the bucket:\n\n```sh\nexport BIGLAKE_SA=$(bq show --connection --format json \"${PROJECT}.${BIGLAKE_LOCATION}.${BIGLAKE_CONNECTION}\" \\\n  | jq -r .cloudResource.serviceAccountId)\n\ngsutil iam ch serviceAccount:${BIGLAKE_SA}:objectViewer gs://${BIGLAKE_BUCKET}\n```\n\n#### KMS setup\n\nCreate a KMS keyring:\n\n```sh\ngcloud kms keyrings create \\\n  integration_tests_keyring \\\n  --location us\n```\n\n```sh\ngcloud kms keys create integration_tests_key \\\n    --keyring integration_tests_keyring \\\n    --location us \\\n    --purpose \"encryption\"\n```\n\nObtain the BigQuery service account name:\n\n```sh\nBQ_SERVICE_ACCOUNT=$(bq show --encryption_service_account --format json | jq -r \".ServiceAccountID\")\n```\n\nAssign the Encrypter/Decrypter role to the BigQuery service account:\n\n```sh\ngcloud kms keys add-iam-policy-binding \\\n  --project=${PROJECT} \\\n  --member serviceAccount:${BQ_SERVICE_ACCOUNT} \\\n  --role roles/cloudkms.cryptoKeyEncrypterDecrypter \\\n  --location=us \\\n  --keyring=integration_tests_keyring \\\n  integration_tests_key\n```\n\n#### Running the tests\n\nYou must use Java version 8, as it's the version that Hive itself uses. Make sure that `JAVA_HOME` points to the Java\n8's base directory.\n\n##### Integration tests\n\n* To run the integration tests:\n  ```sh\n  ./mvnw verify -Pdataproc22,integration\n  ```\n\n* To run a single integration test class:\n  ```sh\n  ./mvnw verify -Pdataproc22,integration -Dit.test=\"BigLakeIntegrationTests\"\n  ```\n\n* To run a specific integration test method:\n  ```sh\n  ./mvnw verify -Pdataproc22,integration -Dit.test=\"BigLakeIntegrationTests#testReadBigLakeTable\"\n  ```\n\n* To debug the tests, add the `-Dmaven.failsafe.debug` property:\n  ```sh\n  ./mvnw verify -Pdataproc22,integration -Dmaven.failsafe.debug\n  ```\n  ... then run a remote debugger in IntelliJ at port `5005`. Read more about debugging with FailSafe\n  here: https://maven.apache.org/surefire/maven-failsafe-plugin/examples/debugging.html\n\n##### Acceptance tests\n\nAcceptance tests create Dataproc clusters with the connector and run jobs to verify it.\n\nThe following environment variables must be set and **exported** first.\n\n* `GOOGLE_APPLICATION_CREDENTIALS` - the full path to a credentials JSON, either a service account or the result of a\n  `gcloud auth login` run\n* `GOOGLE_CLOUD_PROJECT` - The Google cloud platform project used to test the connector\n* `TEST_BUCKET` - The GCS bucked used to test writing to BigQuery during the integration tests\n* `ACCEPTANCE_TEST_BUCKET` - The GCS bucked used to test writing to BigQuery during the acceptance tests\n\nTo run the acceptance tests:\n\n```sh\n./mvnw verify -Pdataproc22,acceptance\n```\n\nIf you want to avoid rebuilding the `shaded-deps-dataproc22` and\n`shaded-acceptance-tests-dependencies` modules if they have no changes, you can break it down into\nthe following steps:\n\n```sh\n# Install hive-bigquery-parent/pom.xml to the Maven local repo\n./mvnw install:install-file -Dpackaging=pom -Dfile=hive-bigquery-parent/pom.xml -DpomFile=hive-bigquery-parent/pom.xml\n\n# Build and install the module JARs to the Maven local repo\n./mvnw clean install -pl shaded-deps-dataproc22,shaded-acceptance-tests-dependencies -Pdataproc22 -DskipTests\n```\n\nAt that point you can just run the tests without rebuilding the modules:\n\n```sh\n./mvnw clean verify -pl hive-bigquery-connector-common,hive-3-bigquery-connector -Pdataproc22,acceptance\n```\n\n##### Running the tests for different Hadoop versions\n\nTo run the tests for Hadoop 2, pass the `-Phadoop2` parameter to the `mvnw verify` command to\nactivate the `hadoop2` Maven profile. For Hadoop 3, pass `-Phadoop3` instead.\n\nBefore you can run the tests with Hadoop 3, you also must install Tez's latest (unreleased) 0.9.3:\n\n* Install Protobuf v2.5.0:\n\n  If you're on MacOS, install these packages:\n\n  ```sh\n  brew install automake libtool wget\n  ```\n\n  Then compile Protobuf from source:\n\n  ```sh\n  cd ~\n  wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.bz2\n  tar -xvjf protobuf-2.5.0.tar.bz2\n  rm protobuf-2.5.0.tar.bz2\n  cd protobuf-2.5.0\n  ./autogen.sh\n  ./configure --prefix=$(PWD)\n  make; make check\n  make install\n  ```\n\n* Get the Tez source:\n\n  ```sh\n  cd ~\n  git clone\n  cd git@github.com:apache/tez.git\n  cd tez\n  git checkout origin/branch-0.9\n  ```\n\n* Compile and install Tez:\n\n  ```sh\n  export PATH=${HOME}/protobuf-2.5.0/bin:${PATH}\n  mvn clean install \\\n    --projects=tez-api,tez-common,tez-mapreduce,tez-dag,hadoop-shim,tez-runtime-library,tez-runtime-internals \\\n    -DskipTests=true -Dmaven.javadoc.skip=true \\\n    -Dprotoc.path=${HOME}/protobuf-2.5.0/bin/protoc -Dhadoop.version=3.2.3\n  ```\n\n  If all steps have succeeded, then Tez's `0.9.2-SNAPSHOT` packages should be installed in your\n  local Maven repository and you should be able to run the tests with Hadoop 3 by using the\n  `-Phadoop3` argument.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogleclouddataproc%2Fhive-bigquery-connector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogleclouddataproc%2Fhive-bigquery-connector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogleclouddataproc%2Fhive-bigquery-connector/lists"}