{"id":18304033,"url":"https://github.com/googleclouddataproc/spark-bigtable-connector","last_synced_at":"2025-04-05T15:31:06.103Z","repository":{"id":236152862,"uuid":"792026562","full_name":"GoogleCloudDataproc/spark-bigtable-connector","owner":"GoogleCloudDataproc","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-12T16:51:28.000Z","size":706,"stargazers_count":3,"open_issues_count":13,"forks_count":9,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-21T06:33:26.231Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GoogleCloudDataproc.png","metadata":{"files":{"readme":"README-template.md","changelog":"CHANGES.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-25T20:42:00.000Z","updated_at":"2025-03-12T16:51:32.000Z","dependencies_parsed_at":"2024-04-30T00:29:00.474Z","dependency_job_id":"8215f6ff-c2ae-48b6-8ca4-bd31ada82b70","html_url":"https://github.com/GoogleCloudDataproc/spark-bigtable-connector","commit_stats":null,"previous_names":["googleclouddataproc/spark-bigtable-connector"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fspark-bigtable-connector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fspark-bigtable-connector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fspark-bigtable-connector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudDataproc%2Fspark-bigtable-connector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GoogleCloudDataproc","download_url":"https://codeload.github.com/GoogleCloudDataproc/spark-bigtable-connector/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247358523,"owners_count":20926235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T15:27:35.563Z","updated_at":"2025-04-05T15:31:06.090Z","avatar_url":"https://github.com/GoogleCloudDataproc.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache Spark SQL connector for Google Bigtable\n\nThis connector allows writing Spark SQL DataFrames\ninto [Google Bigtable](https://cloud.google.com/bigtable) and reading tables\nfrom Bigtable. It uses\nthe [Spark SQL Data Source API V1](https://spark.apache.org/docs/latest/sql-data-sources.html)\nto connect to Bigtable.\n\n## Unreleased Changes\n\nThis Readme may include documentation for changes that haven't been released yet.  The latest release's documentation and source code are found here.\n\nhttps://github.com/GoogleCloudDataproc/spark-bigtable-connector/blob/main/README.md\n\n## Quickstart\n\nYou can access the connector in two different ways:\n\n1. From\n   our [Maven Central repository](https://repo1.maven.org/maven2/com/google/cloud/spark/bigtable).\n2. Through a public GCS bucket, located\n   at `gs://spark-lib/bigtable/spark-bigtable_2.13-\u003cversion\u003e.jar` or `gs://spark-lib/bigtable/spark-bigtable_2.12-\u003cversion\u003e.jar`.\n\nIn Java and Scala applications, you can use different dependency management\ntools (e.g., Maven, sbt, or Gradle) to access the\nconnector `com.google.cloud.spark.bigtable:spark-bigtable_2.13:\u003cversion\u003e` or\n`com.google.cloud.spark.bigtable:spark-bigtable_2.12:\u003cversion\u003e` (current\n`\u003cversion\u003e` is `${next-release-tag}`) and package it inside your application JAR\nusing libraries such as Maven Shade Plugin. For PySpark applications, you can\nuse the `--jars` flag to pass the GCS address of the connector when submitting\nit.\n\nFor Maven, you can add the following snippet to your `pom.xml` file:\n\n```xml\n\u003c!-- If you are using scala 2.13 --\u003e\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.google.cloud.spark.bigtable\u003c/groupId\u003e\n  \u003cartifactId\u003espark-bigtable_2.13\u003c/artifactId\u003e\n  \u003cversion\u003e${next-release-tag}\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n```xml\n\u003c!-- If you are using scala 2.12 --\u003e\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.google.cloud.spark.bigtable\u003c/groupId\u003e\n  \u003cartifactId\u003espark-bigtable_2.12\u003c/artifactId\u003e\n  \u003cversion\u003e${next-release-tag}\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nFor sbt, you can add the following to your `build.sbt` file:\n\n```\n// for scala 2.13\nlibraryDependencies += \"com.google.cloud.spark.bigtable\" % \"spark-bigtable_2.13\" % \"${next-release-tag}\"\n```\n\n```\n// for scala 2.12\nlibraryDependencies += \"com.google.cloud.spark.bigtable\" % \"spark-bigtable_2.12\" % \"${next-release-tag}\"\n```\n\nFinally, you can add the following to your `build.gradle` file when using\nGradle:\n\n```\n// for scala 2.13\ndependencies {\nimplementation group: 'com.google.cloud.bigtable', name: 'spark-bigtable_2.13', version: '${next-release-tag}'\n}\n```\n\n```\n// for scala 2.12\ndependencies {\nimplementation group: 'com.google.cloud.bigtable', name: 'spark-bigtable_2.12', version: '${next-release-tag}'\n}\n```\n\nNote that you need plugins such\nas [Maven Shade Plugin](https://maven.apache.org/plugins/maven-shade-plugin/),\n[sbt-assembly](https://github.com/sbt/sbt-assembly), or\n[Shadow Plugin](https://imperceptiblethoughts.com/shadow/introduction/) to\npackage the connector inside\nyour JAR in Maven, sbt, and Gradle, respectively.\n\n## Getting started\n\n### About Bigtable\n\n[Bigtable](https://cloud.google.com/bigtable) is Google's NoSQL Big Data\ndatabase service.\nIt's the same service powering many of Google's internal applications, e.g.,\nSearch, Maps, etc. You can refer\nto [Bigtable documentations](https://cloud.google.com/bigtable/docs/instances-clusters-nodes)\nto learn more about key concepts, including instances, clusters, nodes, and\ntablets.\n\n### About Apache Spark and Spark SQL\n\nApache Spark is a distributed computing framework designed for fast and\nlarge-scale data processing,\nwhere [resilient distributed dataset (RDD)](https://spark.apache.org/docs/latest/rdd-programming-guide.html)\nis the main data model. Spark SQL is a module built on top of Spark that\nprovides a SQL-like interface for querying and manipulating data. This is done\nthrough [DataFrame and DataSet](https://spark.apache.org/docs/latest/sql-programming-guide.html),\nSpark SQL's data model, built on top of RDDs.\n\n### Supported Spark runtime environments and requirements\n\nYou can use the connector with Spark locally with the\n[Bigtable emulator](https://cloud.google.com/bigtable/docs/emulator),\nas well as in managed environments such as Dataproc cluster or serverless.\nYou need the following depending on the environments you choose to use:\n\n| Runtime environment                                                                 | Bigtable | Dataproc | Cloud Storage |\n|-------------------------------------------------------------------------------------|----------|----------|---------------|\n| Local Spark w/ [Bigtable emulator](https://cloud.google.com/bigtable/docs/emulator) | Optional | Optional | Optional      |\n| Local Spark                                                                         | Required | Optional | Optional      |\n| Dataproc Cluster                                                                    | Required | Required | Optional      |\n| Dataproc Serverless                                                                 | Required | Required | Required      |\n\n### Supported Spark versions\n\nThe connector supports the following Spark versions:\n\n| Scala version | Spark versions                    | Spark Application Languages                             |\n|---------------|-----------------------------------|---------------------------------------------------------|\n| 2.13          | 3.1.x, 3.2.x, 3.4.x, 3.5.x        | Java, Scala, PySpark (`.py` files or Jupyter notebooks) |\n| 2.12          | 2.4.8, 3.1.x, 3.2.x, 3.4.x, 3.5.x | Java, Scala, PySpark (`.py` files or Jupyter notebooks) |\n\n## Main features\n\nFor a detailed list of features and how to use them, you can refer the official\ndocumentation\n[here](https://cloud.google.com/bigtable/docs/use-bigtable-spark-connector).\nA list of main features is as follows:\n\n### Catalog definition\n\nYou can define a catalog as a JSON-formatted string, to convert from the\nDataFrame's schema to a\nformat compatible with Bigtable. This is an example of a catalog JSON:\n\n```\n{\n  \"table\": {\"name\": \"t1\"},\n  \"rowkey\": \"id_rowkey\",\n  \"columns\": {\n    \"id\": {\"cf\": \"rowkey\", \"col\": \"id_rowkey\", \"type\": \"string\"},\n    \"name\": {\"cf\": \"info\", \"col\": \"name\", \"type\": \"string\"},\n    \"birthYear\": {\"cf\": \"info\", \"col\": \"birth_year\", \"type\": \"long\"},\n    \"address\": {\"cf\": \"location\", \"col\": \"address\", \"type\": \"string\"}\n  }\n}\n```\n\nHere, the columns `name`, `birthYear`, and `address` from the DataFrame are\nconverted into Bigtable\ncolumns and the `id` column is used as the row key. Note that you could also\nspecify *compound* row keys,\nwhich are created by concatenating multiple DataFrame columns together.\n\n### Writing to Bigtable\n\nYou can use the `bigtable` format along with specifying the Bigtable\nproject and instance id to write to Bigtable. The catalog definition\nspecifies the table destination. This is a sample snippet of writing\nto Bigtable using Java:\n\n```java\nDataset\u003cRow\u003e dataFrame;\n// Adding some values to dataFrame.\ndataFrame\n  .write()\n  .format(\"bigtable\")\n  .option(\"catalog\", catalog)\n  .option(\"spark.bigtable.project.id\", projectId)\n  .option(\"spark.bigtable.instance.id\", instanceId);\n```\n\n### Reading from Bigtable\n\nYou can use the `bigtable` format and catalog, along with the Bigtable\nproject and instance id to read from Bigtable. This is a sample snippet\nof reading from Bigtable using Java:\n\n```java\nDataset\u003cRow\u003e dataFrame = spark\n    .read()\n    .format(\"bigtable\")\n    .option(\"catalog\", catalog)\n    .option(\"spark.bigtable.project.id\", projectId)\n    .option(\"spark.bigtable.instance.id\", instanceId)\n    .load();\n```\n\n### Runtime configurations\n\nYou can use `.option(\u003cconfig_name\u003e, \u003cconfig_value\u003e)` in Spark to pass different\nruntime configurations to\nthe connector. For example, Bigtable project and instance ID or settings for\ntimestamp and timeout configurations.\nFor a full list of configurations, refer to\n[BigtableSparkConf.scala](spark-bigtable_2.12/src/main/scala/com/google/cloud/spark/bigtable/datasources/BigtableSparkConf.scala),\nwhere these configs are defined.\n\n### Bigtable emulator support\n\nWhen using the connector locally, you can start a Bigtable emulator server and\nset the environment variable\n`export BIGTABLE_EMULATOR_HOST=localhost:\u003cemulator_port\u003e` in the same\nenvironment where Spark is launched. The connector will use the emulator\ninstead of a real Bigtable instance. You can refer to the\n[Bigtable emulator documentations](https://cloud.google.com/bigtable/docs/emulator)\nfor more details on using it.\n\n### Simple data type serialization\n\nThis connector supports encoding a number of Spark's simple data types as byte\narray for storage in Bigtable, with the following table summarizing the Spark\ntype names, the corresponding\n[GoogleSQL data types](https://cloud.google.com/spanner/docs/reference/standard-sql/data-types),\nthe names you need to use in the catalog string, and the encoding description:\n\n| Spark SQL data type | GoogleSQL type | Catalog type | Encoding description                                                                                                        |\n|---------------------|----------------|--------------|-----------------------------------------------------------------------------------------------------------------------------|\n| `BooleanType`       | `BOOL`         | `boolean`    | `True` -\u003e 1 byte corresponding to `-1` in two's-complement \u003cbr/\u003e `False` -\u003e 1 byte corresponding to `0` in two's-complement |\n| `ByteType`          | N/A            | `byte`       | 1-byte signed two's-complement number                                                                                       |\n| `ShortType`         | N/A            | `short`      | 2-byte signed two's-complement number, using big-endian                                                                     |\n| `IntegerType`       | N/A            | `int`        | 4-byte signed two's-complement number, using big-endian                                                                     |\n| `LongType`          | `INT64`        | `long`       | 8-byte signed two's-complement number, using big-endian                                                                     |\n| `FloatType`         | `FLOAT32`      | `float`      | 4-byte single-precision using IEEE 754 standard                                                                             |\n| `DoubleType`        | `FLOAT64`      | `double`     | 64-bit double-precision using IEEE 754 standard                                                                             |\n| `StringType`        | `STRING`       | `string`     | UTF-8 encoded string                                                                                                        |\n| `BinaryType`        | `BYTES`        | `binary`     | The corresponding array of bytes                                                                                            |\n\nYou can refer to\n[this link](https://spark.apache.org/docs/latest/sql-ref-datatypes.html)\nfor more information on Spark SQL types.\nIf you want to use a specific encoding schema for your DataFrame values, you\ncan manually convert these types to byte arrays and then store them on\nBigtable. The examples in the `examples` folder contain samples for converting\na column to `BinaryType` inside your application, in different languages.\n\n### Complex data type serialization using Apache Avro\n\nYou can specify an Avro schema for columns with a complex Spark SQL type such as\n`ArrayType`, `MapType`, or `StructType`,\nto serialize and store them in Bigtable.\n\n### Row key filter push down\n\nThis connector supports pushing down some of the filters on the row key column\nin the DataFrame to Bigtable\nand performing them on the server-side. The list of supported or non-supported\nfilters is as follows:\n\n**_NOTE:_**  The supported row key filters are only pushed to Bigtable when the\nvalue used in the filter has a type Long, String, or Byte array (e.g.,\n`rowKey \u003c \"some-value\"`). Support for other types (e.g., Integer, Float, etc.)\nwill be added in the future.\n\n| Filter               | Push down filter supported |\n|----------------------|----------------------------|\n| `EqualTo`            | Yes                        |\n| `LessThan`           | Yes                        |\n| `GreaterThan`        | Yes                        |\n| `LessThanOrEqual`    | Yes                        |\n| `GreaterThanOrEqual` | Yes                        |\n| `StringStartsWith`   | Yes                        |\n| `Or`                 | Yes                        |\n| `And`                | Yes                        |\n| `Not`                | No                         |\n| Compound Row Key     | No                         |\n\nWhen using compound row keys, filter on those columns are\n**not** pushed to Bigtable and are performed on the client-side (resulting in a\nfull-table scan). If filtering is required, a workaround is to concatenate\nthe intended columns into a *single* DataFrame column of a supported type\n(e.g., string) and use that column as the row key with one of the supported\nfilters above. One option is using the `concat` function, with a sample snippet\nin Scala as follows:\n\n```scala\ndf\n  .withColumn(\n    \"new_row_key\",\n    org.apache.spark.sql.functions.concat(\n      df.col(\"first_col\"),\n      df.col(\"second_col\")\n    )\n  )\n  .drop(\"first_col\")\n  .drop(\"second_col\")\n```\n\n### Client-side metrics\n\nSince the Bigtable Spark connector is based on the\n[Bigtable Client for Java](https://github.com/googleapis/java-bigtable),\nclient-side metrics are enabled\ninside the connector by default. You can refer to the\n[client-side metrics](https://cloud.google.com/bigtable/docs/client-side-metrics)\ndocumentation to find more details on accessing and interpreting these metrics.\n\n### Use with Data Boost (Preview)\n\nFor read-only jobs, you can use Data Boost (Preview) serverless compute, a new\ncompute option for Bigtable that is specially optimized for high-throughput\npipeline job performance and production app serving traffic isolation\nrequirements. To use Data Boost,\nyou must create a Data Boost app profile and then provide the app profile ID for\nthe `spark.bigtable.app_profile.id` Spark option when you add your Bigtable\nconfiguration to your Spark application. You can also convert an existing app\nprofile to a Data Boost app profile or specify a standard app profile to use\nyour instance's cluster nodes. You can refer to documentations on\n[Data Boost](https://cloud.google.com/bigtable/docs/data-boost-overview) and\n[App profiles](https://cloud.google.com/bigtable/docs/configuring-app-profiles)\nfor more information.\n\n### Use low-level RDD functions with Bigtable\n\nYou can use the Bigtable Spark connector to write and read low-level RDDs to\nand from Bigtable. The connector manages steps such as connection\ncreation and caching to simplify the usage, while giving you freedom for how\nexactly to convert between your RDD and Bigtable table (e.g., type conversion,\ncustom timestamps for each row, number of columns in each row, etc.). You can\ncreate a new `BigtableRDD` object and call these functions for read and write\noperations (note that this feature is only supported in Java and Scala,\nnot PySpark):\n\n#### Writing an RDD\nFor writing, you need to pass in an RDD of\n[RowMutationEntry](https://cloud.google.com/java/docs/reference/google-cloud-bigtable/latest/com.google.cloud.bigtable.data.v2.models.RowMutationEntry)\nobjects to the following function:\n\n```scala\nbigtableRDD.writeRDD(\n   rdd: RDD[RowMutationEntry],\n   tableId: String,\n   bigtableSparkConf: BigtableSparkConf\n)\n```\n\n#### Reading an RDD\nWhen reading an RDD, you receive an RDD of Bigtable\n[Row](https://cloud.google.com/java/docs/reference/google-cloud-bigtable/latest/com.google.cloud.bigtable.data.v2.models.Row)\nobjects after calling the following function:\n\n```scala\nBigtableRDD.readRDD(tableId: String, bigtableSparkConf: BigtableSparkConf)\n```\n\nNote that in both cases, you need to pass in a `BigtableSparkConf` object\ncorresponding to the options you want to set in your workflow. You can use the\n`BigtableSparkConfBuilder` class to create an instance of this class:\n\n```scala\nval bigtableSparkConf: BigtableSparkConf =\n   new BigtableSparkConfBuilder()\n     .setProjectId(someProjectId)\n     .setInstanceId(someInstanceId)\n     .build()\n```\n\nA list of the setter methods for the supported configs is as follows:\n\n1. `setProjectId(value: String)`\n2. `setInstanceId(value: String)`\n3. `setAppProfileId(value: String)`\n4. `setReadRowsAttemptTimeoutMs(value: String)`\n5. `setReadRowsTotalTimeoutMs(value: String)`\n6. `setMutateRowsAttemptTimeoutMs(value: String)`\n7. `setMutateRowsTotalTimeoutMs(value: String)`\n8. `setBatchMutateSize(value: Int)`\n9. `setEnableBatchMutateFlowControl(value: Boolean)`\n\nYou can refer to the\n[official documentation](https://cloud.google.com/bigtable/docs/use-bigtable-spark-connector)\nfor more details about each of these options.\n\n### Accessing Bigtable Java Client methods\n\nSince the Bigtable Spark connector is based on the\n[Bigtable client for Java](https://github.com/googleapis/java-bigtable),\nyou can directly use the client in your Spark applications, if you want\nto have even more control over how you interact with Bigtable.\n\nTo use the Bigtable client for Java classes, append the\n`com.google.cloud.spark.bigtable.repackaged` prefix to the package names. For\nexample, instead of using the class name\nas `com.google.cloud.bigtable.data.v2.BigtableDataClient`, use\n`com.google.cloud.spark.bigtable.repackaged.com.google.cloud.bigtable.data.v2.BigtableDataClient`.\n\n## Examples\n\nYou can access examples for Java, Scala, and Python inside the `examples`\ndirectory. Each directory contains a `README.md` file with instruction on\nrunning the example inside.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogleclouddataproc%2Fspark-bigtable-connector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogleclouddataproc%2Fspark-bigtable-connector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogleclouddataproc%2Fspark-bigtable-connector/lists"}