{"id":20883120,"url":"https://github.com/trk54ylmz/spark-bigquery","last_synced_at":"2025-05-12T18:31:12.594Z","repository":{"id":77117515,"uuid":"127169141","full_name":"trK54Ylmz/spark-bigquery","owner":"trK54Ylmz","description":"Google BigQuery support for Spark SQL","archived":false,"fork":false,"pushed_at":"2018-04-02T23:08:13.000Z","size":6,"stargazers_count":5,"open_issues_count":0,"forks_count":6,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-01T09:11:09.271Z","etag":null,"topics":["bigquery","spark"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/trK54Ylmz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-28T16:34:56.000Z","updated_at":"2022-12-07T20:10:33.000Z","dependencies_parsed_at":"2023-03-12T00:05:39.970Z","dependency_job_id":null,"html_url":"https://github.com/trK54Ylmz/spark-bigquery","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trK54Ylmz%2Fspark-bigquery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trK54Ylmz%2Fspark-bigquery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trK54Ylmz%2Fspark-bigquery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trK54Ylmz%2Fspark-bigquery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/trK54Ylmz","download_url":"https://codeload.github.com/trK54Ylmz/spark-bigquery/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253797991,"owners_count":21965986,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","spark"],"created_at":"2024-11-18T07:34:59.625Z","updated_at":"2025-05-12T18:31:12.585Z","avatar_url":"https://github.com/trK54Ylmz.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark JDBC Big Query Connector\n\nGoogle BigQuery support for Spark SQL\n\n## Version\n\n| spark-bigquery | Spark | Scala |\n| :-----: | ----- | ----- |\n| 0.1 | 2.1 | 2.11 |\n\n## Usage\n\n### Scala 2.11\n\n```scala\nimport com.metglobal.oss.spark.jdbc._\n\n// Register BigQuery dialect\nJdbcDialects.registerDialect(BigQueryDialect)\n\nvar projectId = \"[PROJECT ID]\"\nvar oAuthType = \"[OAUTH TYPE, DEFAULT = 0]\"\nvar serviceAccount = \"[SERVICE ACCOUNT EMAIL FOR BIGQUERY]\"\nvar localOAuth = \"[LOCAL OAUTH FILE *.P12]\"\n\nval url = s\"jdbc:bigquery://https://www.googleapis.com/$projectId:443;ProjectId=$projectId;OAuthType=$oAuthType;OAuthServiceAcctEmail=$serviceAccount;OAuthPvtKeyPath=$localOAuth\"\n\nval df = spark.read\n    .format(\"jdbc\")\n    .option(\"driver\", \"com.simba.googlebigquery.jdbc42.Driver\") \\\n    .option(\"url\", url) \\\n    .option(\"dbtable\", \"(SELECT a, SUM(b) AS c, CAST(d AS STRING) FROM test.records GROUP BY a) AS table\") \\\n    .load()\n\n// Unregister dialect\nJdbcDialects.unregisterDialect(BigQueryDialect)\n```\n\n### Python\n\n```python\nsc = spark.sparkContext\n\nsc._jvm.com.metglobal.oss.spark.jdbc.BigQueryRegister.register()\n\ndf = spark.read \\\n    .format(\"jdbc\") \\\n    .option(\"driver\", \"com.simba.googlebigquery.jdbc42.Driver\") \\\n    .option(\"url\", \"jdbc:bigquery://https://www.googleapis.com/...\") \\\n    .option(\"dbtable\", \"(SELECT a, SUM(b) AS c, CAST(d AS STRING) FROM test.records GROUP BY a) AS table\") \\\n    .load()\n\nsc._jvm.com.metglobal.oss.spark.jdbc.BigQueryRegister.unregister()\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrk54ylmz%2Fspark-bigquery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrk54ylmz%2Fspark-bigquery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrk54ylmz%2Fspark-bigquery/lists"}