{"id":13936367,"url":"https://github.com/gabfl/bigquery_fdw","last_synced_at":"2025-05-07T18:09:23.531Z","repository":{"id":40571903,"uuid":"105712749","full_name":"gabfl/bigquery_fdw","owner":"gabfl","description":"BigQuery Foreign Data Wrapper for PostgreSQL","archived":false,"fork":false,"pushed_at":"2024-02-13T16:03:05.000Z","size":247,"stargazers_count":92,"open_issues_count":2,"forks_count":16,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-05-07T18:09:10.507Z","etag":null,"topics":["bigquery","fdw","postgresql","postgresql-extension"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gabfl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2017-10-03T23:07:54.000Z","updated_at":"2025-03-27T09:47:28.000Z","dependencies_parsed_at":"2024-02-13T16:29:30.644Z","dependency_job_id":"7f7ca1f6-a696-4504-870f-94b5facbfa5e","html_url":"https://github.com/gabfl/bigquery_fdw","commit_stats":{"total_commits":88,"total_committers":1,"mean_commits":88.0,"dds":0.0,"last_synced_commit":"ce4c3931f1763891f777ec06cac362fdfc5f2d16"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gabfl%2Fbigquery_fdw","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gabfl%2Fbigquery_fdw/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gabfl%2Fbigquery_fdw/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gabfl%2Fbigquery_fdw/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gabfl","download_url":"https://codeload.github.com/gabfl/bigquery_fdw/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252931555,"owners_count":21827112,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","fdw","postgresql","postgresql-extension"],"created_at":"2024-08-07T23:02:36.405Z","updated_at":"2025-05-07T18:09:23.460Z","avatar_url":"https://github.com/gabfl.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# bigquery_fdw: BigQuery Foreign Data Wrapper for PostgreSQL\n\n[![Pypi](https://img.shields.io/pypi/v/bigquery-fdw.svg)](https://pypi.org/project/bigquery-fdw/)\n[![Build Status](https://github.com/gabfl/bigquery_fdw/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/gabfl/bigquery_fdw/actions)\n[![codecov](https://codecov.io/gh/gabfl/bigquery_fdw/branch/main/graph/badge.svg)](https://codecov.io/gh/gabfl/bigquery_fdw)\n[![MIT licensed](https://img.shields.io/badge/license-MIT-green.svg)](https://raw.githubusercontent.com/gabfl/bigquery_fdw/main/LICENSE)\n\nbigquery_fdw is a BigQuery foreign data wrapper for PostgreSQL using [Multicorn](https://github.com/Segfault-Inc/Multicorn).\n\nIt allows to write queries in PostgreSQL SQL syntax using a foreign table. It supports most of BigQuery's [data types](docs/data_types.md) and [operators](docs/operators.md).\n\n## Features and limitations\n\n - Table partitioning is supported. [You can use partitions in your SQL queries](docs/table_partitioning.md).\n - Queries are parameterized when sent to BigQuery\n - BigQuery's standard SQL support (legacy SQL is not supported)\n - Authentication works with a \"[Service Account](docs/service_account.md)\" Json private key\n\n[Read more](docs/README.md).\n\n## Requirements\n\n - PostgreSQL \u003e= 9.5 up to 14\n - Python \u003e= 3.4\n\n## Get started\n\n### Using docker\n\nSee [getting started with Docker](docs/docker.md)\n\n### Installation on Debian/Ubuntu\n\n#### Dependencies required to install bigquery_fdw:\n\nYou need to install the following dependencies:\n\n```bash\n# Install required packages\napt update\napt install -y postgresql-server-dev-14 python3-setuptools python3-dev make gcc git\n```\n\nAll PostgreSQL versions from 9.2 to 14 should be supported.\n\n#### Installation\n\n```bash\n# Install Multicorn\n# pgsql-io/multicorn2 is a fork of Segfault-Inc/Multicorn that adds support for PostgreSQL 13/14.\n# Alternatively, up to PostgreSQL 12, you can use gabfl/Multicorn that adds better support for Python3.\n# You may also choose to build against the original project instead.\ngit clone https://github.com/pgsql-io/multicorn2.git Multicorn \u0026\u0026 cd Multicorn\nmake \u0026\u0026 make install\n\n# Install bigquery_fdw\npip3 install bigquery-fdw\n```\n\nMajor dependencies installed automatically during the installation process:\n\n - [Google Cloud BigQuery](https://pypi.org/project/google-cloud-bigquery/)\n - [Multicorn](https://github.com/pgsql-io/multicorn2)\n\n## Authentication\n\nbigquery_fdw relies on [Google Cloud API's default authentication](https://cloud.google.com/docs/authentication/getting-started#linux-or-macos). \n\nYour need to have an environment variable `GOOGLE_APPLICATION_CREDENTIALS` that has to be accessible by bigquery_fdw. Setting environment variables varies depending on OS but for Ubuntu or Debian, the preferred way is to edit `/etc/postgresql/[version]/main/environment` and add:\n```\nGOOGLE_APPLICATION_CREDENTIALS = '/path/to/key.json'\n```\n\nRestarting PostgreSQL is required for the environment variable to be loaded.\n\n## Usage\n\nWe recommend testing the [BigQuery client connectivity](docs/test_client.md) before trying to use the FDW.\n\nWith `psql`:\n\n```sql\nCREATE EXTENSION multicorn;\n\nCREATE SERVER bigquery_srv FOREIGN DATA WRAPPER multicorn\nOPTIONS (\n    wrapper 'bigquery_fdw.fdw.ConstantForeignDataWrapper'\n);\n\nCREATE FOREIGN TABLE my_bigquery_table (\n    column1 text,\n    column2 bigint\n) SERVER bigquery_srv\nOPTIONS (\n    fdw_dataset  'my_dataset',\n    fdw_table 'my_table'\n);\n```\n\n## Options\n\nList of options implemented in `CREATE FOREIGN TABLE` syntax:\n\n| Option | Default | Description |\n|-----|----|----|\n| `fdw_dataset` | - | BigQuery dataset name |\n| `fdw_table` | - | BigQuery table name |\n| `fdw_convert_tz` | - | Convert BigQuery time zone for dates and timestamps to selected time zone. Example: `'US/Eastern'`. |\n| `fdw_group` |  `'false'` | See [Remote grouping and counting](docs/remote_grouping.md). |\n| `fdw_casting` |  - | See [Casting](docs/casting.md). |\n| `fdw_verbose` | `'false'` | Set to `'true'` to output debug information in PostrgeSQL's logs |\n| `fdw_sql_dialect` | `'standard'` | BigQuery SQL dialect. Currently only `standard` is supported. |\n\n## More documentation\n\nSee [bigquery_fdw documentation](docs/README.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgabfl%2Fbigquery_fdw","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgabfl%2Fbigquery_fdw","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgabfl%2Fbigquery_fdw/lists"}