{"id":51124810,"url":"https://github.com/nullstone-modules/gcp-datastream-postgres-bigquery","last_synced_at":"2026-06-25T06:30:41.826Z","repository":{"id":366918287,"uuid":"1278182918","full_name":"nullstone-modules/gcp-datastream-postgres-bigquery","owner":"nullstone-modules","description":"Creates a Datastream connection to replicate data from a postgres instance to a BigQuery dataset","archived":false,"fork":false,"pushed_at":"2026-06-23T20:54:31.000Z","size":14,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-23T22:27:03.679Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nullstone-modules.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-23T14:41:17.000Z","updated_at":"2026-06-23T20:54:35.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/nullstone-modules/gcp-datastream-postgres-bigquery","commit_stats":null,"previous_names":["nullstone-modules/gcp-datastream-postgres-bigquery"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/nullstone-modules/gcp-datastream-postgres-bigquery","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nullstone-modules%2Fgcp-datastream-postgres-bigquery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nullstone-modules%2Fgcp-datastream-postgres-bigquery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nullstone-modules%2Fgcp-datastream-postgres-bigquery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nullstone-modules%2Fgcp-datastream-postgres-bigquery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nullstone-modules","download_url":"https://codeload.github.com/nullstone-modules/gcp-datastream-postgres-bigquery/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nullstone-modules%2Fgcp-datastream-postgres-bigquery/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34763481,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-25T02:00:05.521Z","response_time":101,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-25T06:30:41.030Z","updated_at":"2026-06-25T06:30:41.817Z","avatar_url":"https://github.com/nullstone-modules.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# gcp-datastream-postgres-bigquery\n\nCreates a [Google Cloud Datastream](https://cloud.google.com/datastream) pipeline that\ncontinuously replicates change data from a PostgreSQL database into a BigQuery dataset.\n\nThis module wires together three Nullstone datastores:\n\n- a **Postgres** source (`datastore/gcp/postgres`),\n- a **BigQuery** destination (`datastore/gcp/bigquery`), and\n- the **VPC network** (`network/gcp/vpc`) the source database lives in,\n\nand provisions everything Datastream needs to move data between them: a private\nconnection, source/destination connection profiles, and the stream itself.\n\n## How it connects\n\nDatastream runs in a Google-managed project, so it reaches your database over\n[Private Service Connect (PSC)](https://docs.cloud.google.com/datastream/docs/psc-interfaces)\nrather than directly. This module creates a **network attachment** in the database's\nprivate subnet and a Datastream **private connection** bound to it. The source\ndatabase must therefore be reachable over private networking — when using the\n`gcp-cloudsql-postgres` module, provision it with `enable_psc = true`.\n\nThe Postgres connection profile connects to the database endpoint exposed by the\n`postgres` connection (`db_endpoint`), authenticating with `var.postgres_username`,\n`var.postgres_password`, and `var.postgres_database`. Reference a Nullstone secret for\nthe password using the `{{ secret(...) }}` interpolation so it is never stored in\nplaintext config:\n\n```yaml\nvars:\n  postgres_password: \"{{ secret(POSTGRES_PASSWORD) }}\"\n```\n\n## Prerequisites on the source database\n\nPostgres logical replication must be configured on the source **before** you launch\nthis module — Datastream cannot create the stream otherwise. Complete the steps below\nin order.\n\n### 1. Enable logical decoding\n\nLogical replication requires `wal_level \u003e= logical`. On Cloud SQL for Postgres this is\ncontrolled by the `cloudsql.logical_decoding` database flag — set it to `on` (e.g. with\nthe `gcp-cloudsql-postgres` module's flags input, or in the Cloud Console). Changing this\nflag **requires a database restart**, so apply it first and let the instance come back up.\n\nYou can confirm it is active with:\n\n```sql\nSHOW wal_level;  -- should report: logical\n```\n\n### 2. Create the publication and replication slot\n\nOnce `wal_level` is `logical`, create the publication and logical replication slot that\nDatastream reads from. Their names are passed to this module as\n`var.replication_publication` and `var.replication_slot`:\n\n```sql\n-- Publication listing the tables to replicate\nCREATE PUBLICATION nullstone_pub1 FOR ALL TABLES;\n\n-- Logical replication slot using the pgoutput plugin\nSELECT pg_create_logical_replication_slot('datastream_slot1', 'pgoutput');\n```\n\n### 3. Create the replication user\n\nCreate the user Datastream authenticates as (`var.postgres_username` /\n`var.postgres_password`). It needs `REPLICATION` privileges and `SELECT` on the tables\nbeing replicated:\n\n```sql\nCREATE USER datastream WITH REPLICATION LOGIN PASSWORD '...';\nGRANT SELECT ON ALL TABLES IN SCHEMA public TO datastream;\nALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO datastream;\n```\n\nThe `ALTER DEFAULT PRIVILEGES` statement ensures the user also gains `SELECT` on tables\ncreated in the future, not just those that exist today. Repeat the two `GRANT`/`ALTER`\nstatements for any other schemas listed in `var.replication_objects`.\n\nPass this password to the module using `{{ secret(...) }}` interpolation so it is never\nstored in plaintext config (see [How it connects](#how-it-connects) above).\n\n## Selecting what to replicate\n\n`var.replication_objects` controls which schemas and tables are streamed:\n\n- The default (`[{ schema = \"public\" }]`) replicates **every table in the `public` schema**.\n- Set it to empty (`[]`) to replicate **every table in every schema**.\n- List a schema with no `tables` to replicate **all tables in that schema**.\n- List specific `tables` under a schema to replicate **only those tables**.\n\n```hcl\nreplication_objects = [\n  { schema = \"public\" },\n  { schema = \"billing\", tables = [\"invoices\", \"payments\"] },\n]\n```\n\n## Destination layout\n\nThe stream writes to the dataset from the `bigquery` connection using Datastream's\n**merge** mode (`single_target_dataset`). Each source table becomes a BigQuery\ntable that is kept as a current-state replica — inserts, updates, and soft-deletes\nare merged in, so no \"latest row\" logic is needed when querying. `var.data_freshness`\ncontrols how often those merges run (and therefore how fresh the replica is).\n\nThe stream is created with `backfill_all`, so existing rows are backfilled once\nbefore ongoing change data is applied. By default it starts in the `RUNNING`\nstate and replicates as soon as it is created.\n\n## Pausing replication\n\nSet `var.enabled = false` to pause the stream (`desired_state = \"PAUSED\"`) without\ndestroying it or losing its configuration. Replication stops, but the stream,\nconnection profiles, and private connection are all left in place. Set it back to\n`true` to resume.\n\n## Outputs\n\n| Output                              | Description                                                                    |\n|-------------------------------------|--------------------------------------------------------------------------------|\n| `stream_id`                         | The id of the Datastream stream.                                               |\n| `stream_name`                       | The fully-qualified resource name of the Datastream stream.                    |\n| `stream_state`                      | The current state of the Datastream stream (e.g. `RUNNING`).                   |\n| `private_connection_id`             | The id of the Datastream private connection.                                   |\n| `source_connection_profile_id`      | The id of the Postgres source connection profile.                              |\n| `destination_connection_profile_id` | The id of the BigQuery destination connection profile.                         |\n| `network_attachment_id`             | The id of the network attachment Datastream uses to reach the source over PSC. |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnullstone-modules%2Fgcp-datastream-postgres-bigquery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnullstone-modules%2Fgcp-datastream-postgres-bigquery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnullstone-modules%2Fgcp-datastream-postgres-bigquery/lists"}