{"id":25176705,"url":"https://github.com/conduitio/conduit-connector-postgres","last_synced_at":"2025-05-06T23:21:17.250Z","repository":{"id":37897334,"uuid":"467164901","full_name":"ConduitIO/conduit-connector-postgres","owner":"ConduitIO","description":"Conduit connector for PostgreSQL","archived":false,"fork":false,"pushed_at":"2025-05-01T13:19:17.000Z","size":709,"stargazers_count":13,"open_issues_count":22,"forks_count":10,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-05-01T13:31:13.310Z","etag":null,"topics":["conduit","go","golang","postgresql"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ConduitIO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-03-07T16:06:24.000Z","updated_at":"2025-04-30T14:31:59.000Z","dependencies_parsed_at":"2024-05-20T12:30:31.732Z","dependency_job_id":"1e7166ac-23fc-46a6-98ea-9824a15ece81","html_url":"https://github.com/ConduitIO/conduit-connector-postgres","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-connector-postgres","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-connector-postgres/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-connector-postgres/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-connector-postgres/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ConduitIO","download_url":"https://codeload.github.com/ConduitIO/conduit-connector-postgres/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252783880,"owners_count":21803571,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["conduit","go","golang","postgresql"],"created_at":"2025-02-09T13:17:56.642Z","updated_at":"2025-05-06T23:21:17.244Z","avatar_url":"https://github.com/ConduitIO.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Conduit Connector PostgreSQL\n\nThe PostgreSQL connector is a [Conduit](https://github.com/ConduitIO/conduit)\nplugin. It provides both, a source and a destination PostgresSQL connector.\n\n\u003c!-- readmegen:description --\u003e\n## Source\n\nThe Postgres Source Connector connects to a database with the provided `url` and\nstarts creating records for each change detected in the provided tables.\n\nUpon starting, the source takes a snapshot of the provided tables in the database,\nthen switches into CDC mode. In CDC mode, the plugin reads from a buffer of CDC events.\n\n### Snapshot\n\nWhen the connector first starts, snapshot mode is enabled. The connector acquires\na read-only lock on the tables, and then reads all rows of the tables into Conduit.\nOnce all rows in that initial snapshot are read the connector releases its lock and\nswitches into CDC mode.\n\nThis behavior is enabled by default, but can be turned off by adding\n`\"snapshotMode\": \"never\"` to the Source configuration.\n\n### Change Data Capture\n\nThis connector implements Change Data Capture (CDC) features for PostgreSQL by\ncreating a logical replication slot and a publication that listens to changes in the\nconfigured tables. Every detected change is converted into a record. If there are no\nrecords available, the connector blocks until a record is available or the connector\nreceives a stop signal.\n\n#### Logical Replication Configuration\n\nWhen the connector switches to CDC mode, it attempts to run the initial setup commands\nto create its logical replication slot and publication. It will connect to an existing\nslot if one with the configured name exists.\n\nThe Postgres user specified in the connection URL must have sufficient privileges to\nrun all of these setup commands, or it will fail.\n\nExample pipeline configuration that's using logical replication:\n\n```yaml\nversion: 2.2\npipelines:\n  - id: pg-to-log\n    status: running\n    connectors:\n      - id: pg\n        type: source\n        plugin: builtin:postgres\n        settings:\n          url: \"postgres://exampleuser:examplepass@localhost:5433/exampledb?sslmode=disable\"\n          tables: \"users\"\n          cdcMode: \"logrepl\"\n          logrepl.publicationName: \"examplepub\"\n          logrepl.slotName: \"exampleslot\"\n      - id: log\n        type: destination\n        plugin: builtin:log\n        settings:\n          level: info\n```\n\n:warning: When the connector or pipeline is deleted, the connector will automatically\nattempt to delete the replication slot and publication. This is the default behaviour\nand can be disabled by setting `logrepl.autoCleanup` to `false`.\n\n### Key Handling\n\nThe connector will automatically look up the primary key column for the specified tables\nand use them as the key value. If that can't be determined, the connector will return\nan error.\n\n## Destination\n\nThe Postgres Destination takes a Conduit record and stores it using a SQL statement.\nThe Destination is designed to handle different payloads and keys. Because of this,\neach record is individually parsed and upserted.\n\n### Handling record operations\n\nBased on the `Operation` field in the record, the destination will either insert,\nupdate or delete the record in the target table. Snapshot records are always inserted.\n\nIf the target table already contains a record with the same key as a record being\ninserted, the record will be updated (upserted). This can overwrite and thus potentially\nlose data, so keys should be assigned correctly from the Source.\n\nIf the target table does not contain a record with the same key as a record being\ndeleted, the record will be ignored.\n\nIf there is no key, the record will be simply appended.\n\u003c!-- /readmegen:description --\u003e\n\n## Source Configuration Parameters\n\n\u003c!-- readmegen:source.parameters.yaml --\u003e\n```yaml\nversion: 2.2\npipelines:\n  - id: example\n    status: running\n    connectors:\n      - id: example\n        plugin: \"postgres\"\n        settings:\n          # Tables is a List of table names to read from, separated by a comma,\n          # e.g.:\"table1,table2\". Use \"*\" if you'd like to listen to all tables.\n          # Type: string\n          # Required: yes\n          tables: \"\"\n          # URL is the connection string for the Postgres database.\n          # Type: string\n          # Required: yes\n          url: \"\"\n          # CDCMode determines how the connector should listen to changes.\n          # Type: string\n          # Required: no\n          cdcMode: \"auto\"\n          # LogreplAutoCleanup determines if the replication slot and\n          # publication should be removed when the connector is deleted.\n          # Type: bool\n          # Required: no\n          logrepl.autoCleanup: \"true\"\n          # LogreplPublicationName determines the publication name in case the\n          # connector uses logical replication to listen to changes (see\n          # CDCMode).\n          # Type: string\n          # Required: no\n          logrepl.publicationName: \"conduitpub\"\n          # LogreplSlotName determines the replication slot name in case the\n          # connector uses logical replication to listen to changes (see\n          # CDCMode). Can only contain lower-case letters, numbers, and the\n          # underscore character.\n          # Type: string\n          # Required: no\n          logrepl.slotName: \"conduitslot\"\n          # WithAvroSchema determines whether the connector should attach an\n          # avro schema on each record.\n          # Type: bool\n          # Required: no\n          logrepl.withAvroSchema: \"true\"\n          # Snapshot fetcher size determines the number of rows to retrieve at a\n          # time.\n          # Type: int\n          # Required: no\n          snapshot.fetchSize: \"50000\"\n          # SnapshotMode is whether the plugin will take a snapshot of the\n          # entire table before starting cdc mode.\n          # Type: string\n          # Required: no\n          snapshotMode: \"initial\"\n          # Maximum delay before an incomplete batch is read from the source.\n          # Type: duration\n          # Required: no\n          sdk.batch.delay: \"0\"\n          # Maximum size of batch before it gets read from the source.\n          # Type: int\n          # Required: no\n          sdk.batch.size: \"0\"\n          # Specifies whether to use a schema context name. If set to false, no\n          # schema context name will be used, and schemas will be saved with the\n          # subject name specified in the connector (not safe because of name\n          # conflicts).\n          # Type: bool\n          # Required: no\n          sdk.schema.context.enabled: \"true\"\n          # Schema context name to be used. Used as a prefix for all schema\n          # subject names. If empty, defaults to the connector ID.\n          # Type: string\n          # Required: no\n          sdk.schema.context.name: \"\"\n          # Whether to extract and encode the record key with a schema.\n          # Type: bool\n          # Required: no\n          sdk.schema.extract.key.enabled: \"false\"\n          # The subject of the key schema. If the record metadata contains the\n          # field \"opencdc.collection\" it is prepended to the subject name and\n          # separated with a dot.\n          # Type: string\n          # Required: no\n          sdk.schema.extract.key.subject: \"key\"\n          # Whether to extract and encode the record payload with a schema.\n          # Type: bool\n          # Required: no\n          sdk.schema.extract.payload.enabled: \"false\"\n          # The subject of the payload schema. If the record metadata contains\n          # the field \"opencdc.collection\" it is prepended to the subject name\n          # and separated with a dot.\n          # Type: string\n          # Required: no\n          sdk.schema.extract.payload.subject: \"payload\"\n          # The type of the payload schema.\n          # Type: string\n          # Required: no\n          sdk.schema.extract.type: \"avro\"\n```\n\u003c!-- /readmegen:source.parameters.yaml --\u003e\n\n## Destination Configuration Parameters\n\n\u003c!-- readmegen:destination.parameters.yaml --\u003e\n```yaml\nversion: 2.2\npipelines:\n  - id: example\n    status: running\n    connectors:\n      - id: example\n        plugin: \"postgres\"\n        settings:\n          # URL is the connection string for the Postgres database.\n          # Type: string\n          # Required: yes\n          url: \"\"\n          # Key represents the column name for the key used to identify and\n          # update existing rows.\n          # Type: string\n          # Required: no\n          key: \"\"\n          # Table is used as the target table into which records are inserted.\n          # Type: string\n          # Required: no\n          table: \"{{ index .Metadata \"opencdc.collection\" }}\"\n          # Maximum delay before an incomplete batch is written to the\n          # destination.\n          # Type: duration\n          # Required: no\n          sdk.batch.delay: \"0\"\n          # Maximum size of batch before it gets written to the destination.\n          # Type: int\n          # Required: no\n          sdk.batch.size: \"0\"\n          # Allow bursts of at most X records (0 or less means that bursts are\n          # not limited). Only takes effect if a rate limit per second is set.\n          # Note that if `sdk.batch.size` is bigger than `sdk.rate.burst`, the\n          # effective batch size will be equal to `sdk.rate.burst`.\n          # Type: int\n          # Required: no\n          sdk.rate.burst: \"0\"\n          # Maximum number of records written per second (0 means no rate\n          # limit).\n          # Type: float\n          # Required: no\n          sdk.rate.perSecond: \"0\"\n          # The format of the output record. See the Conduit documentation for a\n          # full list of supported formats\n          # (https://conduit.io/docs/using/connectors/configuration-parameters/output-format).\n          # Type: string\n          # Required: no\n          sdk.record.format: \"opencdc/json\"\n          # Options to configure the chosen output record format. Options are\n          # normally key=value pairs separated with comma (e.g.\n          # opt1=val2,opt2=val2), except for the `template` record format, where\n          # options are a Go template.\n          # Type: string\n          # Required: no\n          sdk.record.format.options: \"\"\n          # Whether to extract and decode the record key with a schema.\n          # Type: bool\n          # Required: no\n          sdk.schema.extract.key.enabled: \"true\"\n          # Whether to extract and decode the record payload with a schema.\n          # Type: bool\n          # Required: no\n          sdk.schema.extract.payload.enabled: \"true\"\n```\n\u003c!-- /readmegen:destination.parameters.yaml --\u003e\n\n## Testing\n\nRun `make test` to run all the unit and integration tests, which require Docker\nto be installed and running. The command will handle starting and stopping\ndocker containers for you.\n\n## References\n\n- https://github.com/bitnami/bitnami-docker-postgresql-repmgr\n- https://github.com/Masterminds/squirrel\n\n![scarf pixel](https://static.scarf.sh/a.png?x-pxid=1423de19-24e7-4d64-91cf-0b893ca28cc6)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconduitio%2Fconduit-connector-postgres","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconduitio%2Fconduit-connector-postgres","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconduitio%2Fconduit-connector-postgres/lists"}