{"id":21970793,"url":"https://github.com/kingluo/pgcat","last_synced_at":"2025-04-09T06:11:14.084Z","repository":{"id":57602664,"uuid":"212550776","full_name":"kingluo/pgcat","owner":"kingluo","description":"Enhanced PostgreSQL logical replication","archived":false,"fork":false,"pushed_at":"2024-09-26T15:43:56.000Z","size":68,"stargazers_count":380,"open_issues_count":1,"forks_count":19,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-04-02T04:06:53.583Z","etag":null,"topics":["golang","logical-replication","postgresql"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kingluo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-03T10:16:24.000Z","updated_at":"2025-03-13T17:27:39.000Z","dependencies_parsed_at":"2025-01-03T04:10:54.152Z","dependency_job_id":"1ac5669e-1ba3-462e-8f52-ba99e69282c3","html_url":"https://github.com/kingluo/pgcat","commit_stats":{"total_commits":16,"total_committers":2,"mean_commits":8.0,"dds":0.0625,"last_synced_commit":"4e9c9c92d6525a54a6bf52df79ed75332ac43707"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingluo%2Fpgcat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingluo%2Fpgcat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingluo%2Fpgcat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingluo%2Fpgcat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kingluo","download_url":"https://codeload.github.com/kingluo/pgcat/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247987285,"owners_count":21028895,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","logical-replication","postgresql"],"created_at":"2024-11-29T14:43:09.172Z","updated_at":"2025-04-09T06:11:14.064Z","avatar_url":"https://github.com/kingluo.png","language":"Go","funding_links":[],"categories":["Extensions"],"sub_categories":[],"readme":"# pgcat - Enhanced postgresql logical replication\r\n\r\n* [Why pgcat?](#why-pgcat)\r\n* [Architecture](#architecture)\r\n* [Build from source](#build-from-source)\r\n* [Run](#run)\r\n* [Conflict handling](#conflict-handling)\r\n* [Table mapping](#table-mapping)\r\n* [Replication identity](#replication-identity)\r\n* [Limitations](#limitations)\r\n* [Conflict resolution](#conflict-resolution)\r\n\r\n---\r\n\r\nNote that some facts have changed in postgresql offical over the years since my last commit to pgcat:\r\n\r\n1. Since version 13:\r\n\r\n\u003e Allow partitioned tables to be logically replicated via publications (Amit Langote)\r\n\r\n2. Since version 16:\r\n\r\n\u003e Allow logical replication subscribers to process only changes that have no origin (Vignesh C, Amit Kapila)\r\n\u003e \r\n\u003e This can be used to avoid replication loops. This is controlled by the new CREATE SUBSCRIPTION ... ORIGIN option.\r\n\r\n3. Since version 17\r\n\r\n\u003e Add logical replication failover control to CREATE/ALTER SUBSCRIPTION (Shveta Malik, Hou Zhijie, Ajin Cherian)\r\n\r\nhttps://www.postgresql.org/docs/17/logical-replication-failover.html\r\n\r\n\u003e Allow partitioned tables to have identity columns (Ashutosh Bapat)\r\n\r\n**However, it doesn't cover all the features of pgcat, so pgcat is still meaningful.**\r\n\r\n---\r\n\r\n## Why pgcat?\r\n\r\nThe built-in logicial replication has below shortages:\r\n\r\n* only support base table as replication target (*also support paritioned table since postgresql 13*)\r\n* ~do not filter any origin, which will cause bi-directional dead loop~\r\n* could not do table name mapping\r\n* no conflict resolution\r\n\r\npgcat makes below enhancements:\r\n\r\n* supports any table type as replication target\r\ne.g. view, fdw, partitioned table, citus distributed table\r\n* only replicates local changes\r\nso that you could make bi-directional replication,\r\ne.g. replicates data between two datacenter\r\n* table name mapping\r\n* optional lww (last-writer-win) conflict resolution\r\n* save replication progress in table, so that it would be logged\r\nwhen subscriber failovers, it would retain the progress. ~In contrast,\r\nthe built-in logical replication of pg saves the progress in non-logged file.~\r\n\r\n## Architecture\r\n\r\npgcat is based on logical decoding, and reuses the publication part and pgoutput\r\noutput plugin of the pg built-in logical replication.\r\n\r\nInstead of worker processes and low-level row ingression, pgcat uses sql templates\r\nto apply high-level sql commands, so it could make largest compatibility on target\r\ntable type. It is written in golang and runs in separate process.\r\n\r\nThe lww uses one additional jsonb column to store meta info, e.g. timestamp.\r\nIt supports timestamp-override columns or cassandra-like counter columns.\r\n\r\n## Build from source\r\n\r\npgcat has two parts, pgcat binary and postgresql extension.\r\n\r\n### Dependencies\r\n\r\n* golang \u003e= 1.12\r\n* postgressql \u003e= 11\r\n\r\n### Build pgcat\r\n\r\n```\r\ncd cmd/pgcat\r\ngo build\r\ncd ../pgcat_setup_lww\r\ngo generate\r\ngo build\r\n```\r\n\r\n### Build pgcat-pgxs\r\n\r\nhttps://github.com/kingluo/pgcat-pgxs\r\n\r\n### Add pgcat user\r\n\r\n* Create pgcat user with replication attribute on the publisher database:\r\n\r\n```sql\r\n-- adjust your password here\r\nCREATE USER pgcat with REPLICATION PASSWORD 'pgcat';\r\n```\r\n\r\n* Create pgcat user on the subscriber database:\r\n\r\n```sql\r\n-- adjust your password here\r\nCREATE USER pgcat with PASSWORD 'pgcat';\r\n```\r\n\r\nNote: if you do bi-directional replication, create pgcat user with replication\r\nattribute on both publisher and subscriber databases.\r\n\r\n### Create extension\r\n\r\nUse superuser role to create extension on both the publisher\r\nand subscriber databases.\r\n\r\n```sql\r\nCREATE EXTENSION IF NOT EXISTS pgcat;\r\n```\r\n\r\nThis command would create extension under the pgcat schema.\r\n\r\n### Configure the publisher database\r\n\r\n* set logical wal level\r\n\r\npostgresql.conf\r\n\r\n```\r\nwal_level = logical\r\n```\r\n\r\n* Allow replication connection to the publisher database\r\n\r\npg_hba.conf\r\n\r\n```\r\nhost     all     pgcat     0.0.0.0/0     md5\r\n```\r\n\r\nRestart pg.\r\n\r\n```\r\npg_ctl restart\r\n```\r\n\r\n## Run\r\n\r\n### Setup the table\r\n\r\n#### Grant pgcat\r\n\r\nOn subscriber database, pgcat needs to read/write the table.\r\n\r\n```sql\r\ngrant select,insert,update,delete,truncate on foobar to pgcat;\r\n```\r\n\r\nIf you configure to copy the table in the subscription, then\r\non publisher database, pgcat needs to select the table.\r\n\r\n```sql\r\ngrant select on foobar to pgcat;\r\n```\r\n\r\n#### Setup publication\r\n\r\nOn publisher database:\r\n\r\n```sql\r\nCREATE PUBLICATION foobar FOR TABLE foobar;\r\nalter publication foobar add table foobar;\r\n```\r\n\r\n#### Setup lww (optional)\r\n\r\nIf you need last-writer-win conflict resolution, then run `pgcat_setup_lww` on all pg instances.\r\n\r\n```\r\npgcat_setup_lww -c lww.yml\r\n```\r\n\r\nCheck [lww.yml](https://github.com/kingluo/pgcat/blob/master/cmd/pgcat_setup_lww/lww.yml) for configuration file example.\r\n\r\n#### Setup subscription\r\n\r\nOn subscriber database:\r\n\r\n```sql\r\nINSERT INTO pgcat.pgcat_subscription(name, hostname, port, username, password,\r\ndbname, publications, copy_data, enabled) VALUES ('foobar', '127.0.0.1', 5433,\r\n'pgcat', 'pgcat', 'tmp', '{foobar}', true, true);\r\n```\r\n\r\nThe pgcat would check `pgcat_subscription`, if it changes, pgcat would apply the changes.\r\n\r\n### Run pgcat\r\n\r\n```bash\r\ncd cmd/pgcat\r\n# modify pgcat.yml to fit your need\r\npgcat -c pgcat.yml\r\n```\r\n\r\npgcat uses golang proxy dialer, so if you need to access your database via proxy, you could run below command:\r\n\r\n```bash\r\nall_proxy=socks5h://127.0.0.1:20000 pgcat -c pgcat.yml\r\n```\r\n\r\nIf you need to run pgcat in daemon on Linux, just use `setsid` command:\r\n\r\n```bash\r\nsetsid pgcat -c pgcat.yml \u0026\u003e/dev/null\r\n```\r\n\r\n**Admin HTTP API**:\r\n\r\nIf you configure `admin_listen_address`, e.g. `admin_listen_address: 127.0.0.1:30000`,\r\nthen you could run below commands to admin the pgcat process:\r\n\r\n```bash\r\n# rotate the log file\r\n# you could use logrotate to rotate pgcat log files as you need\r\ncurl http://127.0.0.1:30000/rotate\r\n\r\n# reload the yaml config file, e.g. you could add new databases to replicate\r\ncurl http://127.0.0.1:30000/reload\r\n```\r\n\r\n## Conflict handling\r\n\r\nIf not using lww, it's likely to have conflict, e.g. unique violation,\r\nespecially for bi-directional replication. When conflict happens, pgcat would\r\npanic this subscription, and restarts it in 1 minute, and so on.\r\n\r\nHow to handle conflict? As [pg doc](https://www.postgresql.org/docs/current/logical-replication-conflicts.html)\r\nsaid:\r\n\r\n\u003e either by changing data on the subscriber so that\r\n\u003e it does not conflict with the incoming change or by skipping\r\n\u003e the transaction that conflicts with the existing data.\r\n\r\nIt also works for pgcat. But since pgcat substitutes the subscriber part, so to\r\nskip transaction, you need to use below sql command:\r\n\r\n```sql\r\nupdate pgcat.pgcat_subscription_progress set lsn='0/27FD9B0';\r\n```\r\n\r\nThe lsn is the lsn of the commit record of this transaction, you could find it\r\nin pgcat log when conflict happens:\r\n\r\n```\r\ndml failed, commit_lsn=0/27FD9B0, err=...\r\n```\r\n\r\n## Table mapping\r\n\r\nYou could map publisher table name to different subscriber table name.\r\n\r\nYou could map multiple tables into one table, gathering multiple data source\r\ninto one target, e.g. partition tables, citus shards. Here the target could be\r\npartitioned table, view, or citus main table, so that you could have\r\nheterogeneous layout at different database and do easy replication.\r\n\r\nFor example, at database1, you have tables `foobar2` and `foobar3`, and you need\r\nto configure them be put into table `foobar2` at database2.\r\n\r\nIn database2 (subscriber database), run below sql command via superuser:\r\n\r\n```sql\r\ninsert into pgcat.pgcat_table_mapping(subscription,priority,src,dst)\r\n\tvalues('foobar',1,'^public.foobar[2-3]$','public.foobar2');\r\n```\r\n\r\nNote that the regexp and table name should be full qualified,\r\ni.e. with schema prefix. And the regexp is better to be surrounded with\r\n`^` and `$`, otherwise the matching is error prone.\r\n\r\n## Replication identity\r\n\r\nView and foreign table does not have replica ident, so it needs to configure\r\nthem in pgcat. The configuration table is `pgcat.pgcat_replident`.\r\n\r\nFor example, I need to set `id1` and `id2` columns as replica ident of view\r\n`foobar2_view`.\r\n\r\n\r\n```sql\r\ninsert into pgcat.pgcat_replident values('public.foobar2_view', '{\"id1\", \"id2\"}');\r\n```\r\n\r\nNote that the table name should be full qualified, i.e. with schema prefix.\r\n\r\n## Limitations\r\n\r\n* if the target is view, set `copy_data` to `false` in the subscription, and do\r\nnot use `instead of` trigger on view, please use view rule instead.\r\n`copy to` view requires `instead of` trigger attached, but the current pg\r\nhas no way to set `always` role on the view `instead of` trigger.\r\nOn the other hand, pgcat sets its role to `replica`, so `instead of`\r\ntrigger would not be called, and then `copy to` view would be no-op.\r\n\r\n## Conflict resolution\r\n\r\nConflict resolution is necessary for logical replication.\r\n\r\nLogical replication is normally used for loose-coupling different pg deployments\r\n(or different pg HA deployments, one HA deployment consists of one master and\r\nmultiple slaves, where they are connected via physical replication),\r\nespecially for different data centers, where it does not require\r\nreal-time data consistence).\r\n\r\nFor example, we have two groups of pg:\r\n\r\npg1 consists of pg1-master and pg1-slave, run in datacenter1.\r\n\r\npg2 consists of pg2-master, pg2-slave1 and pg2-slave2, run in datacenter2.\r\n\r\nThey replicate data changes to each other, but not in real time, and the\r\nnetwork between data centers may be broken for time to time.\r\n\r\nEach data center changes data independently, which would normally involves data\r\nwith same identity, e.g. same primary key. Such data needs some sort of policy\r\nto keep consistent in both data centers.\r\n\r\nFor example, pg1 writes \"foo\" to row foo, before this change is applied to pg2,\r\npg2 writes \"bar\" to row foo. Then which would be the final version of value?\r\n\r\n### lww (Last-writer-win)\r\n\r\nSimilar to Cassandra, it could use write timestamp to resolve the conflict.\r\nThe write with latest timestamp would be the last version of data value. The\r\npg deployment would keep time in sync, e.g. via NTP.\r\n\r\nThe `pgcat_setup_lww` command is used to setup the table for used by lww. Note\r\nthat conflict resolution is optional, so if you're sure the data changes are\r\nconsistent by nature or by design, you do not need to run this command on your\r\ntables.\r\n\r\nHow does it work?\r\n\r\nIt would category the columns as:\r\n* Inline columns\r\nInline columns have the same timestamp as the row.\r\n* Individual columns\r\nIndividual columns have their own timestamp associated.\r\n* Counter columns\r\nEach pg deployment would have their own individual copy of counter value. But\r\nwhen you read it, it would sum up all copys as the final value.\r\n\r\nWhen you run `pgcat_setup_lww`, it would:\r\n\r\n* Add a new column `__pgcat_lww` to the target table, which\r\nis in `jsonb` type, used to record the meta info of the row and columns.\r\n* for existing rows, it would populate `__pgcat_lww` concurrently (i.e. would\r\nnot block concurrent data r/w when the table setup is processing).\r\n* Create a trigger, which would\r\n\t* for local change, get the current timestamp as the row timestamp\r\n\t* for remote change, get the row timestamp in `__pgcat_lww`\r\n\t* for inline columns, compare the row timestamp\r\n\t* for each individual column, compare its own timestamp\r\n\t* for counter column, store the remote change in `__pgcat_lww`, use\r\n\tsystem identifier from `pg_control_system()` to identify different\r\n\tremote peers\r\n* Create a view to filter the `__pgcat_lww` so that it would not be exported\r\nto application level. For example, if you have table `foobar`,\r\nthen it would create a view `foobar_pgcat_lww_view`.\r\n* Create a helper function `pgcat_lww_counter()` used to sum up counter column.\r\nYou could call `pgcat_lww_counter(__pgcat_lww, 'foobar')` to\r\nget the value of column `foobar`.\r\n* When you delete a row, it would not be really deleted, instead, it would be\r\nmarked as tombstone in `__pgcat_lww`, so `pgcat_setup_lww` would create index\r\nfor tombstone rows and create a helper function to vacuum them whenever you need,\r\ne.g. remove tombstones older than 3 days:\r\n`foobar_pgcat_lww_vaccum(interval '3 days') `.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkingluo%2Fpgcat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkingluo%2Fpgcat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkingluo%2Fpgcat/lists"}