{"id":19718939,"url":"https://github.com/mtarnovan/ruby_pg_cdc","last_synced_at":"2025-05-07T21:47:03.203Z","repository":{"id":139552394,"uuid":"151433248","full_name":"mtarnovan/ruby_pg_cdc","owner":"mtarnovan","description":"An (experimental) Ruby replication client / CDC for Postgres","archived":false,"fork":false,"pushed_at":"2020-01-13T22:04:54.000Z","size":8,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-07T21:46:53.863Z","etag":null,"topics":["cdc","change-data-capture","logical-replication","postgres","psql","replication","ruby"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mtarnovan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-03T15:16:46.000Z","updated_at":"2024-10-16T09:18:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"8b8a26ac-098c-4195-b4a9-2d4bafa91a1d","html_url":"https://github.com/mtarnovan/ruby_pg_cdc","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtarnovan%2Fruby_pg_cdc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtarnovan%2Fruby_pg_cdc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtarnovan%2Fruby_pg_cdc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtarnovan%2Fruby_pg_cdc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mtarnovan","download_url":"https://codeload.github.com/mtarnovan/ruby_pg_cdc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252961858,"owners_count":21832192,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdc","change-data-capture","logical-replication","postgres","psql","replication","ruby"],"created_at":"2024-11-11T23:02:06.490Z","updated_at":"2025-05-07T21:47:03.178Z","avatar_url":"https://github.com/mtarnovan.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ruby Postgres CDC (Change data capture)\n\nAn experiment in implementing a [CDC](https://en.wikipedia.org/wiki/Change_data_capture)\nsystem using Postgres' [replication protocol](https://www.postgresql.org/docs/11.0/static/protocol-replication.html).\n\nYou need a [WAL decoder plugin](https://wiki.postgresql.org/wiki/Logical_Decoding_Plugins) installed on the server,\nthe defaults use [wal2json](https://github.com/eulerto/wal2json).\n\n## Usage\n\nThe `PgReplicationClient` class can be passed a config (see below for defaults) and a callback which will be called\nwith each received message of type XLogData. The message will be a `Decoder::XLogData` struct, see that module and the\nPostgres docs for details.\n\nThe default config is:\n\n```ruby\n  {\n    host: 'localhost',\n    port: 5432,\n    user: 'postgres',\n    password: nil,\n    dbname: 'postgres',\n    slotname: 'pg_logical_test', # replication slot name\n    status_interval: 10, # interval to sent status updates to server, in seconds\n    plugin: 'wal2json', # server-side WAL decoder plugin\n    plugin_opts: %q((\"include-types\" 'false', \"pretty-print\" 'true')),\n    create_slot: true # create slot on startup\n  }\n```\n\nRun `ruby test.rb` then make some queries (inserts, deletes etc.) on the specified database.\n\nExample:\n\nFrom a psql console:\n```text\n$psql postgres\npsql (10.5)\nType \"help\" for help.\npostgres=# \\d foo\n                                      Table \"public.foo\"\n Column |            Type             | Collation | Nullable |            Default\n--------+-----------------------------+-----------+----------+--------------------------------\n a      | integer                     |           | not null | nextval('foo_a_seq'::regclass)\n b      | character varying(30)       |           |          |\n c      | timestamp without time zone |           | not null |\nIndexes:\n    \"foo_pkey\" PRIMARY KEY, btree (a, c)\n\npostgres=# insert into foo(b, c) values('blabla', now());\nINSERT 0 1\npostgres=# delete from foo;\nDELETE 1\n```\n\n```text\n$ ruby test.rb\nI, [2018-10-03T18:09:49.774895 #75649]  INFO -- : host=localhost dbname=postgres port=5432 user=postgres slotname=pg_logical_test status_interval=10 (seconds)\nW, [2018-10-03T18:09:49.779846 #75649]  WARN -- : pg-logical: tried to create replication slot pg_logical_test, but it already exists\nReceived: {\n  \"change\": [\n    {\n      \"kind\": \"insert\",\n      \"schema\": \"public\",\n      \"table\": \"foo\",\n      \"columnnames\": [\n        \"a\",\n        \"b\",\n        \"c\"\n      ],\n      \"columnvalues\": [\n        1,\n        \"blabla\",\n        \"2018-10-03 18:09:58.859627\"\n      ]\n    }\n  ]\n}\nReceived: {\n  \"change\": [\n    {\n      \"kind\": \"delete\",\n      \"schema\": \"public\",\n      \"table\": \"foo\",\n      \"oldkeys\": {\n        \"keynames\": [\n          \"a\",\n          \"c\"\n        ],\n        \"keyvalues\": [\n          1,\n          \"2018-10-03 18:09:58.859627\"\n        ]\n      }\n    }\n  ]\n}\n\n```\n\n## How it works\n\nAfter connecting we create a replication slot (if `config.create_slot` is true), then execute `IDENTIFY_SYSTEM` to get\nthe current XLog position and start replication from that XLog position. After starting replication we add a\n`Concurrent::TimerTask` that will periodically send `Standby status update` packets to the server (every `config.status_interval`),\nso we keep the connection alive.\n\nThen we monitor the socket underlying the connection object and when we receive something we decode it and if it's a XLogData packet\nwe call the callback with it. If it's a `Keepalive` with the `requires_reply` we reply immediatly (from the main thread).\n\nTo track the current lsn, which is required when replying, we keep a `ReplicationState` object and update it as needed each time\nwe receive something from the server.\n\nSet the log level to `Logger::DEBUG` for a more verbose output, including keepalives and output from an observer of the Status update task\n(`StatusUpdateObserver`).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmtarnovan%2Fruby_pg_cdc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmtarnovan%2Fruby_pg_cdc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmtarnovan%2Fruby_pg_cdc/lists"}