{"id":51082986,"url":"https://github.com/kanutocd/pgoutput-client","last_synced_at":"2026-06-23T20:00:53.429Z","repository":{"id":361841489,"uuid":"1255042230","full_name":"kanutocd/pgoutput-client","owner":"kanutocd","description":"Transport-only PostgreSQL logical replication client for receiving pgoutput CopyData payloads.","archived":false,"fork":false,"pushed_at":"2026-06-01T14:40:39.000Z","size":57,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-13T08:20:11.665Z","etag":null,"topics":["cdc","change-data-capture","event-streaming","logical-decoding","logical-replication","pgoutput","postgresql","replication-client","replication-slots","ruby","streaming","wal"],"latest_commit_sha":null,"homepage":"https://kanutocd.github.io/pgoutput-client/","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kanutocd.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-31T10:27:18.000Z","updated_at":"2026-06-03T04:55:15.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/kanutocd/pgoutput-client","commit_stats":null,"previous_names":["kanutocd/pgoutput-client"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/kanutocd/pgoutput-client","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanutocd%2Fpgoutput-client","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanutocd%2Fpgoutput-client/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanutocd%2Fpgoutput-client/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanutocd%2Fpgoutput-client/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kanutocd","download_url":"https://codeload.github.com/kanutocd/pgoutput-client/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanutocd%2Fpgoutput-client/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34704748,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-23T02:00:07.161Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdc","change-data-capture","event-streaming","logical-decoding","logical-replication","pgoutput","postgresql","replication-client","replication-slots","ruby","streaming","wal"],"created_at":"2026-06-23T20:00:52.503Z","updated_at":"2026-06-23T20:00:53.411Z","avatar_url":"https://github.com/kanutocd.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pgoutput-client\n\n[![Gem Version](https://badge.fury.io/rb/pgoutput-client.svg)](https://badge.fury.io/rb/pgoutput-client)\n[![CI](https://github.com/kanutocd/pgoutput-client/workflows/CI/badge.svg)](https://github.com/kanutocd/pgoutput-client/actions)\n[![Ruby Version](https://img.shields.io/badge/ruby-%3E%3D%203.4-ruby.svg)](https://www.ruby-lang.org/en/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n\nA transport-only PostgreSQL logical replication client for receiving raw `pgoutput` payloads in Ruby.\n\n`pgoutput-client` connects to PostgreSQL using logical replication, starts a `pgoutput` replication stream, receives `CopyData` messages, handles keepalives, sends standby feedback, and yields raw pgoutput payload bytes to downstream gems such as `pgoutput-parser` and `pgoutput-decoder`.\n\nIt intentionally does **not** parse row-change messages or decode PostgreSQL values.\n\n---\n\n## Requirements\n\n- Ruby 3.4+\n- PostgreSQL 10+\n- `pg` gem\n- PostgreSQL publication and logical replication slot\n\n---\n\n## Ecosystem Position\n\n```text\nPostgreSQL logical replication\n        │\n        ▼\npgoutput-client\n        │\n        ▼\nCopyData / pgoutput payloads\n        │\n        ▼\npgoutput-parser\n        │\n        ▼\nProtocol messages\n        │\n        ▼\npgoutput-decoder\n        │\n        ▼\nDecoded row events\n```\n\n`pgoutput-client` is the transport layer only.\n\n---\n\n## Features\n\n- Opens PostgreSQL logical replication connections\n- Builds replication commands\n- Supports `CREATE_REPLICATION_SLOT`\n- Supports `DROP_REPLICATION_SLOT`\n- Supports `START_REPLICATION SLOT ... LOGICAL ...`\n- Parses XLogData envelopes\n- Parses primary keepalive messages\n- Builds standby feedback messages\n- Provides LSN parse/format helpers\n- Yields raw pgoutput payload bytes\n- Includes RBS signatures\n- Includes Minitest coverage\n- No audit, parser, or decoder concerns\n\n---\n\n## Installation\n\n```ruby\ngem \"pgoutput-client\"\n```\n\nThen:\n\n```bash\nbundle install\n```\n\nRequire:\n\n```ruby\nrequire \"pgoutput-client\"\n```\n\n---\n\n## Quick Start\n\n```ruby\nrequire \"pgoutput-client\"\n\nclient =\n  Pgoutput::Client::Runner.new(\n    database_url: ENV.fetch(\"DATABASE_URL\"),\n    slot_name: \"my_slot\",\n    publication_names: [\"my_publication\"],\n    auto_create_slot: true\n  )\n\nclient.start do |payload, metadata|\n  puts \"WAL end: #{metadata.wal_end_lsn}\"\n  puts \"Raw pgoutput payload bytes: #{payload.bytesize}\"\nend\n```\n\n---\n\n## Using With pgoutput-parser\n\n```ruby\nrequire \"pgoutput-client\"\nrequire \"pgoutput\"\n\nclient = Pgoutput::Client::Runner.new(\n  database_url: ENV.fetch(\"DATABASE_URL\"),\n  slot_name: \"my_slot\",\n  publication_names: [\"my_publication\"]\n)\n\ntracker = Pgoutput::RelationTracker.new\n\nclient.start do |payload, metadata|\n  message = tracker.process(payload)\n  p [metadata.wal_end_lsn, message]\nend\n```\n\n---\n\n## Using With pgoutput-decoder\n\n```ruby\nrequire \"pgoutput-client\"\nrequire \"pgoutput\"\nrequire \"pgoutput/decoder\"\n\ntracker = Pgoutput::RelationTracker.new\ndecoder = Pgoutput::Decoder.new\n\nclient.start do |payload, metadata|\n  protocol_message = tracker.process(payload)\n  event = decoder.decode(protocol_message)\n  p [metadata.wal_end_lsn, event]\nend\n```\n\n---\n\n## What This Gem Does\n\n```text\nPostgreSQL replication connection\n        │\n        ▼\nCopyData stream\n        │\n        ▼\nXLogData / Keepalive handling\n        │\n        ▼\nRaw pgoutput payloads\n```\n\nIt owns:\n\n- Replication connection setup\n- Replication command generation\n- CopyData reading\n- XLogData envelope parsing\n- Keepalive handling\n- Standby status feedback\n- LSN conversion\n\n---\n\n## What This Gem Does Not Do\n\nIt does not:\n\n- Parse pgoutput row messages\n- Decode PostgreSQL OIDs\n- Build application events\n- Group transactions\n- Run processor pipelines\n- Manage Ractor worker pools\n- Store audit records\n- Own replay, checkpointing, deduplication, or sink ordering\n\nThose responsibilities belong to higher layers, especially `cdc-core` and the sink that materializes downstream state.\n\n## Failure Semantics\n\nIf the live replication stream loses its connection, `pgoutput-client` retries a small number of times with a backoff and resumes from the latest confirmed WAL position.\n\nIt does not decide replay policy, deduplication strategy, checkpoint storage, or exactly-once delivery. Those concerns belong to the downstream CDC runtime and sink layer.\n\n---\n\n## Logical Replication Setup\n\nExample PostgreSQL setup:\n\n```sql\nALTER SYSTEM SET wal_level = logical;\n\nCREATE PUBLICATION my_publication FOR TABLE users, posts;\n```\n\nCreate a slot automatically:\n\n```ruby\nPgoutput::Client::Runner.new(\n  database_url: ENV.fetch(\"DATABASE_URL\"),\n  slot_name: \"my_slot\",\n  publication_names: [\"my_publication\"],\n  auto_create_slot: true\n)\n```\n\nOr create the slot yourself:\n\n```sql\nSELECT * FROM pg_create_logical_replication_slot('my_slot', 'pgoutput');\n```\n\n---\n\n## Public API\n\n### Pgoutput::Client::Runner\n\nHigh-level facade.\n\n```ruby\nclient = Pgoutput::Client::Runner.new(...)\nclient.start { |payload, metadata| ... }\n```\n\n### Pgoutput::Client::Configuration\n\nImmutable configuration object.\n\n### Pgoutput::Client::Connection\n\nThin wrapper around `PG::Connection` for replication commands.\n\n### Pgoutput::Client::Stream\n\nConsumes CopyData messages and yields pgoutput payloads.\n\n### Pgoutput::Client::LSN\n\n```ruby\nPgoutput::Client::LSN.parse(\"0/16B6C50\")\nPgoutput::Client::LSN.format(23_817_296)\n```\n\n### Pgoutput::Client::XLogData\n\nRepresents a WAL data envelope.\n\n### Pgoutput::Client::Keepalive\n\nRepresents a primary keepalive message.\n\n### Pgoutput::Client::Feedback\n\nBuilds standby status update payloads.\n\n---\n\n## Ractor Position\n\nThe replication connection itself is stateful and ordered. It should normally run as a single reader.\n\nDownstream parsing, decoding, and processing can be parallelized with Ractors:\n\n```text\npgoutput-client reader\n        │\n        ▼\nRactor-safe queue\n        │\n        ▼\nparser / decoder / processor pools\n```\n\n---\n\n## Rake Tasks\n\n### Default\n\nRun them all\n\n```bash\nbundle exec rake\n```\n\n### Code Linting and Formatting\n\n```bash\nbundle exec rake rubocop\n```\n\n### Testing\n\n```bash\nbundle exec rake test\n```\n\nWith coverage:\n\n```bash\nCOVERAGE=true bundle exec rake test\n```\n---\n\n### Type Checking\n\n```bash\nbundle exec rbs:validate\n```\n\n---\n\n### Documentation\n\n```bash\nbundle exec rake yard\n```\n\n### End-to-End PostgreSQL\n\nRun the full Docker-backed E2E flow and clean up afterward:\n\n```bash\nscript/test-e2e\n```\n\nKeep PostgreSQL running after the test for debugging:\n\n```bash\nKEEP_E2E_POSTGRES=1 script/test-e2e\n```\n\nYou can also run the steps manually:\n\n```bash\nscript/e2e-up\nPGOUTPUT_CLIENT_E2E=1 bundle exec rake test:e2e\nscript/e2e-down\n```\n\nEquivalent Rake task:\n\n```bash\nbundle exec rake e2e:run\n```\n\n## Transport lifecycle behavior\n\n`pgoutput-client` owns PostgreSQL logical replication transport and lifecycle\nmanagement. It opens the replication connection, optionally creates the logical\nreplication slot, starts streaming, sends standby status feedback, and retries\nreconnectable failures.\n\n### Idle standby feedback\n\nLong-running replication streams can be quiet for long periods when no WAL\nchanges are produced. During those idle periods the client wakes periodically\nand sends standby status feedback so PostgreSQL does not terminate the walsender\nfor replication timeout.\n\nControl the feedback cadence with `feedback_interval`:\n\n```ruby\nrunner = Pgoutput::Client::Runner.new(\n  database_url: ENV.fetch(\"DATABASE_URL\"),\n  slot_name: \"mammoth_live\",\n  publication_names: [\"mammoth_publication\"],\n  feedback_interval: 10.0\n)\n```\n\n### Idempotent automatic slot creation\n\nWhen `auto_create_slot` is enabled, the client treats slot creation as\n\"ensure this slot exists\". Missing slots are created before streaming; existing\nslots are reused and do not cause startup failure.\n\n```ruby\nrunner = Pgoutput::Client::Runner.new(\n  database_url: ENV.fetch(\"DATABASE_URL\"),\n  slot_name: \"mammoth_live\",\n  publication_names: [\"mammoth_publication\"],\n  auto_create_slot: true,\n  temporary_slot: false\n)\n```\n\nPublication creation remains outside this gem. Create publications through\napplication migrations, database bootstrap SQL, or infrastructure tooling.\n\n### Restart recovery\n\nAfter a stream has connected successfully, transient PostgreSQL outages are\nretried through the reconnect lifecycle. This includes ordinary container or\nprocess restart windows where PostgreSQL temporarily refuses connections or\nreports that the database system is starting up.\n\n---\n\n## License\n\n[MIT](LICENSE.txt).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkanutocd%2Fpgoutput-client","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkanutocd%2Fpgoutput-client","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkanutocd%2Fpgoutput-client/lists"}