{"id":36399842,"url":"https://github.com/vnvo/pgwire-replication","last_synced_at":"2026-01-11T16:02:50.882Z","repository":{"id":331266989,"uuid":"1125513448","full_name":"vnvo/pgwire-replication","owner":"vnvo","description":"Lean tokio-based Postgres wire-protocol logical replication client (pgoutput) with TLS and SCRAM.","archived":false,"fork":false,"pushed_at":"2026-01-03T21:00:15.000Z","size":143,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-05T05:59:04.389Z","etag":null,"topics":["cdc","pgwire","postgresql","replication"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vnvo.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-30T21:36:33.000Z","updated_at":"2026-01-03T20:47:11.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/vnvo/pgwire-replication","commit_stats":null,"previous_names":["vnvo/pgwire-replication"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/vnvo/pgwire-replication","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vnvo%2Fpgwire-replication","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vnvo%2Fpgwire-replication/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vnvo%2Fpgwire-replication/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vnvo%2Fpgwire-replication/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vnvo","download_url":"https://codeload.github.com/vnvo/pgwire-replication/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vnvo%2Fpgwire-replication/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28312188,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-11T14:58:17.114Z","status":"ssl_error","status_checked_at":"2026-01-11T14:55:53.580Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdc","pgwire","postgresql","replication"],"created_at":"2026-01-11T16:02:50.812Z","updated_at":"2026-01-11T16:02:50.867Z","avatar_url":"https://github.com/vnvo.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pgwire-replication\n\n[![CI](https://github.com/vnvo/pgwire-replication/actions/workflows/ci.yml/badge.svg)](https://github.com/vnvo/pgwire-replication/actions/workflows/ci.yml)\n[![Crates.io](https://img.shields.io/crates/v/pgwire-replication.svg)](https://crates.io/crates/pgwire-replication)\n[![docs.rs](https://docs.rs/pgwire-replication/badge.svg)](https://docs.rs/pgwire-replication)\n[![License](https://img.shields.io/badge/license-MIT%2FApache--2.0-blue.svg)](https://github.com/vnvo/pgwire-replication#license)\n[![MSRV](https://img.shields.io/badge/MSRV-1.88-blue.svg)](Cargo.toml)\n\nA low-level, high-performance PostgreSQL logical replication client implemented directly on top of the PostgreSQL wire protocol (pgwire).\n\nThis crate is designed for **CDC, change streaming, and WAL replay systems** that require explicit control over replication state, deterministic restart behavior, and minimal runtime overhead.\n\n`pgwire-replication` intentionally avoids `libpq`, `tokio-postgres`, and other higher-level PostgreSQL clients for the replication path. It interacts with a Postgres instance directly and relies on `START_REPLICATION ... LOGICAL ...` and the built-in `pgoutput` output plugin.\n\n`pgwire-replication` exists to provide:\n\n- a **direct pgwire implementation** for logical replication\n- explicit, user-controlled LSN start and stop semantics\n- predictable feedback and backpressure behavior\n- clean integration into async systems and coordinators\n\nThis crate was originally extracted from the Deltaforge CDC project and is maintained independently.\n\n## Installation\n\nAdd to your `Cargo.toml`:\n\n```toml\n[dependencies]\npgwire-replication = \"0.1\"\n```\n\nOr with specific features:\n\n```toml\n[dependencies]\npgwire-replication = { version = \"0.1\", default-features = false, features = [\"tls-rustls\"] }\n```\n\n## Requirements\n\n- Rust 1.88 or later\n- PostgreSQL 15+ with logical replication enabled (older versions will probably work too)\n\n## Features\n\n- Logical replication using the PostgreSQL wire protocol\n- `pgoutput` logical decoding support (transport-level)\n- Explicit LSN seek (`start_lsn`)\n- Bounded replay (`stop_at_lsn`)\n- Periodic standby status updates\n- Keepalive handling\n- Tokio-based async client\n- SCRAM-SHA-256 and MD5 authentication\n- TLS/mTLS support (via rustls)\n- Designed for checkpoint and replay-based systems\n\n## Non-goals\n\nThis crate intentionally does **not** provide:\n\n- A general-purpose SQL client\n- Automatic checkpoint persistence\n- Exactly-once semantics\n- Schema management or DDL interpretation\n- Full `pgoutput` decoding into rows or events\n\nThese responsibilities belong in higher layers.\n\n## Basic usage\n\n```rust\nuse pgwire_replication::{ReplicationClient, ReplicationEvent};\n\nlet mut repl = ReplicationClient::connect(config).await?;\n\nwhile let Some(event) = repl.recv().await? {\n    match event {\n        ReplicationEvent::XLogData { wal_end, data, .. } =\u003e {\n            process(data);\n            repl.update_applied_lsn(wal_end);\n        }\n        ReplicationEvent::KeepAlive { .. } =\u003e {}\n        ReplicationEvent::StoppedAt { reached } =\u003e break,\n    }\n}\n// Clean end-of-stream\n```\n\nCheck the **Quick Start** and **Examples** for more detailed use cases.\n\n## Seek and Replay Semantics\n\n`pgwire-replication` is built around explicit WAL position control.\nLSNs (Log Sequence Numbers) are treated as first-class inputs and outputs and are never hidden behind opaque offsets.\n\n### Starting from an LSN (Seek)\n\nEvery replication session begins at an explicit LSN:\n\n```rust\nReplicationConfig {\n    start_lsn: Lsn,\n    ..\n}\n```\n\nThis enables:\n\n- resuming replication after a crash\n- replaying WAL from a known checkpoint\n- controlled historical backfills\n\nThe provided LSN is sent verbatim to PostgreSQL via `START_REPLICATION`.\n\n### Bounded Replay (Start -\u003e Stop)\n\nReplication can be bounded using `stop_at_lsn`:\n\n```rust\nReplicationConfig {\n    start_lsn,\n    stop_at_lsn: Some(stop_lsn),\n    ..\n}\n```\n\nWhen configured:\n\n- replication starts at start_lsn\n- WAL is streamed until the stop LSN is reached\n- a ReplicationEvent::StoppedAt { reached } event is emitted\n- After StoppedAt is emitted, the stream ends cleanly and recv() returns Ok(None)\n- the replication connection is terminated cleanly using CopyDone\n\nThis enables:\n\n- deterministic WAL replay\n- offline backfills\n- \"replay up to checkpoint\" workflows\n- controlled reprocessing in recovery scenarios\n\n## Progress Tracking and Feedback\n\nProgress is **not** auto-committed.\nInstead, the consumer explicitly reports progress:\n\n```rust\nrepl.update_applied_lsn(lsn);\n```\nCalling `update_applied_lsn` indicates that **all WAL up to `lsn` has been durably persisted by the consumer** (for example, flushed to disk or a message queue).\n\nThis allows callers to control:\n\n- durability boundaries\n- batching behavior\n- exactly-once or at-least-once semantics (implemented externally)\n\nUpdates are **monotonic**: reporting an older LSN is a no-op.\nStandby status updates are sent asynchronously by the worker using the\nlatest applied LSN, based on `status_interval` or server keepalive requests.\nFor CDC pipelines, progress should typically be reported at **transaction commit boundaries**, not for every message.\n\n## Idle behavior\n\nPostgreSQL logical replication may remain silent for extended periods when no WAL is generated.\nThis is normal.\n\n`idle_wakeup_interval` does not indicate failure. It bounds how long the client may block\nwaiting for server messages before waking up to send a standby status update and continue waiting.\nWhile the system is idle, the effective feedback cadence is bounded by\n`idle_wakeup_interval`, not `status_interval`.\n\n## Shutdown\n\n- `stop()` requests a graceful stop (sends `CopyDone`). The client will continue to yield any buffered events and then `recv()` returns `Ok(None)`.\n- `shutdown().await` is a convenience method that calls `stop()`, drains remaining events, and awaits the worker task result.\n- `abort()` cancels the worker task immediately (hard stop; does not send `CopyDone`).\n\nDropping `ReplicationClient` requests a best-effort graceful stop. When dropped inside a Tokio runtime, the worker is detached and allowed to finish cleanly; when dropped outside a runtime, the worker may be aborted to avoid leaking a task.\n\n## Important Notes on LSN Semantics\n\n- PostgreSQL does not guarantee that every logical replication message advances the WAL end position.\n- Small or fast transactions may share the same WAL position.\n- LSNs should be treated as monotonic but not dense.\n\nToday, bounded replay is evaluated using WAL positions observed during streaming.\nFuture versions may expose commit-boundary LSNs derived from pgoutput decoding for stronger replay guarantees.\nLSNs are formatted exactly as PostgreSQL displays them:\n\n- uppercase hexadecimal\n- `X/Y` format\n- up to 8 hex digits per part\n- leading zeros omitted\n\nExamples: `0/0`, `0/16B6C50`, `16/B374D848`\nParsing accepts both padded and unpadded forms for compatibility.\n\n## TLS support\n\nTLS is optional and uses `rustls`.\nTLS configuration is provided explicitly via `ReplicationConfig` and does not rely on system OpenSSL.\n\n## Quick start\n\nControl plane (publication/slot creation) is typically done using a proper \"Postgres client\" (Your choice).\nThis crate handles only the replication plane.\n\n```rust\nuse pgwire_replication::{\n    client::ReplicationEvent, Lsn, ReplicationClient, ReplicationConfig, SslMode, TlsConfig,\n};\n\n#[tokio::main]\nasync fn main() -\u003e anyhow::Result\u003c()\u003e {\n    // Control plane (publication/slot creation) is typically done using a Postgres client.\n    // This crate implements the replication plane only.\n\n    // Use a real LSN:\n    // - from your checkpoint store, or\n    // - from SQL (pg_current_wal_lsn / slot confirmed_flush_lsn), or\n    // - from a previous run.\n    let start_lsn = Lsn::parse(\"0/16B6C50\")?;\n\n    let cfg = ReplicationConfig {\n        host: \"127.0.0.1\".into(),\n        port: 5432,\n        user: \"postgres\".into(),\n        password: \"postgres\".into(),\n        database: \"postgres\".into(),\n        tls: TlsConfig::disabled(),\n        slot: \"my_slot\".into(),\n        publication: \"my_pub\".into(),\n        start_lsn,\n        stop_at_lsn: None,\n\n        status_interval: std::time::Duration::from_secs(10),\n        idle_wakeup_interval: std::time::Duration::from_secs(10),\n        buffer_events: 8192,\n    };\n\n    let mut client = ReplicationClient::connect(cfg).await?;\n\n    loop {\n        match client.recv().await {\n            Ok(Some(ev)) =\u003e match ev {\n                ReplicationEvent::XLogData { wal_end, data, .. } =\u003e {\n                    println!(\"XLogData wal_end={wal_end} bytes={}\", data.len());\n                    client.update_applied_lsn(wal_end);\n                }\n                ReplicationEvent::KeepAlive {\n                    wal_end,\n                    reply_requested,\n                    ..\n                } =\u003e {\n                    println!(\"KeepAlive wal_end={wal_end} reply_requested={reply_requested}\");\n                }\n                ReplicationEvent::StoppedAt { reached } =\u003e {\n                    println!(\"StoppedAt reached={reached}\");\n                    // break is optional; the stream should end shortly anyway\n                    break;\n                }\n            },\n            Ok(None) =\u003e {\n                println!(\"Replication ended cleanly\");\n                break;\n            }\n            Err(e) =\u003e {\n                eprintln!(\"Replication failed: {e}\");\n                return Err(e.into());\n            }\n        }\n    }\n    Ok(())\n}\n```\n\n## Examples\n\nExamples that use the control-plane SQL client (`tokio-postgres`) require the `examples` feature.\n\n### Replication plane only: `examples/basic.rs`\n\n```bash\nSTART_LSN=\"0/16B6C50\" cargo run --example basic\n```\n\n### Control-plane + streaming: `examples/checkpointed.rs`\n\n```bash\ncargo run --example checkpointed\n```\n\n### Bounded replay: `examples/bounded_replay.rs`\n\n```bash\ncargo run --example bounded_replay\n```\n\n### With TLS enabled: `examples/with_tls.rs`\n\n```bash\nPGHOST=db.example.com \\\nPGPORT=5432 \\\nPGUSER=repl_user \\\nPGPASSWORD=secret \\\nPGDATABASE=postgres \\\nPGSLOT=example_slot_tls \\\nPGPUBLICATION=example_pub_tls \\\nPGTLS_CA=/path/to/ca.pem \\\nPGTLS_SNI=db.example.com \\\ncargo run --example with_tls\n```\n\n### Enabling mTLS : `examples/with_mtls.rs`\n\nInject the fake dns record, if you need to:\n\n```bash\nsudo sh -c 'echo \"127.0.0.1 db.example.com\" \u003e\u003e /etc/hosts'\n```\n\nand then:\n\n```bash\nPGHOST=db.example.com \\\nPGPORT=5432 \\\nPGUSER=repl_user \\\nPGPASSWORD=secret \\\nPGDATABASE=postgres \\\nPGSLOT=example_slot_mtls \\\nPGPUBLICATION=example_pub_mtls \\\nPGTLS_CA=/etc/ssl/ca.pem \\\nPGTLS_CLIENT_CERT=/etc/ssl/client.crt.pem \\\nPGTLS_CLIENT_KEY=/etc/ssl/client.key.pem \\\nPGTLS_SNI=db.example.com \\\ncargo run --example with_mtls\n```\n\n- `PGUSER/PGPASSWORD` are used for control-plane setup (publication/slot).\n- `REPL_USER/REPL_PASSWORD` are used for the replication stream.\n- If PGHOST is an **IP address**, you must set `PGTLS_SNI` to a DNS name on the cert.\n- Client key should be **PKCS#8 PEM** for best compatibility.\n- `VerifyCa` can be used instead of `VerifyFull` if hostname validation is not possible.\n\n# Testing\n\nIntegration tests use Docker via `testcontainers` and are gated behind a feature flag:\n\n```bash\ncargo test --features integration-tests -- --nocapture\n```\n\n# License\n\nLicensed under either of:\n\n- Apache License, Version 2.0\n- MIT License\n\nat your option.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvnvo%2Fpgwire-replication","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvnvo%2Fpgwire-replication","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvnvo%2Fpgwire-replication/lists"}