{"id":51082983,"url":"https://github.com/kanutocd/pgoutput-parser","last_synced_at":"2026-06-23T20:00:53.506Z","repository":{"id":361551855,"uuid":"1254868066","full_name":"kanutocd/pgoutput-parser","owner":"kanutocd","description":"A pure Ruby parser for PostgreSQL pgoutput logical replication CopyData payloads.","archived":false,"fork":false,"pushed_at":"2026-06-06T09:35:04.000Z","size":95,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-13T08:20:11.572Z","etag":null,"topics":["binary-parser","cdc","change-data-capture","logical-decoding","logical-replication","pgoutput","postgres","postgresql","postgresql-replication","protocol-parser","ractor","ractor-safe","replication","ruby","stream-processing","wal"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kanutocd.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-31T05:10:32.000Z","updated_at":"2026-06-06T09:31:39.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/kanutocd/pgoutput-parser","commit_stats":null,"previous_names":["kanutocd/pgoutput-parser"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/kanutocd/pgoutput-parser","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanutocd%2Fpgoutput-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanutocd%2Fpgoutput-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanutocd%2Fpgoutput-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanutocd%2Fpgoutput-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kanutocd","download_url":"https://codeload.github.com/kanutocd/pgoutput-parser/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanutocd%2Fpgoutput-parser/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34704748,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-23T02:00:07.161Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary-parser","cdc","change-data-capture","logical-decoding","logical-replication","pgoutput","postgres","postgresql","postgresql-replication","protocol-parser","ractor","ractor-safe","replication","ruby","stream-processing","wal"],"created_at":"2026-06-23T20:00:52.410Z","updated_at":"2026-06-23T20:00:53.500Z","avatar_url":"https://github.com/kanutocd.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pgoutput-parser\n\nA high-performance, Ractor-safe PostgreSQL `pgoutput` logical replication protocol parser written in pure Ruby.\n\n`pgoutput-parser` parses PostgreSQL logical replication `CopyData` payloads into immutable protocol message objects. It focuses on the `pgoutput` wire format: transaction boundaries, relation metadata, DML message structure, tuple payload markers, and raw tuple bytes.\n\nIt intentionally does **not** convert PostgreSQL values into application-specific Ruby objects. That belongs to a higher-level decoder layer, such as a future `pgoutput-decoder` gem.\n\n---\n\n## Requirements\n\n- Ruby 4+\n- PostgreSQL 10+\n\n---\n\n## Features\n\n- Pure Ruby implementation\n- Ruby 4+\n- Ractor-safe parsed messages\n- Immutable protocol message objects\n- PostgreSQL logical replication protocol support\n- Relation metadata tracking\n- Binary-safe tuple parsing\n- RBS type signatures included\n- YARD documentation included\n- No runtime dependencies\n\nThe generated documentation also includes a project glossary:\n[docs/glossary.md](docs/glossary.md).\n\n---\n\n## Why Another pgoutput Library?\n\nThis gem focuses exclusively on protocol parsing.\n\nIt intentionally separates:\n\n- Protocol parsing (`pgoutput-parser`)\n- Type decoding (`pgoutput-decoder`)\n- Replication transport/client management\n\nThis keeps the parser small, predictable, dependency-free, and faithful to PostgreSQL's wire format.\n\n---\n\n## Supported MVP Scope\n\nSupports the core non-streaming pgoutput logical replication messages:\n\n- Begin (`B`)\n- Message (`M`)\n- Origin (`O`)\n- Relation (`R`)\n- Type (`Y`)\n- Insert (`I`)\n- Update (`U`)\n- Delete (`D`)\n- Truncate (`T`)\n- Commit (`C`)\n\nThe currently supported message formats are stable across PostgreSQL 10 through PostgreSQL 18.\n\nTupleData supports all base column markers:\n\n| Tuple Value Tag | Meaning                    |\n| --------------- | -------------------------- |\n| `n`             | NULL                       |\n| `u`             | Unchanged TOAST value      |\n| `t`             | Text-formatted raw value   |\n| `b`             | Binary-formatted raw value |\n\n### Planned Support\n\nFuture releases may add support for:\n\n- Stream Start (`S`)\n- Stream Stop (`E`)\n- Stream Commit (`c`)\n- Stream Abort (`A`)\n- Two-Phase Commit messages\n\n---\n\n## What This Gem Does\n\n```text\nPostgreSQL CopyData payload\n           │\n           ▼\n    pgoutput-parser\n           │\n           ▼\nImmutable protocol messages\n```\n\nThe parser understands:\n\n- Message tags and binary field sizes\n- Transaction begin metadata\n- Transaction commit metadata\n- Relation metadata\n- Column flags\n- Column names\n- PostgreSQL type OIDs\n- PostgreSQL type modifiers\n- Insert tuples\n- Update old-key tuples\n- Update old full tuples\n- Update new tuples\n- Delete old-key tuples\n- Delete old full tuples\n- Tuple value markers (`n`, `u`, `t`, `b`)\n\n---\n\n## What This Gem Does Not Do\n\nThe parser does not perform application-level type decoding.\n\nIt does not convert:\n\n- UUID\n- JSONB\n- Timestamp\n- Numeric\n- Array\n- Range\n- PostGIS\n- Custom PostgreSQL types\n\nExample:\n\n```ruby\nvalue.raw\n# =\u003e \"2026-05-31 12:34:56+00\"\n```\n\nThe raw value is preserved exactly as received.\n\nA higher-level decoder may later interpret it.\n\n---\n\n## Non-goals\n\nThis project intentionally does not:\n\n- Manage replication slots\n- Open replication connections\n- Maintain WAL positions\n- Reconnect to PostgreSQL\n- Decode PostgreSQL types\n- Integrate with ActiveRecord\n- Publish events\n- Build CDC pipelines\n\nIts sole responsibility is parsing pgoutput protocol messages.\n\n---\n\n## Installation\n\nAdd this line to your Gemfile:\n\n```ruby\ngem \"pgoutput-parser\"\n```\n\nThen run:\n\n```bash\nbundle install\n```\n\nRequire the library:\n\n```ruby\nrequire \"pgoutput\"\n```\n\n---\n\n## Quick Start\n\n```ruby\nrequire \"pgoutput\"\n\nstream = Pgoutput::RelationTracker.new\n\nstream.process(relation_payload)\n\ninsert = stream.process(insert_payload)\n\ninsert.relation_id\n# =\u003e 42\n\ninsert.tuple.first.raw\n# =\u003e \"7\"\n\ninsert.tuple.first.oid\n# =\u003e 23\n```\n\n---\n\n## Binary Tuple Values\n\nWhen PostgreSQL publishes tuple values using binary format (`b`), the parser preserves the raw bytes exactly as received.\n\n```ruby\nvalue.raw\n# =\u003e \"\\x00\\x00\\x00\\x07\".b\n```\n\nThe parser does not interpret binary values.\n\n---\n\n## Update Messages\n\nPostgreSQL `Update` messages may contain:\n\n- No old tuple\n- An old key tuple (`K`)\n- An old full tuple (`O`)\n\nThey always contain a new tuple (`N`).\n\n```ruby\nupdate = stream.process(update_payload)\n\nupdate.old_key_tuple\n# =\u003e [Pgoutput::Messages::TupleValue, ...] or nil\n\nupdate.old_tuple\n# =\u003e [Pgoutput::Messages::TupleValue, ...] or nil\n\nupdate.new_tuple\n# =\u003e [Pgoutput::Messages::TupleValue, ...]\n```\n\n### Update Tuple Example\n\n```ruby\nupdate = stream.process(update_payload)\n\nupdate.old_key_tuple\nupdate.old_tuple\nupdate.new_tuple\n```\n\n---\n\n## Delete Messages\n\nPostgreSQL `Delete` messages contain either:\n\n- An old key tuple (`K`)\n- An old full tuple (`O`)\n\n```ruby\ndelete = stream.process(delete_payload)\n\ndelete.old_key_tuple\n# =\u003e [Pgoutput::Messages::TupleValue, ...] or nil\n\ndelete.old_tuple\n# =\u003e [Pgoutput::Messages::TupleValue, ...] or nil\n```\n\n---\n\n## Relation Metadata Tracking\n\n`RelationTracker` keeps a local relation cache so tuple values can be associated with PostgreSQL column OIDs defined by preceding Relation (`R`) messages.\n\nThe tracker accepts an optional `relation_cache:` argument. The default is a\nplain Hash, but callers can inject `Ratomic::Map` for a Ractor-safe cache in\nexperimental or parallel setups.\n\nFor a deeper guide, including stream-order behavior, tuple arity validation, and\n`Ratomic::Map` usage, see [docs/relation_tracker.md](docs/relation_tracker.md).\n\nNo type conversion is performed.\n\nOnly protocol metadata is attached.\n\n```ruby\nstream.process(relation_payload)\n\nmessage = stream.process(update_payload)\n\nmessage.new_tuple.map(\u0026:oid)\n# =\u003e [23, 25, 16]\n```\n\nThe relation tracker itself is stateful and maintains relation metadata encountered in the replication stream.\n\nIf a DML tuple's value count does not match the cached relation column count, `RelationTracker` raises\n`Pgoutput::TupleArityError`. This keeps malformed payloads or mismatched stream state from being silently\nannotated with incomplete column metadata.\n\n---\n\n## Ractor Safety\n\n```ruby\nmessage = stream.process(update_payload)\n\nRactor.shareable?(message)\n# =\u003e true\n```\n\nPassing parsed messages to a Ractor:\n\n```ruby\nmessage = stream.process(update_payload)\n\nresult = Ractor.new(message) do |update|\n  update.new_tuple.map(\u0026:raw)\nend.take\n```\n\n---\n\n## Architecture\n\n```text\nPostgreSQL\n      │\n      ▼\nCopyData payload\n      │\n      ▼\nPgoutput::BinaryParser\n      │\n      ▼\nParsed protocol message\n      │\n      ▼\nPgoutput::RelationTracker\n      │\n      ▼\nProtocol message with relation metadata\n      │\n      ▼\nRactor-safe protocol message\n```\n\n---\n\n## Public API\n\n### Pgoutput::BinaryParser\n\nParses a single pgoutput payload without stream state.\n\n```ruby\nmessage = Pgoutput::BinaryParser.new(payload).parse\n```\n\n### Pgoutput::RelationTracker\n\nParses messages in stream order and remembers relation metadata.\n\n```ruby\nstream = Pgoutput::RelationTracker.new\n\nstream.process(relation_payload)\n\nmessage = stream.process(insert_payload)\n```\n\n### Optional Usage\n\n`RelationTracker` is optional.\n\nIf relation metadata tracking is not required, payloads can be parsed directly:\n\n```ruby\nmessage =\n  Pgoutput::BinaryParser\n    .new(payload)\n    .parse\n```\n\n---\n\n## RelationTracker Lifecycle\n\nA `RelationTracker` should be created per logical replication stream.\n\n```ruby\nstream = Pgoutput::RelationTracker.new\n```\n\nThe tracker maintains relation metadata discovered during the stream and therefore should not be reused across unrelated replication sessions.\n\n---\n\n## Message Classes\n\n```ruby\nPgoutput::Messages::Begin\nPgoutput::Messages::Message\nPgoutput::Messages::Origin\nPgoutput::Messages::Relation\nPgoutput::Messages::Type\nPgoutput::Messages::Column\nPgoutput::Messages::TupleValue\nPgoutput::Messages::Insert\nPgoutput::Messages::Update\nPgoutput::Messages::Delete\nPgoutput::Messages::Truncate\nPgoutput::Messages::Commit\n```\n\n---\n\n## Type Signatures\n\nRBS signatures are included:\n\n```text\nsig/pgoutput.rbs\n```\n\nRun Steep:\n\n```bash\nbundle exec steep check\n```\n\n---\n\n## Testing\n\nRun all tests:\n\n```bash\nbundle exec rake test\n```\n\nRun with coverage:\n\n```bash\nCOVERAGE=true bundle exec rake test\n```\n\n---\n\n## Benchmarking\n\nRun the parser throughput benchmark:\n\n```bash\nruby benchmark/parser_throughput.rb\n```\n\nThe benchmark reports single-process parser throughput, relation-tracker throughput, and Ractor-parallel throughput. It is intended to show both the single-thread baseline and the Ruby 4 Ractor path this parser is designed to support. Relation-tracker scenarios can also compare the default Hash relation cache with an optional `Ratomic::Map` cache.\n\nTune the run with environment variables:\n\n| Variable | Default | Description |\n| -------- | ------- | ----------- |\n| `PGOUTPUT_BENCH_ITERATIONS` | `100000` | Iterations per selected scenario. |\n| `PGOUTPUT_BENCH_WARMUP` | `1000` | Warmup iterations before timing. |\n| `PGOUTPUT_BENCH_RACTORS` | `2` or CPU count, whichever is lower | Number of Ractor workers for Ractor scenarios. |\n| `PGOUTPUT_BENCH_SCENARIOS` | `all` | Comma-separated scenarios: `binary`, `tracker_dml`, `tracker_mixed`, `ractor_binary`, `ractor_tracker`, or `all`. |\n| `PGOUTPUT_BENCH_RELATION_CACHE` | `hash` | Comma-separated relation-cache backends for tracker scenarios: `hash`, `ratomic`, or `all`. |\n\nExamples:\n\n```bash\nPGOUTPUT_BENCH_ITERATIONS=10000 ruby benchmark/parser_throughput.rb\nPGOUTPUT_BENCH_SCENARIOS=binary,tracker_mixed ruby benchmark/parser_throughput.rb\nPGOUTPUT_BENCH_RACTORS=4 PGOUTPUT_BENCH_SCENARIOS=ractor_binary,ractor_tracker ruby benchmark/parser_throughput.rb\nPGOUTPUT_BENCH_RELATION_CACHE=all PGOUTPUT_BENCH_SCENARIOS=tracker_mixed,ractor_tracker ruby benchmark/parser_throughput.rb\n```\n\nSample Ruby 4 output:\n\n```text\npgoutput-parser throughput\niterations=1000 warmup=10 ractors=2 scenarios=tracker_mixed,ractor_tracker relation_cache=hash,ratomic ruby=4.0.5\nRelationTracker hash               7000 messages in   0.163s        42891 msg/s\nRelationTracker ratomic            7000 messages in   0.131s        53579 msg/s\nRactor RelationTracker hash       14000 messages in   0.197s        71097 msg/s\nRactor RelationTracker ratomic      14000 messages in   0.146s        96190 msg/s\n```\n\nInterpret the Ractor rows as aggregate throughput across workers. They are not a replacement for the single-process rows; they demonstrate the parser's shareable-message design under parallel execution.\n\n---\n\n## Development\n\nGenerate YARD documentation:\n\n```bash\nbundle exec yard doc\n```\n\n---\n\n## Ecosystem Direction\n\nThis gem is the protocol layer.\n\n```text\npgoutput-parser\n      │\n      ▼\nProtocol messages\n      │\n      ▼\npgoutput-decoder\n      │\n      ▼\nApplication objects\n```\n\n`pgoutput-parser` should remain small, dependency-free, binary-safe, and faithful to PostgreSQL's wire format.\n\n---\n\n## License\n\nMIT.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkanutocd%2Fpgoutput-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkanutocd%2Fpgoutput-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkanutocd%2Fpgoutput-parser/lists"}