{"id":30151885,"url":"https://github.com/misachi/pg_rusted_wire","last_synced_at":"2026-05-15T12:09:03.156Z","repository":{"id":307091687,"uuid":"1027384263","full_name":"misachi/pg_rusted_wire","owner":"misachi","description":"Postgres Wire Protocol implementation","archived":false,"fork":false,"pushed_at":"2025-11-05T13:05:31.000Z","size":212,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-05T14:23:38.134Z","etag":null,"topics":["postgres","replication","wire-protocol"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/misachi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-27T23:04:46.000Z","updated_at":"2025-11-05T13:05:34.000Z","dependencies_parsed_at":"2025-09-11T20:17:52.906Z","dependency_job_id":"2bf632d6-6800-45d7-be43-563bf0ded872","html_url":"https://github.com/misachi/pg_rusted_wire","commit_stats":null,"previous_names":["misachi/pg_rusted_wire"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/misachi/pg_rusted_wire","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/misachi%2Fpg_rusted_wire","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/misachi%2Fpg_rusted_wire/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/misachi%2Fpg_rusted_wire/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/misachi%2Fpg_rusted_wire/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/misachi","download_url":"https://codeload.github.com/misachi/pg_rusted_wire/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/misachi%2Fpg_rusted_wire/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33066193,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-15T11:35:32.926Z","status":"ssl_error","status_checked_at":"2026-05-15T11:35:31.362Z","response_time":103,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["postgres","replication","wire-protocol"],"created_at":"2025-08-11T11:07:58.480Z","updated_at":"2026-05-15T12:09:03.150Z","avatar_url":"https://github.com/misachi.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PG_RUSTED_WIRE\n\n`pg_rusted_wire` is a Rust implementation of the PostgreSQL Wire Protocol, enabling direct communication with PostgreSQL servers. It supports multiple authentication methods (`SCRAM-SHA-256`, `md5`, and `password`) and provides tools for logical replication and interactive SQL querying.\n\nStream table changes to Iceberg tables(or local files) or use the psql-like client for basic database operations, all with a focus on simplicity. This tool is helpful for basic database interactions and testing the wire protocol implementation.\n\n## Requirements\nStreaming to an iceberg table requires the dependencies listed below\n1. Python shared library `python3-dev` for Ubuntu or `python3-devel` for RPM based distributions\n2. PyIceberg which can be installed with the command `pip install \"pyiceberg[s3fs,sql-postgres]\"`. Replace `sql-postgres` with a Catalog store of your choice. Check [https://py.iceberg.apache.org/#installation](https://py.iceberg.apache.org/#installation) for more information.\n3. PyArrow. Install with `pip install pyarrow`\n4. Sqlite database. For running tests locally\n\n## Example Usage\n\n### Logical Replication\n\n#### Streaming to a local file\nThis feature allows you to stream real-time changes from a PostgreSQL table directly to a local file using logical replication. When you start the process, a snapshot of the specified table is first saved as `\u003ctable_name\u003e.data` in your chosen directory configured in `with_config_dir`.\n\nAs new INSERT operations occur in the database, they are appended to this file, enabling you to track changes over time.\n\n```\nuse std::net::Ipv4Addr;\nuse std::str::FromStr;\n\nuse pg_rusted_wire::wire::*;\n\nlet mut client = Client::new(\n    Ipv4Addr::from_str(\u0026args.host).expect(\"IPV4 address error\"),\n    args.port,\n)\n.with_database(\"my_database\") // database to connect to\n.with_user(\"user\") // user with permissions to connect to the database table to be streamed\n.with_replication(\"database\") // this is required for logical replication\n.with_protocol(PROTOCOL_VERSION) // Supported protocol version to use\n.with_config_dir(\"/my/dir\"); // Directory with the right permisions to store the state in. The streamed table file data is also stored here.\n\n// Connect and start streaming changes from the table\nmatch client.connect() {\n    Ok(mut stream) =\u003e {\n        if let Err(e) = client.authenticate2(\u0026mut stream, \u0026args.password) {\n            eprintln!(\"Client Authentication: {}\", e);\n            return;\n        }\n\n        client.run(\n            \u0026mut stream,\n            \"table_name\",\n            \"publication_name\",\n            None,\n        );\n    }\n    Err(e) =\u003e {\n        eprintln!(\"No stream available for client: {}\", e);\n        return;\n    }\n}\n```\nWhen you start the process, a snapshot of the specified table is first saved as `\u003ctable_name\u003e.data` in your chosen `--config-dir` directory\n\n#### Streaming to a Iceberg Table\nYou can also stream real-time changes from a PostgreSQL table directly into an existing Apache Iceberg table. This feature is useful for integrating PostgreSQL logical replication with modern data lake architectures. This has only been tested with PostgreSQL(as the catalog) and Minio(as the object storage backend).\n\nNote that the target Iceberg table must exist before starting replication.\n\nConfiguration: Create a configuration file with your Iceberg and storage details:\n```\nS3_SECRET_KEY=pass1234\nS3_ENDPOINT=http://1.1.1.1:9000\nS3_ACCESS_KEY=minio_user\nCATALOG_URI=postgresql+psycopg2://username:password@1.1.1.1:5432/iceberg\n\n# TABLE_NAME is in the format catalog.schema.table\nTABLE_NAME=catalog_todo.example_s3_schema.employees_test\n```\n\nBelow is sample code for streaming to iceberg. \n```\nuse std::net::Ipv4Addr;\nuse std::str::FromStr;\n\nuse pg_rusted_wire::wire::*;\n\nlet mut client = Client::new(\n    Ipv4Addr::from_str(\u0026args.host).expect(\"IPV4 address error\"),\n    args.port,\n)\n.with_database(\"my_database\") // database to connect to\n.with_user(\"user\") // user with permissions to connect to the database table to be streamed\n.with_replication(\"database\") // this is required for logical replication\n.with_protocol(PROTOCOL_VERSION) // Supported protocol version to use\n.with_config_dir(\"/my/dir\"); // Directory with the right permisions to store the state in. The streamed table file data is also stored here.\n\n// Connect and start streaming changes from the table\nmatch client.connect() {\n    Ok(mut stream) =\u003e {\n        if let Err(e) = client.authenticate2(\u0026mut stream, \u0026args.password) {\n            eprintln!(\"Client Authentication: {}\", e);\n            return;\n        }\n\n        client.run(\n            \u0026mut stream,\n            \"table_name\",\n            \"publication_name\",\n            Some(OutResource::Iceberg{\n                config_path: \".tmp/config\".to_string(), // Provide actual path to the iceberg config file described above\n                schema: None,\n                key: None,\n                buffer_opt: BufferOpt::OnDisk{append: None, delete: None}} // Can be either OnDisk or InMemory buffer\n            ),\n        );\n\n    }\n    Err(e) =\u003e {\n        eprintln!(\"No stream available for client: {}\", e);\n        return;\n    }\n}\n```\n\nThis integration enables you to capture and store PostgreSQL table changes in Iceberg, supporting scalable analytics and simple data lake workflows.\n\n\n\nYou can also use the command-line interface with the example code in [lrepl](examples/lrepl.rs) for convenience:\n\n```\ncargo run --example lrepl -- -u \u003cuser\u003e -P \u003cpassword\u003e -H \u003chost\u003e -d \u003cdatabase\u003e -p \u003cport\u003e --table \u003cname\u003e --publication \u003cpub1\u003e --config-dir \u003cdir\u003e\n```\n\u003e Replace the placeholders with your actual connection information. Use `cargo run --example lrepl -- -h` for help on available flags and defaults.\n\nCurrently, only INSERT and DELETE operations are fully supported. There is partial support for UPDATE operations -- updates to columns that make the key may be added later. ~~Also, support for DELETEs will be added in the future.~~\n\nNote that only one table can be replicated at a time. This tool is ideal for capturing and auditing table changes or for simple data synchronization tasks.\n\n\n### PSQL-Like CLient\nA simple interactive SQL client, similar to psql, is also included as an example. You can use it to connect to your PostgreSQL database and run queries directly from the terminal.\n\nTo start the client, run:\n```\ncargo run --example simpsql\n```\n\u003e Before running, update the connection settings (database name, user, password, host, and port) in the [simpsql](examples/simpsql.rs) file.\n\nOnce running, you can enter SQL commands interactively. Example usage:\n```\nClient connection\nUse Ctrl+c to end\n\npsql\u003e CREATE TABLE foo(id SERIAL PRIMARY KEY, k INT NOT NULL);\npsql\u003e INSERT INTO foo(k) SELECT i FROM generate_series(1002, 1011) i;\npsql\u003e SELECT * FROM foo;\n\nid|k\n1|1002\n2|1003\n3|1004\n4|1005\n5|1006\n6|1007\n7|1008\n8|1009\n9|1010\n10|1011\npsql\u003e\n\nResults are displayed in a simple, readable format. Use Ctrl+C to exit the client. This tool is useful for basic database operations and testing your PostgreSQL connection.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmisachi%2Fpg_rusted_wire","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmisachi%2Fpg_rusted_wire","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmisachi%2Fpg_rusted_wire/lists"}