{"id":50422587,"url":"https://github.com/youyuanwu/pg_transport","last_synced_at":"2026-05-31T09:02:56.878Z","repository":{"id":360328882,"uuid":"1241010923","full_name":"youyuanwu/pg_transport","owner":"youyuanwu","description":"pgwire listener inside PostgreSQL ","archived":false,"fork":false,"pushed_at":"2026-05-26T00:13:50.000Z","size":840,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-26T01:26:36.894Z","etag":null,"topics":["pgwire","postgresql"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/youyuanwu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-16T21:12:46.000Z","updated_at":"2026-05-26T00:13:54.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/youyuanwu/pg_transport","commit_stats":null,"previous_names":["youyuanwu/pg_transport"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/youyuanwu/pg_transport","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/youyuanwu%2Fpg_transport","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/youyuanwu%2Fpg_transport/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/youyuanwu%2Fpg_transport/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/youyuanwu%2Fpg_transport/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/youyuanwu","download_url":"https://codeload.github.com/youyuanwu/pg_transport/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/youyuanwu%2Fpg_transport/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33725060,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-31T02:00:06.040Z","response_time":95,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["pgwire","postgresql"],"created_at":"2026-05-31T09:02:55.922Z","updated_at":"2026-05-31T09:02:56.872Z","avatar_url":"https://github.com/youyuanwu.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pg_transport\n\n\u003e Status: **research framework** — not production software.\n\u003e Language: **Rust + pgrx 0.18** · Target: **PostgreSQL 18** · License: MIT.\n\n`pg_transport` is a PostgreSQL extension that experiments with\n**alternative in-process network entry points** for PG. It runs as a\nsingle `shared_preload_libraries` cdylib that boots a tokio-based\nfrontend bgworker and an autoscaling pool of backend bgworkers. New\nconnections are accepted by the frontend, then handed off to a\npre-warmed slot bgworker via `SCM_RIGHTS` (kernel fd-passing over a\nUnix socket) — skipping postmaster `fork()` and the per-backend\nCatCache/RelCache cold start.\n\nThe explicit goal is to make swapping transports a small, controlled\nexperiment with comparable benchmark numbers, *not* to ship a\nproduction server. That said, the warm-pool + fd-handoff design\nalready **outperforms vanilla PG 18 in connection-churn workloads\n(up to ~15×) and in concurrent prepared-statement OLTP past physical-core\nsaturation (up to ~2.2×)**, while staying at parity for steady-state\nsingle-statement workloads on a long-lived connection — see\n[Benchmarks](#benchmarks) below for the full picture, including the\nregimes where it currently regresses.\n\n## Features\n\n- Transport plugin: **`tcp_handoff`** (FE/BE v3 over TCP, optional\n  TLS via rustls).\n- Autoscaling backend pool of bgworkers (`max_backend_pool_size`\n  GUC, demand-driven grow, idle reap).\n- Two execution backends, selectable per-session via\n  `pg_transport.execution_backend`:\n  - `spi`     — routes SQL through `SPI_execute_plan_with_params`.\n  - `direct`  — routes SQL through `CreateCachedPlan` + `Portal*`\n    with a custom `DestReceiver` (no `SPI_tuptable` materialisation).\n- Both simple-query (`'Q'`) and extended-query\n  (Parse/Bind/Describe/Execute) paths.\n- Auth via `pg_hba.conf` or extension-owned table\n  (`pg_transport.auth_source` GUC).\n\nSee [docs/design/README.md](docs/design/README.md) for the full doc index.\n\n## Process architecture\n\n```\n                       ┌───────────────────────────────────────────────┐\n                       │  PostgreSQL postmaster   (unmodified PG 18)   │\n                       │  spawns bgworkers from                        │\n                       │  shared_preload_libraries                     │\n                       └────────────────────────┬──────────────────────┘\n                                                │  register + fork\n                                                │  (frontend at startup,\n                                                │   slots via load_dynamic)\n                                                ▼\n   ┌────────────┐    TCP    ┌─────────────────────────────────────────┐\n   │   client   │──────────▶│  Frontend bgworker   (1 process)        │\n   │   libpq /  │   :5454   │  • tokio current-thread + LocalSet      │\n   │   psql /   │           │  • tcp_handoff accept loop              │\n   │   driver   │           │  • backend pool dispatcher              │\n   └──────┬─────┘           │  • HandoffHandle vending                │\n          │                 └────────────────────┬────────────────────┘\n          │                                      │  SCM_RIGHTS fd-pass\n          │                                      │  over UDS frontend.sock\n          │  FE/BE v3                            ▼\n          │  (post-handoff) ┌─────────────────────────────────────────┐\n          │                 │  Slot bgworker pool   (autoscaling,     │\n          │                 │                        up to 64 slots)  │\n          │                 │  ┌──────────────┐  ┌──────────────┐     │\n          └────────────────▶│  │ slot[0]      │  │ slot[1]      │ ... │\n                            │  │ pgwire-v3    │  │ pgwire-v3    │     │\n                            │  │ TLS + auth   │  │ TLS + auth   │     │\n                            │  │ SPI / direct │  │ SPI / direct │     │\n                            │  └──────────────┘  └──────────────┘     │\n                            └────────────────────┬────────────────────┘\n                                                 │  SPI_execute_plan_with_params   (spi)\n                                                 │  CreateCachedPlan + Portal*     (direct)\n                                                 ▼\n                                  PG shared catalogs, buffers, WAL, locks\n```\n\nThree process classes, one cdylib:\n\n- **Postmaster** — unmodified PG 18. Boots the extension via\n  `shared_preload_libraries`; spawns the frontend bgworker at\n  startup and slot bgworkers on demand (`load_dynamic`).\n- **Frontend bgworker** (1 process) — owns the tokio runtime, the\n  TCP listener (`tcp_handoff` on `:5454`), and the pool dispatcher.\n  Accepts client connections and ships their fds to ready slots over\n  a single UDS (`SCM_RIGHTS`). Never touches the FE/BE wire bytes.\n- **Slot bgworkers** (autoscaling up to `max_backend_pool_size`,\n  default 64) — one tokio runtime per slot. Receives client fds,\n  drives the FE/BE v3 wire (via the `pgwire` crate), terminates TLS,\n  performs auth, then executes SQL through the **SPI bridge** or the\n  **direct backend** (`Portal*` + custom `DestReceiver`) per the\n  session's `pg_transport.execution_backend` setting. Slots are reset\n  between handoffs and reaped when idle.\n\nSource map: [`crates/core/`](crates/core) (cdylib),\n[`crates/api/`](crates/api) (trait surface),\n[`crates/bench/`](crates/bench) + [`crates/e2e/`](crates/e2e) (drivers),\n[`docs/`](docs) (design + benchmark artefacts),\n[`scripts/`](scripts) (sweep runners), [`just/`](just) (recipes).\n\n## Quick start\n\nRequires Rust 1.95+ and the `just` task runner.\n\n```sh\n# One-time: provision PG 18 under $PGRX_HOME (~5–10 min, ~1.5 GB)\njust init\n\n# Build + install into the pgrx PG 18 tree, run e2e suite\njust e2e\n```\n\nConnect via either port emitted by the e2e harness — vanilla PG on\n`127.0.0.1:54329`, pg_transport on `127.0.0.1:5454`.\n\n## Benchmarks\n\nMeasured 2026-05-24 on an Azure 8-vCPU (4 physical core) VM at commit\n[`61f7e16`](docs/bench/README.md). Full methodology, raw TSV, and\nreproduction commands in [docs/bench/README.md](docs/bench/README.md).\n\n| Regime                                       | Tool             | pg_transport vs vanilla PG 18 |\n| -------------------------------------------- | ---------------- | ----------------------------- |\n| Long-lived conn, `SELECT 1` round-trip       | custom bench     | **1.12 – 1.26×** (1 → 16 conns, direct) |\n| Long-lived conn, single-statement OLTP       | pgbench `-S` / `-N` | **~parity** |\n| Long-lived conn, multi-statement OLTP (TPC-B)| pgbench (default)   | **1.07 – 1.10×** |\n| **Connection-churn (`pgbench -S -C`)**       | pgbench          | **10.9× / 14.4× / 14.7×** at 4 / 16 / 32 clients |\n| Prepared-statement point select              | sysbench         | **1.20 – 1.31×** (4 → 32 threads) |\n| Prepared-statement BEGIN/COMMIT read-only    | sysbench         | **1.50 – 2.18×** (12 statements per tx) |\n| Prepared-statement indexed UPDATE            | sysbench         | **0.74 – 0.97×** (regresses past physical-core saturation) |\n\nThe connection-churn row is the architectural headline: vanilla pays\n`fork()` + cache warm-up per connection (~1–10 ms); pg_transport\nhands the fd to an already-warm slot bgworker (~0 ms). The\n`oltp_read_only` row at threads=32 is the steady-state headline:\nwarm pool multiplexing under over-committed concurrency.\n\nThe write regression at threads ≥ 16 is host-shaped (4 physical cores\n-- tuple-lock serialization) and is reported as-is rather than tuned\naround; see [docs/bench/README.md §6](docs/bench/README.md#6-sysbench--prepared-statement-oltp).\n\n## Further reading\n\n- [docs/design/README.md](docs/design/README.md) — full design doc index.\n- [docs/design/architecture.md](docs/design/architecture.md) — 3-layer\n  architecture (transport / frontend core / backend pool).\n- [docs/design/performance.md](docs/design/performance.md) —\n  per-query data-path trace and the structural fast paths.\n- [docs/design/comparison.md](docs/design/comparison.md) — how\n  pg_transport relates to pgbouncer, Odyssey, pg_background, Omnigres.\n- [docs/design/roadmap.md](docs/design/roadmap.md) — phased build\n  plan, open questions, risks.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyouyuanwu%2Fpg_transport","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyouyuanwu%2Fpg_transport","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyouyuanwu%2Fpg_transport/lists"}