{"id":25996907,"url":"https://github.com/clickup/pg-tube","last_synced_at":"2025-08-08T19:05:03.806Z","repository":{"id":245092875,"uuid":"741397463","full_name":"clickup/pg-tube","owner":"clickup","description":"Simple PostgreSQL-to-anywhere data stream pipeline without Kafka and other external buses","archived":false,"fork":false,"pushed_at":"2025-02-20T16:26:35.000Z","size":156,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-05T16:55:23.776Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clickup.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-10T10:06:00.000Z","updated_at":"2025-02-26T13:05:45.000Z","dependencies_parsed_at":"2024-06-19T19:25:52.133Z","dependency_job_id":"7f6b95f3-98d2-4a4a-bdbd-475a962709a9","html_url":"https://github.com/clickup/pg-tube","commit_stats":null,"previous_names":["clickup/pg-tube"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/clickup/pg-tube","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clickup%2Fpg-tube","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clickup%2Fpg-tube/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clickup%2Fpg-tube/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clickup%2Fpg-tube/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clickup","download_url":"https://codeload.github.com/clickup/pg-tube/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clickup%2Fpg-tube/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269474785,"owners_count":24423102,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-08T02:00:09.200Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-05T16:55:26.851Z","updated_at":"2025-08-08T19:05:03.729Z","avatar_url":"https://github.com/clickup.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pg-tube: collect touch events from tables\n\n- \"Tube\" is internally a queue table with batches of touch event ids (\"pods\")\n  from some table/shard.\n- Ideologically, \"tube\" is a user-space logical replication slot which only\n  records touch events (ids).\n- Tubes can be added (`tube_ensure_exists(tube, partitions, predicate)`) and\n  removed (`tube_ensure_absent(tube)`) at run-time. When removing, all the\n  tables are automatically detached from the tube.\n- Tube may have a SQL predicate assigned. This predicate will apply to the rows\n  of the attached tables that trigger touch events. If the predicate returns\n  false, then the id will not be touched. This also applies to backfill: only\n  the rows matching predicate will generate backfill pods. Predicate introduces\n  some assumption about the structure of the tables which can be attached to the\n  tube. (Ideally, predicates should've been per-tube-table pairs, but it is\n  harder to implement - we'd need per-tube-table trigger functions as well - so\n  we end up with a more pragmatic solution for now.)\n- Tubes and attached tables can be enumerated using `tube_list()` function.\n- Each tube table has 5 main columns: 1) some auto-incremental sequence, 2)\n  shard (some number which is assigned to a table when it's attached), 3)\n  operation type (with a special operations '}' to denote the backfill ending),\n  4) an array (chunk) of bigint ids, and 5) some optional JSON payload.\n- Pods can be \"regular\" (contain non-empty ids) and \"control\" (have empty ids\n  and non-empty payload). Control pods are: \"backfill_schedule\", \"backfill_end\".\n- An arbitrary table can be \"attached\" to a tube\n  (`tube_table_ensure_attached(tube, table, shard)`). When attaching, you must\n  also provide some \"shard number\" value which will then be recorded in the tube\n  along with ids. (If shard number is not provided, pg-tube tries to infer it\n  from the table name's numeric suffix.) Once attached, all insert/update/delete\n  operations will result into adding a \"touch\" pod to the corresponding tube.\n- A table can also be \"detached\" from a tube (with\n  `tube_table_ensure_detached(tube, table)`).\n- To run the initial piping, backfill function should be called in separate\n  transactions, one after another (`tube_backfill_step1()`,\n  `tube_backfill_step2()` and then `tube_backfill_step3(step1_result, tube,\n  table, order_col, shard)`). This will bulk-insert all the ids from the table\n  to the tube (op='B') record. Ordering by the provided column in step3 is\n  important, because for encrypted databases (e.g. ECIES), ephemeral shared key\n  for neighbor rows will likely be cached with short expiration time. So the\n  closer the updates we process are in time, the higher is the chance to have a\n  Diffie-Hellman hit.\n- To bulk-insert touch events from some large SQL query results (e.g. to\n  generate a large number of \"fanout touches\"), use `tube_pods_insert()`\n  function.\n- There is `tube_stats()` function which shows all the details about the current\n  tubes structure and content.\n\nIn its nature, tubes are eventual-consistent, mainly because of enrichment\nprocess which takes time and delivers the results with unpredictable latency. It\nmeans that there is absolutely no way to rely on some external source of order\nin the events (like row version), and the only way to solve it all is to never\nreplay the same id concurrently. We also strictly rely on the fact that the\ndownstream for the replay should be eventual-consistent and e.g. never reorder\nwrites A and B for the same id once an ack from write A is received.\n\nThe application code should constantly monitor all existing tubes. One tube may\ncorrespond to one logical unit of work (e.g. we may have a live ES index, a\nspare ES index and a cache-invalidation tube, thus 3 tubes). For each tube, it\nspawns a beforehand-known number of separate replication stream workers, each\nworker processing only ids from their own set of shards (e.g. a \"shard % 3\"\nformula can be used as a function to allocate different touch events to\ndifferent workers with numbers 0, 1, 2). Each worker reads ids from the tube,\nprocesses them in a strictly serial way ordering from lowest seq to highest (so\nit's guaranteed that there is never a concurrency when processing the same id)\nand removes from the tube. To reach eventual consistency, the workers must do\nenrichment (i.e. loading the actual data from the DB after receiving a touch\nevent for some id). It's critical that each shard is processed by exactly one\nworker (no parallelism within one shard), and that the same id is never\nprocessed concurrently; having these assumptions allows us to NOT keep record\nversions in the tube and just rely on natural eventual consistency ordering (we\ncan't even count on whether a record in the tube corresponds to a deletion or to\nan insert/update).\n\nBackfill events (op='B' or ALL pods scheduled when PG's application_name\ncontained substring \"backfill\") are processed in the exact same way as\ninsert/update/delete/touch events, but with lower priority (the application uses\n`ORDER BY op='B', seq` clause in SELECT which matches the index exactly). Also,\nonce a control pod with `type=backfill_end, start_seq=S` payload is received, it\nsignals the worker that pods with seq \u003c S are not needed on the downstream\nanymore, and they should be removed (garbage collected).\n\n## Downstream shard number vs. upstream shard number\n\nCurrently, pg-tube can choose downstream shard number only deterministically by\nPG microshard number. There are several reasons for that, which this chapter is\nabout.\n\nWhen we delete a row from PG, we receive a touch event with `{id, shard}` info.\nThis is the only info we have to be able to identify the downstream shard and\ndownstream doc id to replay the deletion over into the downstream (we can't look\nat other row props since they don't exist at the time of the deletion). Which\nmeans that downstream shard routing must be deterministic by either a prefix of\ntouch.id or by touch.shard (which is typically the same).\n\nAnother use-case is garbage collection: it happens after the backfill-over\nphase. There, we query an individual “touch shard number” in the downstream (see\nGarbage.ts) using the routing based on that shard number for “docs updated\nearlier than the backfill has started”. All found ids get removed (in fact, the\nexactly same workflow applies as for deletion, with enrichment etc., to\neliminate race conditions).\n\nFor the future, no-one in theory stops us from writing an arbitrary value to\n`touch.shard` at the moment of its injection into the tube. In this sense, the\nnaming is slightly misleading: it should've been named `touch.routing`, not\n`touch.shard`, since it only determines the downstream routing, and it also\ndefines the work distribution among multiple running jobs (to not let them\ncollide).\n\nCurrently, pg-tube infers the value for `touch.shard` from the numeric part of\nthe table schema's name. And it does it statically, at tube attachment time\n(i.e. not for every touch event). I.e. the shard number is hardcoded in the e.g.\ntube trigger (available via `TG_ARGV[0]`) which receives a set of\ninserted/updated/deleted rows.\n\nSo, it seems like a doable feature, to implement custom routing. Unfortunately,\nimplementing it is way not so easy. This is mainly because of the above static\nassumptions (like if we receive a batch of touched rows in the tube trigger, we\nalways assume they relate to the same shard, since the shard is determined by\nthe table's schema).\n\n## Testing\n\nUse `yarn test` and `yarn test:db` to run the automated tests.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclickup%2Fpg-tube","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclickup%2Fpg-tube","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclickup%2Fpg-tube/lists"}