{"id":21436758,"url":"https://github.com/modfin/creek-pg-client","last_synced_at":"2025-03-16T23:23:47.228Z","repository":{"id":192010671,"uuid":"675064288","full_name":"modfin/creek-pg-client","owner":"modfin","description":"A PostgreSQL consumer for Creek","archived":false,"fork":false,"pushed_at":"2024-03-14T22:08:33.000Z","size":178,"stargazers_count":0,"open_issues_count":4,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-01-23T09:33:58.365Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/modfin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-05T16:46:31.000Z","updated_at":"2023-08-31T14:52:29.000Z","dependencies_parsed_at":"2023-09-02T02:11:19.785Z","dependency_job_id":"d2b56fd2-197e-4cc0-95a0-92ede79a53dd","html_url":"https://github.com/modfin/creek-pg-client","commit_stats":null,"previous_names":["modfin/creek-pg-client"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fcreek-pg-client","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fcreek-pg-client/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fcreek-pg-client/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modfin%2Fcreek-pg-client/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/modfin","download_url":"https://codeload.github.com/modfin/creek-pg-client/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243945775,"owners_count":20372929,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-23T00:15:12.961Z","updated_at":"2025-03-16T23:23:47.207Z","avatar_url":"https://github.com/modfin.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Creek postgres consumer\n\nConsumer for [Creek](https://github.com/modfin/creek) that applies changes from\nCreek to a postgres database. It has the following features:\n\n* Listen for WAL events on creek and apply changes in \"realtime\" to a target\n  PostgreSQL database.\n* Create database tables based on the schema of a table in creek.\n* List available taken snapshots of a table in creek.\n* Apply an already existing snapshot to a corresponding table in the target\n  database.\n* Request a new snapshot from creek, and apply it to a table in the target \n  database.\n* An API that allows interacting with and changing the configuration of the\n  consumer while it is running.\n\nUsed in conjunction with a [Creek producer](https://github.com/modfin/creek),\nthis system enables keeping Postgres tables in sync across databases, including\nacross different versions of Postgres.\n\n## How to run\n\nBuild using\n\n```\ngo build cmd/clientd.go\n```\n\n## Configuring\n\nThere are two main ways of interacting with and configuring the program.\nFirstly, it is possible to configure the client using a CLI. To view all\ncommands and options, you can run `./clientd --help`.\n\nPlease note that configuring using CLI the **while the client is running** (ie.\n`./clientd serve`) is **not recommended** and may lead to unintended\nconsequences. It is mainly intended to be used when first setting up a client\nand for ergonomic reasons when testing/debugging.\n\nIf you want to interact with the client while it is running, please use the\nPostgreSQL API. This API has functionality for starting and stopping processing\nof WAL events for specific tables, as well as applying snapshots.\n\n### Configuring using CLI\n\nCreate a table in the target database.\n\n```\n./clientd --nats-uri uri --db-uri uri create-schemas \\\n    db.namespace.table:target_namespace.target_table\n```\n\nRequest and apply a new snapshot of the table.\n\n```\n./clientd --nats-uri uri --db-uri uri snapshot \\\n     db.namespace.table:target_namespace.target_table\n```\n\nAdd the table to start listening for WAL events.\n\n```\n./clientd --nats-uri uri --db-uri uri add-tables \\\n    db.namespace.table:target_namespace.target_table\n```\n\nList active streams.\n\n```\n./clientd --nats-uri uri --db-uri uri list-tables\n```\n\nList existing snapshots for a table.\n\n```\n./clientd --nats-uri uri --db-uri uri list-snapshots db.namespace.table\n```\n\nApply an existing snapshot for a table to a target table.\n\n```\n./clientd --nats-uri uri --db-uri uri apply-snapshot \\\n     --name CREEK.db.snap.namespace.table.YYYYMMDDHHMMSS_ms_id \\\n     db.namespace.table:target_namespace.target_table\n```\n\nStop listening for WAL events for a table.\n\n```\n./clientd --nats-uri uri --db-uri uri remove-table target_namespace.target_table\n```\n\n### Configuring using the PostgreSQL API\n\nStart the service\n\n```\n./clientd --nats-uri uri --db-uri uri serve\n```\n\nIn a PostgreSQL shell connected to the same database and namespace:\n\n```SQL\n-- Create a copy of the db.namespace.table table in the database\nSELECT _creek_consumer.create_schema('db.namespace.table:target_namespace.target_table');\n\n-- Request and apply a new snapshot of the table. Valid modes are 'upsert' and 'clean'\nSELECT _creek_consumer.snapshot('db.namespace.table:target_namespace.target_table', 'clean');\n\n-- Start listening to WAL events and applying them to table\nSELECT _creek_consumer.add_table('db.namespace.table:target_namespace.target_table');\n\n-- View current status\nSELECT * FROM _creek_consumer.subscriptions;\n\n-- Stop listening and applying changes for table. Valid modes are 'upsert' and 'clean'\nSELECT _creek_consumer.remove_table('target_namespace.target_table', 'upsert');\n\n-- Apply a specific snapshot\nSELECT _creek_consumer.apply_snapshot(\n    'CREEK.db.snap.namespace.table.YYYYMMDDHHMMSS_ms_id', \n    'db.namespace.table:target_namespace.target_table', \n    'upsert');\n\n-- View status or errors from commands that have been run\nSELECT * FROM _creek_consumer._notification_log;\n\n-- View active subscriptions, ie tables that are being listened to\nSELECT * FROM _creek_consumer.subscriptions WHERE active = true;\n```\n\n#### Snapshot modes\n\nThere are currently two different snapshot modes: `'upsert'` and `'clean'`.\nUpsert simply performs `INSERT ... ON CONFLICT UPDATE` statements. Clean creates\na new temporary table, applies the snapshot to the new table, and then replaces\nthe old table with the new table once the snapshot is complete.\n\n#### Notes\n\nIf applying a snapshot to a table that is currently subscribing to WAL events,\nprocessing of WAL events for this table will be paused until the snapshot is\ncomplete. Processing of events will continue from the time the snapshot was\ntaken after it is complete.\n\nIt is safe to restart the service as it will persist its last location in the \ndatabase that it is connected to, and continue streaming WAL events from this\nlocation on startup.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodfin%2Fcreek-pg-client","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmodfin%2Fcreek-pg-client","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodfin%2Fcreek-pg-client/lists"}