{"id":13700922,"url":"https://github.com/tmc/pqstream","last_synced_at":"2025-04-05T03:12:58.327Z","repository":{"id":40444493,"uuid":"102319698","full_name":"tmc/pqstream","owner":"tmc","description":"pqstream turns your postgres database into an event stream","archived":false,"fork":false,"pushed_at":"2023-12-15T02:18:00.000Z","size":668,"stargazers_count":477,"open_issues_count":28,"forks_count":24,"subscribers_count":17,"default_branch":"master","last_synced_at":"2024-10-30T23:52:17.556Z","etag":null,"topics":["golang","grpc","postgres"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tmc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-09-04T04:47:45.000Z","updated_at":"2024-10-05T10:54:48.000Z","dependencies_parsed_at":"2024-06-18T21:41:21.373Z","dependency_job_id":null,"html_url":"https://github.com/tmc/pqstream","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmc%2Fpqstream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmc%2Fpqstream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmc%2Fpqstream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tmc%2Fpqstream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tmc","download_url":"https://codeload.github.com/tmc/pqstream/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247280272,"owners_count":20912967,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","grpc","postgres"],"created_at":"2024-08-02T20:01:08.689Z","updated_at":"2025-04-05T03:12:58.312Z","avatar_url":"https://github.com/tmc.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# pqstream\n\npqstream is a program that streams changes out of a postgres database with the intent of populating other systems and enabling stream processing of data sets.\n\n[![ci status](https://circleci.com/gh/tmc/pqstream.svg?style=shield)](https://circleci.com/gh/tmc/workflows/pqstream/tree/master) \n[![go report card](https://goreportcard.com/badge/github.com/tmc/pqstream)](https://goreportcard.com/report/github.com/tmc/pqstream)\n[![coverage](https://codecov.io/gh/tmc/pqstream/branch/master/graph/badge.svg)](https://codecov.io/gh/tmc/pqstream)\n\n## installation\n\n```sh\n$ go get -u github.com/tmc/pqstream/cmd/{pqs,pqsd}\n```\n\n## basic usage\n\ncreate an example database:\n\n```sh\n$ createdb dbname\n# echo \"create table notes (id serial, created_at timestamp, note text)\" | psql dbname\n```\n\nconnect the agent:\n\n```sh\n$ pqsd -connect postgresql://user:pass@host/dbname\n```\n\nconnect the cli:\n```sh\n$ pqs\n```\n\nat this point you will see streams of database operations rendered to stdout:\n\n\n(in a psql shell):\n\n```sql\ndbname=# insert into notes values (default, default, 'here is a sample note');\nINSERT 0 1\ndbname=# insert into notes values (default, default, 'here is a sample note');\nINSERT 0 1\ndbname=# update notes set note = 'here is an updated note' where id=1;\nUPDATE 1\ndbname=# delete from notes where id = 1;\nDELETE 1\ndbname=#\n```\n\nour client should now show our operations:\n```sh\n$ pqs\n{\"schema\":\"public\",\"table\":\"notes\",\"op\":\"INSERT\",\"id\":\"1\",\"payload\":{\"created_at\":null,\"id\":1,\"note\":\"here is a sample note\"}}\n{\"schema\":\"public\",\"table\":\"notes\",\"op\":\"INSERT\",\"id\":\"2\",\"payload\":{\"created_at\":null,\"id\":2,\"note\":\"here is a sample note\"}}\n{\"schema\":\"public\",\"table\":\"notes\",\"op\":\"UPDATE\",\"id\":\"1\",\"payload\":{\"created_at\":null,\"id\":1,\"note\":\"here is an updated note\"},\"changes\":{\"note\":\"here is a sample note\"}}\n{\"schema\":\"public\",\"table\":\"notes\",\"op\":\"DELETE\",\"id\":\"1\",\"payload\":{\"created_at\":null,\"id\":1,\"note\":\"here is an updated note\"}}\n```\n\n\n## field redaction\n\nIf there's a need to prevent sensitive fields (i.e. PII) from being exported the `redactions` flag can be used with `pqsd`:\n\n\n```sh\n$ pqsd -connect postgresql://user:pass@host/dbname -redactions='{\"public\":{\"users\":[\"first_name\",\"last_name\",\"email\"]}}'\n```\n\nThe `redactions` is encoded in [JSON](http://json.org/) and conforms to the following layout: \n``` json\n'{\"schema\":{\"table\":[\"field1\",\"field2\"]}}'`\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftmc%2Fpqstream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftmc%2Fpqstream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftmc%2Fpqstream/lists"}