https://github.com/tmc/pqstream

pqstream turns your postgres database into an event stream
https://github.com/tmc/pqstream

golang grpc postgres

Last synced: 19 days ago
JSON representation

pqstream turns your postgres database into an event stream

• Host: GitHub
• URL: https://github.com/tmc/pqstream
• Owner: tmc
• License: mit
• Created: 2017-09-04T04:47:45.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2023-12-15T02:18:00.000Z (12 months ago)
• Last Synced: 2024-10-30T23:52:17.556Z (about 1 month ago)
• Topics: golang, grpc, postgres
• Language: Go
• Homepage:
• Size: 652 KB
• Stars: 477
• Watchers: 17
• Forks: 24
• Open Issues: 28
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE

Awesome Lists containing this project

• starred-awesome - pqstream - pqstream turns your postgres database into an event stream (Go)

README

        
# pqstream

pqstream is a program that streams changes out of a postgres database with the intent of populating other systems and enabling stream processing of data sets.

[![ci status](https://circleci.com/gh/tmc/pqstream.svg?style=shield)](https://circleci.com/gh/tmc/workflows/pqstream/tree/master) 

[![go report card](https://goreportcard.com/badge/github.com/tmc/pqstream)](https://goreportcard.com/report/github.com/tmc/pqstream)

[![coverage](https://codecov.io/gh/tmc/pqstream/branch/master/graph/badge.svg)](https://codecov.io/gh/tmc/pqstream)

## installation

```sh

$ go get -u github.com/tmc/pqstream/cmd/{pqs,pqsd}

```

## basic usage

create an example database:

```sh

$ createdb dbname

# echo "create table notes (id serial, created_at timestamp, note text)" | psql dbname

```

connect the agent:

```sh

$ pqsd -connect postgresql://user:pass@host/dbname

```

connect the cli:

```sh

$ pqs

```

at this point you will see streams of database operations rendered to stdout:

(in a psql shell):

```sql

dbname=# insert into notes values (default, default, 'here is a sample note');

INSERT 0 1

dbname=# insert into notes values (default, default, 'here is a sample note');

INSERT 0 1

dbname=# update notes set note = 'here is an updated note' where id=1;

UPDATE 1

dbname=# delete from notes where id = 1;

DELETE 1

dbname=#

```

our client should now show our operations:

```sh

$ pqs

{"schema":"public","table":"notes","op":"INSERT","id":"1","payload":{"created_at":null,"id":1,"note":"here is a sample note"}}

{"schema":"public","table":"notes","op":"INSERT","id":"2","payload":{"created_at":null,"id":2,"note":"here is a sample note"}}

{"schema":"public","table":"notes","op":"UPDATE","id":"1","payload":{"created_at":null,"id":1,"note":"here is an updated note"},"changes":{"note":"here is a sample note"}}

{"schema":"public","table":"notes","op":"DELETE","id":"1","payload":{"created_at":null,"id":1,"note":"here is an updated note"}}

```

## field redaction

If there's a need to prevent sensitive fields (i.e. PII) from being exported the `redactions` flag can be used with `pqsd`:

```sh

$ pqsd -connect postgresql://user:pass@host/dbname -redactions='{"public":{"users":["first_name","last_name","email"]}}'

```

The `redactions` is encoded in [JSON](http://json.org/) and conforms to the following layout: 

``` json

'{"schema":{"table":["field1","field2"]}}'`

```