{"id":26410994,"url":"https://github.com/edgeflare/pgo","last_synced_at":"2025-03-17T20:20:08.624Z","repository":{"id":281130947,"uuid":"934917821","full_name":"edgeflare/pgo","owner":"edgeflare","description":"PostgREST-compatible REST API, Debezium-compatible CDC","archived":false,"fork":false,"pushed_at":"2025-03-15T15:36:46.000Z","size":355,"stargazers_count":12,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-15T16:29:32.082Z","etag":null,"topics":["change-data-capture","event-stream","logical-replication","oidc","postgresql","rest-api"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/edgeflare.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-18T16:02:25.000Z","updated_at":"2025-03-15T15:36:49.000Z","dependencies_parsed_at":"2025-03-07T06:39:41.012Z","dependency_job_id":null,"html_url":"https://github.com/edgeflare/pgo","commit_stats":null,"previous_names":["edgeflare/pgo"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edgeflare%2Fpgo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edgeflare%2Fpgo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edgeflare%2Fpgo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edgeflare%2Fpgo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/edgeflare","download_url":"https://codeload.github.com/edgeflare/pgo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244102783,"owners_count":20398386,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["change-data-capture","event-stream","logical-replication","oidc","postgresql","rest-api"],"created_at":"2025-03-17T20:20:08.036Z","updated_at":"2025-03-17T20:20:08.604Z","avatar_url":"https://github.com/edgeflare.png","language":"Go","readme":"# pgo (`/pɪɡəʊ/`): Postgres integrations in Go\n\n\u003e **usability status: experimental**\n\nThis project follows a **Work → Right → Fast** approach:\n\n1. make it work\n2. refine it\n3. optimize it\n\nIt's in the early stage, so the code may be rough/incomplete. Join us in building and improving it!\n\n```sh\ngo install github.com/edgeflare/pgo@latest # or make build or download from release page\n```\n\n## [PostgREST](https://docs.postgrest.org/en/stable/references/api/tables_views.html) compatible REST API\n\n```sh\nPGO_REST_PG_CONN_STRING=\"host=localhost port=5432 user=postgres password=secret dbname=testdb\"\npgo rest --config pkg/config/example.config.yaml\n```\n\nSee [godoc](https://pkg.go.dev/github.com/edgeflare/pgo/pkg/rest) and `pgo rest --help` for more.\n\n## Stream Postgres changes to NATS, MQTT, Kafka, Clickhouse, etc\n\n[![asciicast](https://asciinema.org/a/704523.svg)](https://asciinema.org/a/704523)\n\n1. Start Postgres, NATS, Kafka, MQTT broker and pgo pipeline as containers\n\n```sh\ngit clone git@github.com:edgeflare/pgo.git\n\nmake image\n\n# Set KAFKA_CFG_ADVERTISED_LISTENERS env var in docs/docker-compose.yaml to host IP for local access,\n# as opposed to from within container network. adjust Kafka brokers IP in docs/pipeline-example.docker.yaml\nmake up # docker compose up\n```\n\n2. Postgres\n- As a source: Create a test table, eg `users` in source postgres database\n\n```sh\nPGUSER=postgres PGPASSWORD=secret PGHOST=localhost PGDATABASE=testdb psql\n```\n\n```sql\nCREATE TABLE IF NOT EXISTS public.users (\n  id INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,\n  name TEXT\n);\n\nALTER TABLE public.users REPLICA IDENTITY FULL;\n```\n\n- As a sink\n\n```sh\nPGUSER=postgres PGPASSWORD=secret PGHOST=localhost PGDATABASE=testdb PGPORT=5431 psql\n```\n\n- Create the same users table in sink database for mirroring. altering replica identity may not be needed in sink\n\n- Create a second table in sink database which stores **transformed** data \n\n```sql\nCREATE SCHEMA IF NOT EXISTS another_schema;\n\nCREATE TABLE IF NOT EXISTS  another_schema.transformed_users (\n  uuid UUID DEFAULT gen_random_uuid(), -- because we're extracting only `name` field\n  -- new_id INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, -- to handle UPDATE operations, primary key column type must match in source and sink\n  new_name TEXT\n);\n```\n\npgo caches the table schemas for simpler parsing of CDC events (rows). To update pgo cache with newly created tables,\neither `docker restart pgo_pgo_1` or `NOTIFY` it to reload cache by executing on database\n\n```sql\nNOTIFY pgo, 'reload schema';\n```\n\n4. Subscribe\n\n- MQTT: `/any/prefix/schemaName/tableName/operation` topic (testing with mosquitto client)\n\n```sh\nmosquitto_sub -t pgo/public/users/c # operation: c=create, u=update, d=delete, r=read\n```\n\n- Kafka: topic convention is `[prefix].[schema_name].[table_name].[operation]`. use any kafka client eg [`kaf`](https://github.com/birdayz/kaf)\n\n```sh\nkaf consume pgo.public.users.c --follow # consume messages until program execution\n```\n\n- NATS:\n\n```sh\nnats sub -s nats://localhost:4222 'pgo.public.users.\u003e' # wildcard. includes all nested parts\n# nats sub -s nats://localhost:4222 'pgo.public.users.c' # specific\n```\n\n5. `INSERT` (or update etc) into users table\n\n```sql\nINSERT INTO users (name) VALUES ('alice');\nINSERT INTO users (name) VALUES ('bob');\n```\n\nAnd notice NATS, MQTT, Kafka, postgres-sink, or debug peer's respective subscriber receiving the message.\nIt's not Postgres only source. Other peers too can be sources (not all peers fully functional yet).\n\n\nClean up\n\n```sh\nmake down\n```\n\nIt's also possible to import functions, etc around\n- net/http.Handler\n  - router\n  - middleware (authentication, logging, CORS, RequestID, ...)\n  - Postgres middleware attaches a pgxpool.Conn to request context for authorized user; useful for RLS\n\nIf you're curious, start by browsing the [examples](./examples/), skimming over any doc.go, *.md files.\n\n## Contributing\nPlease see [CONTRIBUTING.md](CONTRIBUTING.md).\n\n## License\nApache License 2.0\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedgeflare%2Fpgo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fedgeflare%2Fpgo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedgeflare%2Fpgo/lists"}