{"id":14470106,"url":"https://github.com/xataio/pgstream","last_synced_at":"2026-02-16T13:13:13.182Z","repository":{"id":246803630,"uuid":"784740560","full_name":"xataio/pgstream","owner":"xataio","description":"PostgreSQL replication with DDL changes","archived":false,"fork":false,"pushed_at":"2025-06-18T15:07:07.000Z","size":37263,"stargazers_count":586,"open_issues_count":27,"forks_count":21,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-06-18T16:19:53.864Z","etag":null,"topics":["ddl","elasticsearch","golang","opensearch","postgresql","replication","schema","webhooks"],"latest_commit_sha":null,"homepage":"http://www.xata.io","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xataio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-04-10T13:14:14.000Z","updated_at":"2025-06-17T21:13:48.000Z","dependencies_parsed_at":"2024-08-09T08:29:37.563Z","dependency_job_id":"79483873-ac4d-4609-ac96-cb2e8611e634","html_url":"https://github.com/xataio/pgstream","commit_stats":null,"previous_names":["xataio/pgstream"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/xataio/pgstream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xataio%2Fpgstream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xataio%2Fpgstream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xataio%2Fpgstream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xataio%2Fpgstream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xataio","download_url":"https://codeload.github.com/xataio/pgstream/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xataio%2Fpgstream/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261466638,"owners_count":23162558,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ddl","elasticsearch","golang","opensearch","postgresql","replication","schema","webhooks"],"created_at":"2024-09-02T04:01:08.673Z","updated_at":"2026-02-16T13:13:13.176Z","avatar_url":"https://github.com/xataio.png","language":"Go","funding_links":[],"categories":["Go","golang"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"brand-kit/banner/pgstream-banner@2x.png\" alt=\"pgstream logo\" /\u003e\n\u003c/div\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/xataio/pgstream/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-Apache_2.0-green\" alt=\"License - Apache 2.0\"\u003e\u003c/a\u003e\u0026nbsp;\n  \u003ca href=\"https://github.com/xataio/pgstream/actions?query=branch%3Amain\"\u003e\u003cimg src=\"https://github.com/xataio/pgstream/actions/workflows/build.yml/badge.svg\" alt=\"CI Build\"\u003e\u003c/a\u003e \u0026nbsp;\n  \u003ca href=\"https://pkg.go.dev/github.com/xataio/pgstream\"\u003e\u003cimg src=\"https://pkg.go.dev/badge/github.com/xataio/pgstream.svg\" alt=\"Go Reference\"\u003e\u003c/a\u003e\u0026nbsp;\n  \u003ca href=\"https://github.com/xataio/pgstream/releases\"\u003e\u003cimg src=\"https://img.shields.io/github/release/xataio/pgstream.svg?label=Release\" alt=\"Release\"\u003e\u003c/a\u003e \u0026nbsp;\n  \u003ca href=\"https://somsubhra.github.io/github-release-stats/?username=xataio\u0026repository=pgstream\u0026page=1\u0026per_page=5\"\u003e\u003cimg src=\"https://img.shields.io/github/downloads/xataio/pgstream/total\" alt=\"Downloads\"\u003e\u003c/a\u003e \u0026nbsp;\n  \u003ca href=\"https://goreportcard.com/report/github.com/xataio/pgstream\"\u003e\u003cimg src=\"https://goreportcard.com/badge/github.com/xataio/pgstream\" alt=\"Go Report Card\"\u003e\u003c/a\u003e \u0026nbsp;\n  \u003ca href=\"https://xata.io/discord\"\u003e\u003cimg src=\"https://img.shields.io/discord/996791218879086662?label=Discord\u0026logo=discord\" alt=\"Discord\"\u003e\u003c/a\u003e \u0026nbsp;\n  \u003ca href=\"https://twitter.com/xata\"\u003e\u003cimg src=\"https://img.shields.io/badge/@xata-6c47ff?label=Follow\u0026logo=x\" alt=\"X (formerly Twitter) Follow\" /\u003e \u003c/a\u003e\n  \u003ca href=\"https://bsky.app/profile/xata.io\"\u003e\u003cimg src=\"https://img.shields.io/badge/@xata-6c47ff?label=Follow\u0026logo=bluesky\" alt=\"Bluesky Follow\" /\u003e \u003c/a\u003e\n  \u003ca href=\"https://www.youtube.com/@xataio\"\u003e\u003cimg src=\"https://img.shields.io/badge/@xataio-6c47ff?label=Youtube\u0026logo=youtube\" alt=\"Youtube Subscribe\" /\u003e \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://xata.io\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Made%20with%20♥%20%20-%20%20by%20Xata-6c47ff?style=flat\u0026logo=postgresql\u0026logoColor=white\" alt=\"Made with ♥ by Xata\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n# pgstream - Postgres replication with DDL changes\n\n`pgstream` is an open source CDC command-line tool and library that offers Postgres replication support with DDL changes to any provided target.\n\n![pg2pg demo with transformers](https://github.com/user-attachments/assets/6f11b326-d8ed-44eb-b743-756910b9fedd)\n\n## Features\n\n- Schema change tracking and replication of DDL changes\n- Support for multiple out of the box targets\n  - Elasticsearch/OpenSearch\n  - Webhooks\n  - PostgreSQL\n- Initial and on demand PostgreSQL snapshots (for when you don't need continuous replication)\n- Column value transformations (anonymise your data on the go!)\n- Modular deployment configuration, only requires Postgres\n- Kafka support with schema based partitioning\n- Extendable support for custom targets\n\n## Table of Contents\n\n- [Usage](#usage)\n- [Tutorials](#tutorials)\n- [Documentation](#documentation)\n- [Benchmarks](#benchmarks)\n- [Limitations](#limitations)\n- [Releases](#releases)\n- [Contributing](#contributing)\n- [License](#license)\n- [Support](#support)\n\n## Usage\n\n`pgstream` can be used via the readily available CLI or as a library. For detailed information about the CLI usage, check out the dedicated [CLI documentation section](docs/cli.md).\n\n### CLI Installation\n\n```bash\n# Download the latest release\ncurl -L https://github.com/xataio/pgstream/releases/latest/download/pgstream.linux.amd64 -o pgstream\nchmod +x pgstream\nsudo mv pgstream /usr/local/bin/\n\n# Or use go install\ngo install github.com/xataio/pgstream@latest\n\n# Or build from source\ngit clone https://github.com/xataio/pgstream.git\ncd pgstream\ngo build -o pgstream ./cmd\n\n# Or install via homebrew on macOS or Linux\nbrew tap xataio/pgstream\nbrew install pgstream\n```\n\n### Environment setup\n\nIf you have an environment available, with at least Postgres and whichever resources you're planning on running, then you can skip this step. Otherwise, a docker setup is available in this repository with profiles that selectively start Postgres, Kafka and OpenSearch.\n\nTo run all profiles:\n\n```\ndocker-compose -f build/docker/docker-compose.yml up\n```\n\nIf you only want to run PostgreSQL to PostgreSQL pgstream replication you can use the `pg2pg` profile:\n\n```\ndocker-compose -f build/docker/docker-compose.yml --profile pg2pg up\n```\n\nYou can also run multiple profiles. For example to start two PostgreSQL instances and Kafka:\n\n```\ndocker-compose -f build/docker/docker-compose.yml --profile pg2pg --profile kafka up\n```\n\nList of supported docker profiles:\n\n- pg2pg\n- pg2os\n- pg2webhook\n- kafka\n\n### Configuration\n\nPgstream source and target need to be configured appropriately before the commands can be run. This can be done:\n\n- Using the relevant CLI flags for each command\n- Using a yaml configuration file\n- Using environment variables (.env file supported)\n\nCheck the [configuration documentation](docs/configuration.md) for more information about the configuration options, or check the [CLI documentation](docs/cli.md) for details on the available flags. Additionally, you can find sample files for both .env and .yaml [here](docs/examples).\n\nIf you want to configure column transformations, leveraging [greenmask](https://github.com/GreenmaskIO/greenmask), [neosync](https://github.com/nucleuscloud/neosync) and [go-masker](https://github.com/ggwhite/go-masker) open source integrations, as well as custom transformers, check the [transformation rules](docs/transformers.md#transformation-rules) configuration for more details, along with the list of [available transformers](docs/transformers.md#supported-transformers).\n\n### Run `pgstream`\n\n#### Replication mode\n\nRun will start streaming data from the configured source into the configured target. By passing the `--init` flag to the run command, pgstream will initialise the pgstream state in the source Postgres database before starting replication. It will:\n\n- Create a `pgstream` schema\n- Create tables/functions/triggers to keep track of schema changes for DDL replication (see [Tracking schema changes](docs/architecture.md#tracking-schema-changes) for more details)\n- Create a replication slot\n\nInitialisation is required for pgstream replication. It can alternatively be performed by running the `pgstream init` command separately before `pgstream run`. Check out the [CLI documentation](docs/cli.md#init) for more details.\n\nExample running pgstream replication from Postgres -\u003e OpenSearch:\n\n```sh\n# using the environment configuration file\npgstream run -c docs/examples/pg2os.env --init --log-level trace\n# using the yaml configuration file\npgstream run -c docs/examples/pg2os.yaml --init --log-level info\n# using the CLI flags\npgstream run --source postgres --source-url \"postgres://postgres:postgres@localhost:5432?sslmode=disable\" --target opensearch --target-url \"http://admin:admin@localhost:9200\" --init\n```\n\nExample running pgstream with Postgres -\u003e Kafka, and in a separate terminal, Kafka-\u003eOpenSearch:\n\n```sh\n# using the environment configuration file\npgstream run -c docs/examples/pg2kafka.env --init --log-level trace\n# using the yaml configuration file\npgstream run -c docs/examples/pg2kafka.yaml --init --log-level info\n# using the CLI flags\npgstream run --source postgres --source-url \"postgres://postgres:postgres@localhost:5432?sslmode=disable\" --target kafka --target-url \"localhost:9092\" --init\n```\n\n```sh\n# using the environment configuration file\npgstream run -c docs/examples/kafka2os.env --init --log-level trace\n# using the yaml configuration file\npgstream run -c docs/examples/kafka2os.yaml --init --log-level info\n# using the CLI flags\npgstream run --source kafka --source-url \"localhost:9092\" --target opensearch --target-url \"http://admin:admin@localhost:9200\" --init\n```\n\nAn initial snapshot can be performed before starting replication by providing `--snapshot-tables` flag or by setting the relevant configuration fields (check the [configuration documentation](docs/configuration.md) for more details on advanced configuration options).\n\nExample running pgstream with PostgreSQL -\u003e PostgreSQL with initial snapshot enabled:\n\n```sh\n# using the CLI flags\npgstream run --source postgres --source-url \"postgres://postgres:postgres@localhost:5432?sslmode=disable\" --target postgres --target-url \"postgres://postgres:postgres@localhost:7654?sslmode=disable\" --snapshot-tables test --init\n```\n\n#### Snapshot mode\n\npgstream can also be used to perform a point in time snapshot of the source database. This is helpful if you don't require continuous replication, but want to keep the source and target in sync by running nightly snapshots for example.\n\nThe `snapshot` command doesn't require any initialisation or pgstream specific state, since it only performs read operations on the source Postgres database.\n\nExample running pgstream to perform a snapshot from PostgreSQL -\u003e PostgreSQL:\n\n```sh\n# using the environment configuration file\npgstream snapshot -c docs/examples/snapshot2pg.env --log-level trace\n# using the yaml configuration file\npgstream snapshot -c docs/examples/snapshot2pg.yaml --log-level info\n# using the CLI flags\npgstream snapshot --postgres-url=\"postgres://postgres:postgres@localhost:5432?sslmode=disable\" --target=postgres --target-url=\"postgres://postgres:postgres@localhost:7654?sslmode=disable\" --tables=\"test\" --reset\n```\n\n## Tutorials\n\n- [PostgreSQL replication to PostgreSQL](docs/tutorials/postgres_to_postgres.md)\n- [PostgreSQL replication to OpenSearch](docs/tutorials/postgres_to_opensearch.md)\n- [PostgreSQL replication to webhooks](docs/tutorials/postgres_to_webhooks.md)\n- [PostgreSQL replication using Kafka](docs/tutorials/postgres_kafka.md)\n- [PostgreSQL snapshots](docs/tutorials/postgres_snapshot.md)\n- [PostgreSQL column transformations](docs/tutorials/postgres_transformer.md)\n\n## Documentation\n\nFor more advanced usage, implementation details, and detailed configuration settings, please refer to the full documentation below.\n\n1. [Architecture](docs/architecture.md)\n2. [Configuration](docs/configuration.md)\n   - [Yaml](docs/configuration.md#yaml)\n   - [Environment Variables](docs/configuration.md#environment-variables)\n   - [Examples](docs/examples/)\n3. [Snapshots](docs/snapshots.md)\n4. [Transformers](docs/transformers.md)\n   - [Supported Transformers](docs/transformers.md#supported-transformers)\n   - [Transformation Rules](docs/transformers.md#transformation-rules)\n5. [Observability](docs/observability.md)\n6. [CLI](docs/cli.md)\n7. [Privileges](docs/privileges.md)\n   - [Xata](docs/xata.md)\n   - [AWS](docs/aws.md)\n   - [GCP CloudSQL](docs/gcp_cloudsql.md)\n   - [Neon](docs/neon.md)\n8. [Glossary](docs/glossary.md)\n\n## Benchmarks\n\n### Snapshots\n\n\u003cp align=\"left\"\u003e\n    \u003cimg src=\"docs/img/pgstream_snapshot_benchmarks.png\" alt=\"pgstream snapshot benchmarks\" width=\"700\"\u003e\n\u003c/p\u003e\n\nDatasets used: [IMDB database](https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/2QYZBT), [MusicBrainz database](https://musicbrainz.org/doc/MusicBrainz_Database), [Firenibble database](https://github.com/PeerDB-io/ab-scale-testing/tree/main).\n\nAll benchmarks were run using the same setup, with pgstream `v0.7.2`, pg_dump/pg_restore (PostgreSQL) 17.4 and PostgreSQL 17.4, using identical resources to ensure a fair comparison.\n\nFor more details into performance benchmarking for snapshots to PostgreSQL with `pgstream`, check out this [blogpost](https://xata.io/blog/behind-the-scenes-speeding-up-pgstream-snapshots-for-postgresql).\n\n## Limitations\n\nSome of the limitations of the initial release include:\n\n- Single Kafka topic support\n- Postgres plugin support limited to `wal2json`\n- No row level filtering support\n- Primary key/unique not null column required for replication\n- Kafka serialisation support limited to JSON\n\n## Releases\n\nYou can find below the release notes for major pgstream versions, along with migration guides whenever relevant:\n\n- [v1.0.0](docs/releases/RELEASE_NOTES_v1.md)\n\n## Contributing\n\nWe welcome contributions from the community! If you'd like to contribute to pgstream, please follow [these guidelines](https://github.com/xataio/pgstream/tree/main?tab=contributing-ov-file) and adhere to our [code of conduct](https://github.com/xataio/pgstream?tab=coc-ov-file).\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.\n\n## Support\n\nIf you have any questions, encounter issues, or need assistance, open an issue in this repository our join our [Discord](https://xata.io/discord), and our community will be happy to help.\n\n\u003cbr\u003e\n\u003cp align=\"right\"\u003eMade with 💜 by \u003ca href=\"https://xata.io\"\u003eXata 🦋\u003c/a\u003e\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxataio%2Fpgstream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxataio%2Fpgstream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxataio%2Fpgstream/lists"}