{"id":13644485,"url":"https://github.com/gocardless/stolon-pgbouncer","last_synced_at":"2025-03-04T09:10:02.969Z","repository":{"id":34608918,"uuid":"178033549","full_name":"gocardless/stolon-pgbouncer","owner":"gocardless","description":"Add-on to stolon for providing zero-downtime failover and PgBouncer integration","archived":false,"fork":false,"pushed_at":"2024-08-29T19:29:20.000Z","size":6435,"stargazers_count":64,"open_issues_count":14,"forks_count":12,"subscribers_count":63,"default_branch":"master","last_synced_at":"2025-02-25T08:13:20.220Z","etag":null,"topics":["archive-no","pgbouncer","postgresql","stolon","team-core-infrastructure","tier-1"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gocardless.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-27T16:28:02.000Z","updated_at":"2025-01-15T10:51:14.000Z","dependencies_parsed_at":"2024-01-14T09:22:02.566Z","dependency_job_id":"18b3ce36-ac4d-478d-84be-ac913c49201c","html_url":"https://github.com/gocardless/stolon-pgbouncer","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fstolon-pgbouncer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fstolon-pgbouncer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fstolon-pgbouncer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fstolon-pgbouncer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gocardless","download_url":"https://codeload.github.com/gocardless/stolon-pgbouncer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241818901,"owners_count":20025210,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archive-no","pgbouncer","postgresql","stolon","team-core-infrastructure","tier-1"],"created_at":"2024-08-02T01:02:05.814Z","updated_at":"2025-03-04T09:10:02.931Z","avatar_url":"https://github.com/gocardless.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# stolon-pgbouncer [![CircleCI](https://circleci.com/gh/gocardless/stolon-pgbouncer/tree/master.svg?style=svg)](https://circleci.com/gh/gocardless/stolon-pgbouncer/tree/master)\n\nstolon-pgbouncer extends a [stolon](https://github.com/sorintlab/stolon)\nPostgreSQL setup with PgBouncer connection pooling and zero-downtime planned\nfailover of the PostgreSQL primary.\n\nSee [Playground](#playground) for how to start a Dockerised three node stolon\nPostgreSQL cluster utilising stolon-pgbouncer.\n\n- [Overview](#overview)\n  - [Stolon Recap](#stolon-recap)\n  - [Playground](#playground)\n  - [Node Roles](#node-roles)\n    - [Postgres](#postgres)\n    - [Proxy](#proxy)\n  - [Zero-Downtime Failover](#zero-downtime-failover)\n- [Development](#development)\n  - [Testing](#testing)\n  - [Images](#images)\n  - [Releasing](#releasing)\n\n## Overview\n\n[stolon](https://github.com/sorintlab/stolon) is a tool for running highly\navailable Postgres clusters. stolon aims to recover from node failures and\nensure data durability across your cluster.\n\nstolon-pgbouncer extends stolon with first class support for PgBouncer, a\nPostgres connection pooler. By introducing PgBouncer, it's possible to offer\nzero-downtime planned failovers of Postgres primaries, allowing users to perform\nmaintenance operations without taking downtime.\n\nLive information about cluster health is maintained by stolon in a consistent\ndata store such as etcd. stolon-pgbouncer runs two services that use this data\nto work with the cluster:\n\n- `supervise` manages PgBouncer processes to proxy connections to the currently\n  elected Postgres primary\n- `pauser` exposes an API that can perform zero-downtime failover by pausing\n  PgBouncer traffic\n\nBoth these services are commands on the `stolon-pgbouncer` binary, with a third\ncommand called `failover` which speaks with the pauser API.\n\n### Stolon Recap\n\nThis README assumes familiarity with stolon and associated tooling that can be\nacquired by reading the [stolon docs](https://github.com/sorintlab/stolon/blob/master/doc/architecture.md).\nWhile we advise you read these first, we'll summarise each stolon component for\nconvenience here:\n\n- `keeper` supervises, configures, and converges PostgreSQL on each PostgreSQL\n  node according to the clusterview\n- `sentinel` discovers and monitors the keepers, and calculates the optimal\n  clusterview\n- `proxy` ensures connections are pointing to the master PostgreSQL node and fences\n  (forcibly closes connections) to unelected masters\n\nWe use these terms throughout this README, and encourage referring to the stolon\ndocs whenever anything is unclear.\n\n### Playground\n\nWe have created a Dockerised sandbox environment that boots a three node\nPostgres cluster with the stolon-pgbouncer services installed, using etcd as our\nconsistent stolon store. We recommend playing around in this environment to\ndevelop an understanding of how this setup works and to simulate failure\nsituations (network partitions, node crashes, etc).\n\n**It also helps to have this playground running while reading through the\nREADME, in order to try out the commands you see along the way.**\n\nFirst install [Docker](https://docker.io/) and Golang \u003e=1.12, then run:\n\n```\n# Clone into your GOPATH\n$ git clone https://github.com/gocardless/stolon-pgbouncer\n$ cd stolon-pgbouncer\n$ make docker-compose\n...\n\n# List all docker-compose services\n$ docker-compose ps\n    Name                Command                         Ports\n------------------------------------------------------------------------------------\netcd-store_1   etcd --data-dir=/data --li ...   0.0.0.0:2379-\u003e2379, 2380\nkeeper0_1      supervisord -n -c /stolon- ...   5432, 0.0.0.0:6433-\u003e6432, 7432, 8080\nkeeper1_1      supervisord -n -c /stolon- ...   5432, 0.0.0.0:6434-\u003e6432, 7432, 8080\nkeeper2_1      supervisord -n -c /stolon- ...   5432, 0.0.0.0:6435-\u003e6432, 7432, 8080\npgbouncer_1    /stolon-pgbouncer/bin/stol ...   5432, 0.0.0.0:6432-\u003e6432, 7432, 8080\nsentinel_1     /usr/local/bin/stolon-sent ...   5432, 6432, 7432, 8080\n\n# Query clusterview for status\n$ docker exec stolon-pgbouncer_pgbouncer_1 stolonctl status\n=== Keepers ===\n\nUID     HEALTHY PG LISTENADDRESS        PG HEALTHY\nkeeper0 true    172.24.0.4:5432         true\nkeeper1 true    172.24.0.5:5432         true\nkeeper2 true    172.24.0.6:5432         true\n\n...\n```\n\n### Node Roles\n\nIn a stolon-pgbouncer cluster, you will typically run two types of nodes: the\nPostgres nodes where we run the keeper/Postgres/PgBouncer, and the proxy nodes\nthat run a supervised PgBouncer that provides connectivity to our cluster (this\nis what applications will connect via).\n\n![Playground architecture](resources/playground-architecture.svg)\n\nIn our playground setup we run a single proxy node (called pgbouncer in our\ndocker-compose) and three Postgres nodes (`keeper0`, `keeper1`, `keeper2`)\nwhich- in addition to the keeper and Postgres- run the stolon-pgbouncer pauser.\n\n#### Postgres\n\nThe Postgres node role is provisioned to run the stolon keeper (and therefore\nPostgres) and proxy on the same machines, exposing our Postgres service via a\nPgBouncer. Incoming database connections should only ever arrive via the\nPgBouncer service, which in turn will point at the host-local proxy.\n\nWe leverage stolon's fencing by directing connections through the proxy, which\nwill terminate clients in the case of failover. PgBouncer is placed in front of\nour proxy to provide pausing for planned failover, as existing client\nconnections need to be paused before we move the Postgres primary.\n\nThe intention is for all cluster connections to be routed to just one PgBouncer\nat any one time, and for that PgBouncer to be co-located with our primary to\navoid unnecessary network hops. While you could connect via any of the keeper\nnode PgBouncers, our stolon-pgbouncer `supervise` processes will ensure we\nconverge on the primary.\n\n#### Proxy\n\nProxy nodes can be run separately from our Postgres cluster, ideally close to\nwherever the application that uses Postgres is located. These nodes run\nstolon-pgbouncers `supervise` service which manages a PgBouncer to point at the\ncurrent primary. Our aim is to have applications connect to our PgBouncer\nservice and be routed to the PgBouncer that exists on the Postgres nodes.\n\nTo do this, we provision proxy nodes with PgBouncer and a templatable\nconfiguration file that looks like this:\n\n```ini\n# /etc/pgbouncer/pgbouncer.ini.template\n[databases]\npostgres = host={{.Host}} port=6432\n```\n\nWhenever the clusterview (managed by our stolon sentinels) changes, the\nstolon-pgbouncer supervise process will respond by templating our\n`pgbouncer.ini` config with the IP address of our elected primary. Application\nconnects will be re-routed to the current primary, where we expect them to\nconnect to PgBouncer (port 6432).\n\n### Zero-Downtime Failover\n\nstolon-pgbouncer provides ability to failover cluster nodes without\nimpacting traffic. We do this by exposing an API on the Postgres nodes that can\npause database connections before instructing stolon to elect a new node as the\ncluster primary.\n\nThis API is served by the `supervise` service, which should run on all the\nPostgres nodes participating in the cluster. It's important to note that this\nflow is only supported when all database clients are using PgBouncer transaction\npools in order to support pausing connections. Any clients that use session\npools will need to be turned off for the duration of the failover.\n\nThe failover process is as follows:\n\n1. Confirm cluster is healthy and can survive a node failure\n1. Acquire lock in etcd (ensuring only one failover takes place at a time)\n1. Pause all PgBouncer pools on Postgres nodes\n1. Mark primary keeper as unhealthy\n1. Once stolon has elected a new primary, resume PgBouncer pools\n1. Release etcd lock\n\nThis flow is encoded in the [`Run`](pkg/failover/failover.go) method,\nand looks like this:\n\n```go\nPipeline(\n  Step(f.CheckClusterHealthy),\n  Step(f.HealthCheckClients),\n  Step(f.AcquireLock).Defer(f.ReleaseLock),\n  Step(f.Pause).Defer(f.Resume),\n  Step(f.Failkeeper),\n)\n```\n\nOnce the new primary is ready, our Proxy nodes running stolon-pgbouncer's\n`supervise` will template a new PgBouncer configuration that points at the new\nmaster. Connections will resume their operation unaware that they now speak to a\ndifferent Postgres server than before.\n\nRunning the failover within the playground environment looks like this:\n\n```\nts=31 event=metrics.listen address=127.0.0.1 port=9446\nts=31 event=client_dial client=\"keeper2 (172.27.0.4)\"\nts=31 event=client_dial client=\"keeper1 (172.27.0.6)\"\nts=31 event=client_dial client=\"keeper0 (172.27.0.5)\"\nts=31 event=setting_pauser_token\nts=31 event=check_cluster_healthy msg=\"checking health of cluster\"\nts=31 event=clients_health_check msg=\"health checking all clients\"\nts=31 event=etcd_lock_acquire msg=\"acquiring failover lock in etcd\"\nts=31 event=pgbouncer_pause msg=\"requesting all pgbouncers pause\"\nts=31 event=pgbouncer_pause endpoint=keeper0 elapsed=0.0023349\nts=31 event=pgbouncer_pause endpoint=keeper2 elapsed=0.0095867\nts=31 event=pgbouncer_pause endpoint=keeper1 elapsed=0.0116491\nts=31 key=stolon/cluster/main/clusterdata msg=\"waiting for stolon to report master change\"\nts=31 keys=stolon/cluster/main/clusterdata event=watch.start\nts=31 keys=stolon/cluster/main/clusterdata event=poll.start\nts=31 key=stolon/cluster/main/clusterdata event=pending_failover master=\"keeper2 (172.27.0.4)\" msg=\"master has not changed nodes\"\nts=36 keys=stolon/cluster/main/clusterdata event=poll.start\nts=36 key=stolon/cluster/main/clusterdata event=insufficient_standbys healthy=0 minimum=1 msg=\"do not have enough healthy standbys to satisfy the minSynchronousStandbys\"\nts=41 keys=stolon/cluster/main/clusterdata event=poll.start\nts=41 key=stolon/cluster/main/clusterdata master=\"keeper0 (172.27.0.5)\" msg=\"master is available for writes\"\nts=41 msg=\"cluster successfully recovered\" master=\"keeper0 (172.27.0.5)\"\nts=41 event=pgbouncer_resume msg=\"requesting all pgbouncers resume\"\nts=41 event=pgbouncer_resume endpoint=keeper1 elapsed=0.0029219\nts=41 event=pgbouncer_resume endpoint=keeper0 elapsed=0.00493\nts=41 event=pgbouncer_resume endpoint=keeper2 elapsed=0.0124522\nts=41 event=etcd_lock_release msg=\"releasing failover lock in etcd\"\nts=41 event=shutdown\n```\n\nThis flow is subject to several timeouts that need configuring to suit your\nproduction environment. Pause expiry is notable as it needs pairing with load\nbalancer timeouts to ensure you don't drop requests. See the stolon-pgbouncer\n`--help` for more details.\n\n## Development\n\n### Testing\n\nstolon-pgbouncer uses [ginkgo](https://github.com/onsi/ginkgo) and\n[gomega](https://onsi.github.io/gomega/) for testing. Tests are grouped into\nthree categories:\n\n- unit, co-located with the Go package they target, relying on no external\n  dependencies (you could run these tests in a scratch container with no\n  external tools and they should succeed)\n- integration, placed within an `integration` folder inside the Go package\n  directory they target. Integration tests can assume access to an external\n  Postgres database along with PgBouncer and etcd binaries and will directly\n  boot and manage these dependencies\n- acceptance, written as a standalone binary build from\n  `cmd/stolon-pgbouncer-acceptance/main.go`. This environment assumes you have\n  booted the docker-compose playground\n\nFor those developing stolon-pgbouncer, we advise configuring your dev machine to\nbe suitable for the integration environment and testing via `ginkgo -r`. All\ntests are run in CI as a final check before merge: refer to the\n[`circle.yml`](circle.yml) file as a complete reference for a test environment.\n\n### Images\n\nWe use several docker images to power our CI and development environments. See\nthe [README](docker) to understand what each image is for.\n\nEach image can be built and published using a Makefile target, and we generate\ntags as `YYYYMMDDXX` where `XX` is an index into the current day. An example of\npublishing a new base image is:\n\n```\n$ make publish-base\n```\n\n### Releasing\n\nWe use [goreleaser](https://github.com/goreleaser/goreleaser) to create\nreleases and publish docker images. Just update the [`VERSION`](VERSION) file\nwith the new version and push to master.\n\nOur versioning system follows [semver guidelines](https://semver.org/) and care\nshould be taken to adhere to these rules.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgocardless%2Fstolon-pgbouncer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgocardless%2Fstolon-pgbouncer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgocardless%2Fstolon-pgbouncer/lists"}