{"id":22288913,"url":"https://github.com/marceloneppel/patroni-restart-test","last_synced_at":"2026-02-02T15:03:55.053Z","repository":{"id":61311620,"uuid":"538766476","full_name":"marceloneppel/patroni-restart-test","owner":"marceloneppel","description":"Patroni restart test with Pebble and Supervisord.","archived":false,"fork":false,"pushed_at":"2022-10-12T20:16:20.000Z","size":13,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-08T11:04:05.658Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marceloneppel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-09-20T01:46:15.000Z","updated_at":"2022-09-20T01:46:36.000Z","dependencies_parsed_at":"2022-10-14T20:57:05.441Z","dependency_job_id":null,"html_url":"https://github.com/marceloneppel/patroni-restart-test","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marceloneppel%2Fpatroni-restart-test","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marceloneppel%2Fpatroni-restart-test/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marceloneppel%2Fpatroni-restart-test/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marceloneppel%2Fpatroni-restart-test/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marceloneppel","download_url":"https://codeload.github.com/marceloneppel/patroni-restart-test/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marceloneppel%2Fpatroni-restart-test/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259181934,"owners_count":22818033,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-03T17:07:31.176Z","updated_at":"2026-02-02T15:03:50.003Z","avatar_url":"https://github.com/marceloneppel.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Patroni Restart Test\n\nThis repository aims to show a difference on executing [patroni](https://github.com/zalando/patroni) on a situation where it's started by [pebble](https://github.com/canonical/pebble) and on another situation where it's started by [supervisord](http://supervisord.org).\n\nThe behavior that is shown here is related to killing the Patroni OS process and check whether each of the tools (pebble and supervisord) would correctly restart the process.\n\n## Dependencies\n\nThe command below will install `curl`, `gettext-base` package (in order to use `envsubst`) and `microk8s` snap (you will also need Docker installed in your system).\n```sh\nmake dependencies\n```\n\nDocker should be already installed in your system. Use a test environment to not mess with your microk8s or system.\n\n## Building the images\n\nBuild the docker images (`test-pebble` and `test-supervisord`) using the following command:\n```sh\nmake build\n```\n\n## Testing Patroni with Pebble starting it\n\nFirstly you run the command to deploy the pods with pebble being the entrypoint. This time pebble starts patroni (we run `make clean` first in order to remove the previous deployment):\n\n```sh\nmake clean\nmake pebble\n```\n\nThen, after the pod is deployed, you can login into the container with `microk8s kubectl exec -it patronidemo-0 -n pebble -- bash` and check the running processes:\n\n```\nroot@patronidemo-0:/home/postgres# ps aux\nUSER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND\nroot           1  0.0  0.0 926448 10328 ?        Ssl  01:25   0:00 /usr/bin/pebble run\npostgres      18  0.8  0.2 489700 34648 ?        Sl   01:25   0:00 /usr/bin/python3 /usr/local/bin/patroni /home/postgres/patroni.yml\npostgres      47  0.0  0.1 215572 28052 ?        S    01:25   0:00 postgres -D /home/postgres/pgdata/pgroot/data --config-file=/home/postgres/pgdata/pgroot/data/postgresql.conf --li\npostgres      50  0.0  0.0 215572  4304 ?        Ss   01:25   0:00 postgres: patronidemo: checkpointer   \npostgres      51  0.0  0.0 215572  5772 ?        Ss   01:25   0:00 postgres: patronidemo: background writer   \npostgres      52  0.0  0.0 215572 10004 ?        Ss   01:25   0:00 postgres: patronidemo: walwriter   \npostgres      53  0.0  0.0 216128  6848 ?        Ss   01:25   0:00 postgres: patronidemo: autovacuum launcher   \npostgres      54  0.0  0.0  69936  4928 ?        Ss   01:25   0:00 postgres: patronidemo: stats collector   \npostgres      55  0.0  0.0 216024  6676 ?        Ss   01:25   0:00 postgres: patronidemo: logical replication launcher   \npostgres      59  0.0  0.1 217756 18720 ?        Ss   01:25   0:00 postgres: patronidemo: postgres postgres 127.0.0.1(58156) idle\nroot          66  0.0  0.0   7244  3932 pts/0    Ss   01:25   0:00 bash\nroot          81  0.0  0.0   8896  3248 pts/0    R+   01:26   0:00 ps aux\n```\n\nThe line with `/usr/bin/python3 /usr/local/bin/patroni /home/postgres/patroni.yml` is the Patroni OS process.\n\nYou can then kill the OS process and check that it is removed from the process list.\n\n```\nroot@patronidemo-0:/home/postgres# pkill --signal SIGKILL -f /usr/local/bin/patroni\n\nroot@patronidemo-0:/home/postgres# ps aux\nUSER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND\nroot           1  0.0  0.0 926448 10328 ?        Ssl  01:25   0:00 /usr/bin/pebble run\npostgres      47  0.0  0.1 215572 28052 ?        S    01:25   0:00 postgres -D /home/postgres/pgdata/pgroot/data --config-file=/home/postgres/pgdata/pgroot/data/postgresql.conf --li\npostgres      50  0.0  0.0 215572  4304 ?        Ss   01:25   0:00 postgres: patronidemo: checkpointer   \npostgres      51  0.0  0.0 215572  5772 ?        Ss   01:25   0:00 postgres: patronidemo: background writer   \npostgres      52  0.0  0.0 215572 10004 ?        Ss   01:25   0:00 postgres: patronidemo: walwriter   \npostgres      53  0.0  0.0 216128  6848 ?        Ss   01:25   0:00 postgres: patronidemo: autovacuum launcher   \npostgres      54  0.0  0.0  69936  4928 ?        Ss   01:25   0:00 postgres: patronidemo: stats collector   \npostgres      55  0.0  0.0 216024  6676 ?        Ss   01:25   0:00 postgres: patronidemo: logical replication launcher   \nroot          66  0.0  0.0   7244  3932 pts/0    Ss   01:25   0:00 bash\nroot          87  0.0  0.0   8896  3260 pts/0    R+   01:26   0:00 ps aux\n```\n\nBut even with the `on-check-failure` section in the pebble layer, the service is not restarted.\n\n```\nroot@patronidemo-0:/home/postgres# pebble services\nService  Startup  Current\npatroni  enabled  inactive\n\nroot@patronidemo-0:/home/postgres# pebble checks\nCheck    Level  Status  Failures\npatroni  -      down    4/3\n\nroot@patronidemo-0:/home/postgres# pebble changes\nID   Status  Spawn               Ready               Summary\n1    Done    today at 01:25 UTC  today at 01:25 UTC  Autostart service \"patroni\"\n\nroot@patronidemo-0:/home/postgres# pebble tasks 1\nStatus  Spawn               Ready               Summary\nDone    today at 01:25 UTC  today at 01:25 UTC  Start service \"patroni\"\n\nroot@patronidemo-0:/home/postgres# ps aux\nUSER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND\nroot           1  0.0  0.0 926448 10328 ?        Ssl  01:25   0:00 /usr/bin/pebble run\npostgres      47  0.0  0.1 215572 28052 ?        S    01:25   0:00 postgres -D /home/postgres/pgdata/pgroot/data --config-file=/home/postgres/pgdata/pgroot/data/postgresql.conf --li\npostgres      50  0.0  0.0 215708  8176 ?        Ss   01:25   0:00 postgres: patronidemo: checkpointer   \npostgres      51  0.0  0.0 215572  5772 ?        Ss   01:25   0:00 postgres: patronidemo: background writer   \npostgres      52  0.0  0.0 215572 10004 ?        Ss   01:25   0:00 postgres: patronidemo: walwriter   \npostgres      53  0.0  0.0 216128  8544 ?        Ss   01:25   0:00 postgres: patronidemo: autovacuum launcher   \npostgres      54  0.0  0.0  69936  4928 ?        Ss   01:25   0:00 postgres: patronidemo: stats collector   \npostgres      55  0.0  0.0 216024  6676 ?        Ss   01:25   0:00 postgres: patronidemo: logical replication launcher   \nroot          66  0.0  0.0   7244  3936 pts/0    Ss   01:25   0:00 bash\nroot         139  0.0  0.0   8896  3248 pts/0    R+   01:31   0:00 ps aux\n```\n\nAlso, if we check the pod logs using `microk8s kubectl logs patronidemo-0 -n pebble` we get more and more errors (we also get the messages related to the process not existing):\n\n```\n2022-09-20T01:25:33.392Z [pebble] Started daemon.\n2022-09-20T01:25:33.396Z [pebble] POST /v1/services 3.784058ms 202\n2022-09-20T01:25:33.396Z [pebble] Started default services with change 1.\n2022-09-20T01:25:33.400Z [pebble] Service \"patroni\" starting: /usr/bin/python3 /usr/local/bin/patroni /home/postgres/patroni.yml\n2022-09-20T01:26:33.393Z [pebble] Check \"patroni\" failure 1 (threshold 3): Get http://127.0.0.1:8008/health: dial tcp 127.0.0.1:8008: connect: connection refused\n2022-09-20T01:26:43.393Z [pebble] Check \"patroni\" failure 2 (threshold 3): Get http://127.0.0.1:8008/health: dial tcp 127.0.0.1:8008: connect: connection refused\n2022-09-20T01:26:53.393Z [pebble] Check \"patroni\" failure 3 (threshold 3): Get http://127.0.0.1:8008/health: dial tcp 127.0.0.1:8008: connect: connection refused\n2022-09-20T01:26:53.393Z [pebble] Check \"patroni\" failure threshold 3 hit, triggering action\n2022-09-20T01:26:53.393Z [pebble] Service \"patroni\" on-check-failure action is \"restart\", terminating process before restarting\n2022-09-20T01:26:53.393Z [pebble] Cannot send SIGTERM to process: no such process\n2022-09-20T01:26:58.394Z [pebble] Cannot send SIGKILL to process: no such process\n2022-09-20T01:27:01.037Z [pebble] GET /v1/services?names= 134.38µs 200\n2022-09-20T01:27:03.393Z [pebble] Check \"patroni\" failure 4 (threshold 3): Get http://127.0.0.1:8008/health: dial tcp 127.0.0.1:8008: connect: connection refused\n2022-09-20T01:27:03.394Z [pebble] Service \"patroni\" still running after SIGTERM and SIGKILL\n2022-09-20T01:27:06.262Z [pebble] GET /v1/checks 172.91µs 200\n2022-09-20T01:27:11.650Z [pebble] GET /v1/changes?select=all 253.188µs 200\n2022-09-20T01:27:13.393Z [pebble] Check \"patroni\" failure 5 (threshold 3): Get http://127.0.0.1:8008/health: dial tcp 127.0.0.1:8008: connect: connection refused\n2022-09-20T01:27:23.393Z [pebble] Check \"patroni\" failure 6 (threshold 3): Get http://127.0.0.1:8008/health: dial tcp 127.0.0.1:8008: connect: connection refused\n2022-09-20T01:27:29.564Z [pebble] GET /v1/logs?n=30 378.396µs 200\n2022-09-20T01:27:33.393Z [pebble] Check \"patroni\" failure 7 (threshold 3): Get http://127.0.0.1:8008/health: dial tcp 127.0.0.1:8008: connect: connection refused\n2022-09-20T01:27:37.962Z [pebble] GET /v1/warnings 107.777µs 200\n2022-09-20T01:27:43.393Z [pebble] Check \"patroni\" failure 8 (threshold 3): Get http://127.0.0.1:8008/health: dial tcp 127.0.0.1:8008: connect: connection refused\n```\n\n## Testing Patroni with Supervisord starting it\n\nFirstly you run the command to deploy the pods with supervisord being the entrypoint. This time supervisord starts patroni (we run `make clean` first in order to remove the previous deployment):\n\n```sh\nmake clean\nmake supervisord\n```\n\nThen, after the pod is deployed, you can login into the container with `microk8s kubectl exec -it patronidemo-0 -n supervisord -- bash` and check the running processes:\n\n```\nroot@patronidemo-0:/home/postgres# ps aux\nUSER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND\nroot           1  0.1  0.1  31300 24020 ?        Ss   01:39   0:00 /usr/bin/python3 /usr/bin/supervisord\npostgres       7  0.3  0.2 489700 34768 ?        Sl   01:39   0:00 /usr/bin/python3 /usr/local/bin/patroni /home/postgres/patroni.yml\npostgres      35  0.0  0.1 215572 28080 ?        S    01:39   0:00 postgres -D /home/postgres/pgdata/pgroot/data --config-file=/home/postgres/pgdata/pgroot/data/postgresql.conf --li\npostgres      38  0.0  0.0 215572  4204 ?        Ss   01:39   0:00 postgres: patronidemo: checkpointer   \npostgres      39  0.0  0.0 215572  5680 ?        Ss   01:39   0:00 postgres: patronidemo: background writer   \npostgres      40  0.0  0.0 215572  9900 ?        Ss   01:39   0:00 postgres: patronidemo: walwriter   \npostgres      41  0.0  0.0 216128  8392 ?        Ss   01:39   0:00 postgres: patronidemo: autovacuum launcher   \npostgres      42  0.0  0.0  69936  4912 ?        Ss   01:39   0:00 postgres: patronidemo: stats collector   \npostgres      43  0.0  0.0 216024  6636 ?        Ss   01:39   0:00 postgres: patronidemo: logical replication launcher   \npostgres      47  0.0  0.1 217532 18572 ?        Ss   01:39   0:00 postgres: patronidemo: postgres postgres 127.0.0.1(44912) idle\nroot          61  0.3  0.0   7244  3928 pts/0    Ss   01:40   0:00 bash\nroot          70  0.0  0.0   8896  3344 pts/0    R+   01:40   0:00 ps aux\n```\n\nThe line with `/usr/bin/python3 /usr/local/bin/patroni /home/postgres/patroni.yml` is the Patroni OS process.\n\nYou can then kill the OS process and check that it is removed from the process list (and a new one was created - the restart process). Ignore the old PostgreSQL process in the process list.\n\n```\nroot@patronidemo-0:/home/postgres# pkill --signal SIGKILL -f /usr/local/bin/patroni\n\nroot@patronidemo-0:/home/postgres# ps aux\nUSER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND\nroot           1  0.1  0.1  31300 24036 ?        Ss   01:39   0:00 /usr/bin/python3 /usr/bin/supervisord\npostgres      35  0.0  0.1 215572 28080 ?        S    01:39   0:00 postgres -D /home/postgres/pgdata/pgroot/data --config-file=/home/postgres/pgdata/pgroot/data/postgresql.conf --li\npostgres      38  0.0  0.0 215572  4204 ?        Ss   01:39   0:00 postgres: patronidemo: checkpointer   \npostgres      39  0.0  0.0 215572  5680 ?        Ss   01:39   0:00 postgres: patronidemo: background writer   \npostgres      40  0.0  0.0 215572  9900 ?        Ss   01:39   0:00 postgres: patronidemo: walwriter   \npostgres      41  0.0  0.0 216128  8392 ?        Ss   01:39   0:00 postgres: patronidemo: autovacuum launcher   \npostgres      42  0.0  0.0  69936  4912 ?        Ss   01:39   0:00 postgres: patronidemo: stats collector   \npostgres      43  0.0  0.0 216024  6636 ?        Ss   01:39   0:00 postgres: patronidemo: logical replication launcher   \nroot          61  0.0  0.0   7244  3928 pts/0    Ss   01:40   0:00 bash\npostgres      79 19.0  0.2 415344 33524 ?        Sl   01:41   0:00 /usr/bin/python3 /usr/local/bin/patroni /home/postgres/patroni.yml\npostgres      85  0.0  0.0 216800 14948 ?        Ss   01:41   0:00 postgres: patronidemo: postgres postgres 127.0.0.1(37688) idle\nroot          90  0.0  0.0   8896  3312 pts/0    R+   01:41   0:00 ps aux\n```\n\nDifferently from pebble, supervisord correctly restarts the service after it was killed due to other reasons.\n\n## Known issues\n\nIf you see a message like the one below when running `make pebble` or `make supervisord`, just run `make clean` and rerun the original command.\n\n```\nWarning: resource endpoints/patronidemo is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.\n```\n\nAlso, we are using `host replication standby 0.0.0.0/0 md5` in the Postgres authentication rules just to make this example work the way we need. It's more secure to use the pod's IP.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarceloneppel%2Fpatroni-restart-test","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarceloneppel%2Fpatroni-restart-test","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarceloneppel%2Fpatroni-restart-test/lists"}