{"id":21327537,"url":"https://github.com/jille/etcd-postgresql-sync","last_synced_at":"2025-10-23T17:50:21.815Z","repository":{"id":64305613,"uuid":"448589673","full_name":"Jille/etcd-postgresql-sync","owner":"Jille","description":"A simple tool to sync your etcd cluster to PostgreSQL in realtime.","archived":false,"fork":false,"pushed_at":"2023-07-05T21:05:23.000Z","size":139,"stargazers_count":6,"open_issues_count":3,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-14T23:42:04.471Z","etag":null,"topics":["etcd","etcd-backup","postgresql"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Jille.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-16T15:12:47.000Z","updated_at":"2025-03-18T07:13:42.000Z","dependencies_parsed_at":"2024-06-20T03:16:01.842Z","dependency_job_id":"cfb800b6-b93e-45cb-be0c-2fcca9f47422","html_url":"https://github.com/Jille/etcd-postgresql-sync","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/Jille/etcd-postgresql-sync","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jille%2Fetcd-postgresql-sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jille%2Fetcd-postgresql-sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jille%2Fetcd-postgresql-sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jille%2Fetcd-postgresql-sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Jille","download_url":"https://codeload.github.com/Jille/etcd-postgresql-sync/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jille%2Fetcd-postgresql-sync/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260797291,"owners_count":23064741,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etcd","etcd-backup","postgresql"],"created_at":"2024-11-21T21:18:09.231Z","updated_at":"2025-10-23T17:50:16.767Z","avatar_url":"https://github.com/Jille.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# etcd-postgresql-syncer\n\nA simple tool to sync your etcd cluster to PostgreSQL in realtime. It sets up a watcher on etcd and commits all changes to PostgreSQL.\n\nI use it to have a realtime backup of my etcd data. My PostgreSQL database is already replicated offsite, so I can leverage those benefits for free.\n\nNote that this syncer is asynchronous, so you might lose a few transactions when your cluster dies and you use PostgreSQL to restore your data from. (But that's fewer transactions than using a nightly backup.) If you want a synchronous replica, set up another etcd node offsite (which comes with a latency cost, of course).\n\nIt can also use https://github.com/Jille/etcd-auth-dump to periodically dump the authentication configuration to PostgreSQL. It needs to have the `root` role in etcd for that however.\n\n## Setup\n\n```\npostgres=# CREATE DATABASE etcd;\npostgres=# CREATE USER etcd_syncer PASSWORD 'hackme';\npostgres=# GRANT ALL PRIVILEGES ON DATABASE etcd TO etcd_syncer;\n\npsql -U etcd_syncer etcd -f database/schema.sql\n\netcdctl user add postgres_syncer\n\u003e Enter password hackme2\netcdctl user grant-role postgres_syncer root\n\nETCD_ENDPOINTS=https://127.0.0.1:2379 ETCD_USER=postgres_syncer ETCD_PASSWORD=hackme2 DATABASE_DSN=\"user=etcd_syncer password=hackme host=127.0.0.1 port=5432 dbname=etcd\" etcd-postgresql-sync\n```\n\nIf you don't want to grant it etcd root (and don't care about syncing authentication config), you can create a new role instead:\n\n```\netcdctl role add postgres_syncer\netcdctl role grant-permission postgres_syncer read \"\" --prefix\netcdctl user grant-role postgres_syncer postgres_syncer\n```\n\n## Parameters\n\nAll configuration is passed in through environment variables. It takes these settings:\n\n- ETCD_ENDPOINTS is where to find your etcd cluster\n- ETCD_USERNAME and ETCD_PASSWORD are used to connect to etcd. No authentication is used if you leave them unset/empty.\n- DATABASE_DSN specifies how to connect to PostgreSQL.\n- SYNCER_DEBUG can be set to \"true\" to make it log all queries sent to PostgreSQL.\n\nSee https://github.com/Jille/etcd-client-from-env for more parameters for connecting to etcd.\n\nSee the Setup section for example values.\n\n## Docker-compose\n\nAdd the following snippet to your docker-compose.yml to run it in Docker:\n\n```\n  postgresql-syncer:\n    image: \"ghcr.io/jille/etcd-postgresql-sync\"\n    restart: always\n    environment:\n      DATABASE_DSN: \"user=etcd_syncer password=hackme host=127.0.0.1 port=5432 dbname=etcd\"\n      ETCD_ENDPOINTS: https://etcd_etcd_1:2379\n      ETCD_USERNAME: postgres_syncer\n      ETCD_PASSWORD: hackme2\n      ETCD_SERVER_CA: |\n        -----BEGIN CERTIFICATE-----\n        [...]\n        -----END CERTIFICATE-----\n```\n\n## Future improvements\n\nCurrently, when the syncer starts it loads all data from etcd in memory and then starts pushing it to PostgreSQL. We could keep track of the revision we've synced up to and start watching again from that point, and only need to do a full copy when that revision has been compacted.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjille%2Fetcd-postgresql-sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjille%2Fetcd-postgresql-sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjille%2Fetcd-postgresql-sync/lists"}