{"id":19838525,"url":"https://github.com/metal-stack/backup-restore-sidecar","last_synced_at":"2025-05-01T18:31:20.061Z","repository":{"id":41084780,"uuid":"236530529","full_name":"metal-stack/backup-restore-sidecar","owner":"metal-stack","description":"K8s sidecar for database auto-backup and -recovery","archived":false,"fork":false,"pushed_at":"2025-04-17T05:39:09.000Z","size":507,"stargazers_count":14,"open_issues_count":13,"forks_count":1,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-27T09:32:59.301Z","etag":null,"topics":["k8s-sidecar"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/metal-stack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-27T16:03:56.000Z","updated_at":"2025-04-17T05:38:13.000Z","dependencies_parsed_at":"2023-01-19T23:17:54.896Z","dependency_job_id":"2b5a9b18-d52e-4e77-96b7-c53bf375e50d","html_url":"https://github.com/metal-stack/backup-restore-sidecar","commit_stats":null,"previous_names":["metal-pod/backup-restore-sidecar"],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metal-stack%2Fbackup-restore-sidecar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metal-stack%2Fbackup-restore-sidecar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metal-stack%2Fbackup-restore-sidecar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metal-stack%2Fbackup-restore-sidecar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/metal-stack","download_url":"https://codeload.github.com/metal-stack/backup-restore-sidecar/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251924718,"owners_count":21666025,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["k8s-sidecar"],"created_at":"2024-11-12T12:18:03.046Z","updated_at":"2025-05-01T18:31:20.056Z","avatar_url":"https://github.com/metal-stack.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# K8s Backup Restore Sidecar for Databases\n\nThis project adds automatic backup and recovery to databases managed by K8s via sidecar.\n\nThe idea is taken from the [etcd-backup-restore](https://github.com/gardener/etcd-backup-restore) project.\n\nProbably, it does not make sense to use this project with large databases. However, if it is certain that a database will never grow large, the auto-recovery mechanism can come in very handy.\n\n## Supported Databases\n\n| Database    | Image        | Status | Upgrade Support |\n| ----------- | ------------ | :----: | :-------------: |\n| postgres    | \u003e= 12-alpine |  beta  |       ✅        |\n| rethinkdb   | \u003e= 2.4.0     |  beta  |       ❌        |\n| ETCD        | \u003e= 3.5       | alpha  |       ❌        |\n| meilisearch | \u003e= 1.2.0     | alpha  |       ✅        |\n| redis       | \u003e= 6.0       | alpha  |       ❌        |\n| keydb       | \u003e= 6.0       | alpha  |       ❌        |\n| localfs     |              | alpha  |       ❌        |\n\nPostgres also supports updates when using the TimescaleDB extension. Please consider the integration test for supported upgrade paths.\n\n## Database Upgrades\n\n### Postgres\n\nPostgres requires special treatment if a major version upgrade is planned. `pg_upgrade` needs to be called with the old and new binaries, also the old data directory and a already initialized data directory which was initialized with the new binary, e.g. `initdb \u003cnew directory\u003e`.\n\nTo make this process as smooth as possible, backup-restore-sidecar will detect if the version of the database files and the version of the postgres binary. If the binary is newer than the database files it will start the upgrade process. Strict validation to ensure all prerequisites are met is done before actually starting the upgrade process.\n\nTo achieve this, `backup-restore-sidecar` saves the postgres binaries in the database directory in the form of `pg-bin-v12` for postgres 12. If later the database version is upgraded, the previous postgres binaries are present for doing the actual upgrade.\n\n## Supported Compression Methods\n\nWith `--compression-method` you can define how generated backups are compressed before stored at the storage provider. Available compression methods are:\n\n| compression-method | suffix   | comments                                                                                     |\n|--------------------|----------|----------------------------------------------------------------------------------------------|\n| tar                | .tar     | no compression, best suited for already compressed content                                   |\n| targz              | .tar.gz  | tar and gzip, most commonly used, best compression ratio, average performance                |\n| tarlz4             | .tar.lz4 | tar and lz4, very fast compression/decompression speed compared to gz, slightly bigger files |\n\n## Supported Storage Providers\n\n- GCS Buckets\n- S3 Buckets (tested against Ceph RADOS gateway)\n- Local\n\n## Encryption\n\nFor all three storage providers AES encryption is supported and can be enabled with `--encryption-key=\u003cYOUR_KEY\u003e`.\nThe key must be 32 bytes (AES-256) long.\nThe backups are stored at the storage provider with the `.aes` suffix. If the file does not have this suffix, decryption is skipped.\n\n## How it works\n\n![Sequence Diagram](docs/sequence.drawio.svg)\n\n## Limitations\n\n- The database is deployed unclustered / standalone\n- The database is deployed as a statefulset and the data is backed by a PVC\n- No \"Point in Time Recovery\" (PITR)\n\n## Using Multiple Backup-Restore-Sidecars On a Single Bucket\n\nIt is possible to let multiple backup-restore-sidecars (for different databases) use the same backup bucket at an external provider. However, it has to be noted that these sidecars must all configure a dedicated object prefix in which they store the backups. Otherwise they would overwrite each other's data.\n\nBe aware that, if you change the object prefix under which the backups are stored, the old lifecycle policies matching this prefix are not automatically cleaned up and have to be removed manually.\n\n## Try it out\n\nRequires:\n\n- [docker](https://www.docker.com/)\n- [kind](https://github.com/kubernetes-sigs/kind)\n- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)\n- [stern](https://github.com/wercker/stern)\n\nTo start a demo / devel setup, run: `make start-postgres` or `make start-rethinkdb`.\n\nBy default, the backup-restore-sidecar will start with the `local` backup provider, which is probably not very useful for most use-cases. If you want to test storing the data at a real backup provider, then:\n\n1. Configure the backup provider secret in `deploy/provider-secret-\u003cbackup-provider\u003e.yaml`.\n2. Run `BACKUP_PROVIDER=\u003cbackup-provider\u003e make start-postgres` instead.\n\n## Manual restoration\n\nFollow the documentation [here](docs/manual_restore.md) in order to manually restore a specific version of your database.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetal-stack%2Fbackup-restore-sidecar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmetal-stack%2Fbackup-restore-sidecar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetal-stack%2Fbackup-restore-sidecar/lists"}