https://github.com/theandrew168/pg2s3
Simple PostgreSQL backups to S3-compatible storage
https://github.com/theandrew168/pg2s3
backup postgresql s3
Last synced: about 1 year ago
JSON representation
Simple PostgreSQL backups to S3-compatible storage
- Host: GitHub
- URL: https://github.com/theandrew168/pg2s3
- Owner: theandrew168
- License: mit
- Created: 2021-09-17T19:43:27.000Z (almost 5 years ago)
- Default Branch: main
- Last Pushed: 2025-02-14T04:22:09.000Z (over 1 year ago)
- Last Synced: 2025-04-19T13:09:41.223Z (about 1 year ago)
- Topics: backup, postgresql, s3
- Language: Go
- Homepage:
- Size: 759 KB
- Stars: 4
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# pg2s3
Simple PostgreSQL backups to S3-compatible storage
## Overview
This project strives to be a simple and reliable backup solution for [PostgreSQL](https://www.postgresql.org/) databases.
In general, pg2s3 dumps a given database and uploads it to an S3-compatible storage bucket.
However, there is a bit more nuance involved in bookkeeping, restoration, and pruning.
## Install
The pg2s3 tool is distributed as a single, static binary for all major platforms.
It is also released as a `.deb` for Debian-based Linux environments.
Check the [releases page](https://github.com/theandrew168/pg2s3/releases) to find and download the latest version.
Additionally, the environment where pg2s3 is executed must have `pg_dump` and `pg_restore` installed.
These tools are part of the collection of [PostgreSQL Client Applications](https://www.postgresql.org/docs/12/reference-client.html).
On an Ubuntu server, these tools are contained within the package `postgresql-client-` based on the major version of PostgreSQL being used.
## Configuration
Configuration for pg2s3 is handled exclusively through a config file written in [TOML](https://github.com/toml-lang/toml).
By default, pg2s3 will look for a config file named `pg2s3.conf` in the current directory.
This file can be overridden by using the `-conf` flag.
Note that the S3 bucket defined by `s3_url` must be created outside of this tool.
Bucket creation has more configuration and security options than pg2s3 is positioned to deal with.
Additionally, the value defined by `backup.retention` simply refers to the _number_ of backups kept during a prune.
It has nothing to do with a backup's age or total bucket size.
If `backup.schedule` is set, you'll want to consider the scheduling frequency when determining an appropriate retention count.
The following settings are required to run pg2s3:
| Setting | Required? | Description |
| ------------------ | --------- | ----------- |
| `pg_url` | Yes | PostgreSQL connection string |
| `s3_url` | Yes | S3-compatible storage connection string |
| `backup.prefix` | No | Prefix attached to the name of each backup (default `"pg2s3"`) |
| `backup.retention` | No | Number of backups to retain after pruning |
| `backup.schedule` | No | Backup schedule as a standard cron expression (UTC) |
| `restore.schemas` | No | List of schemas to restore (default `["public"]`) |
## Encryption
Backups managed by pg2s3 can be optionally encrypted using [age](https://github.com/FiloSottile/age).
To enable this feature, an age public key must be defined within the config file.
Note that the private key associated with this public key must be kept safe and secure!
When restoring a backup, pg2s3 will prompt for the private key.
This key is intentionally absent from pg2s3's configuration in order to require user intervention for any data decryption.
| Setting | Required? | Description |
| ------------------------ | --------- | ----------- |
| `encryption.public_keys` | No | Public keys for backup encryption |
## Usage
The pg2s3 command-line tool offers three mutually-exclusive actions:
* `pg2s3 backup` - Create a new backup and upload to S3
* `pg2s3 restore` - Download the latest backup from S3 and restore
* `pg2s3 prune` - Prune old backups from S3
If none of these are provided, pg2s3 will attempt to run in scheduled mode: sleeping until `backup.schedule` arrives and then performing a backup + prune.
## Local Development
To develop and test locally, containers for [PostgreSQL](https://www.postgresql.org/) and [MinIO](https://min.io/) must be running:
```
docker compose up -d
```
These containers can be stopped via:
```
docker compose down
```
## Testing
With the above containers running:
```
go test ./...
```