Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/tomcz/s3backup

No more custom backup scripts please.
https://github.com/tomcz/s3backup

backup backup-cli backup-utility encrypts s3-bucket vault

Last synced: about 1 month ago
JSON representation

No more custom backup scripts please.

Awesome Lists containing this project

README 

          
# S3 backup script in a single binary



Provides a standard way of backing up an archive to a S3 bucket, and restoring the backed up archive from its S3 bucket. No more custom backup scripts please ...



You can download the latest release from [here](https://github.com/tomcz/s3backup/releases).



## Upload process



1. Encrypt the file to be backed up (optional but highly recommended). `s3backup` uses AES-256 encryption via a password of your choice (with argon2 key derivation), a Base64-encoded secret key, or a PEM-encoded RSA public key. If a public key is provided, `s3backup` will generate a random 256-bit symmetric key which will be encrypted using the public key and stored with the encrypted file. To make key creation easier, you can use the `keygen` commands as outlined [below](#backup-key-generation).



2. Calculate SHA-256 checksum for the file to be uploaded. For encrypted uploads the checksum is calculated on the encrypted file.



3. Upload to AWS S3 using concurrent uploads to handle large files and store the checksum with the uploaded file.



## Download process



1. Download file from AWS S3 using concurrent downloads to handle large files and retrieve the stored checksum of the uploaded file.



2. Verify that the stored checksum matches the downloaded file.



3. Optionally decrypt the downloaded file using either the same password or symmetric key that was used to encrypt it, or the RSA private key matching the RSA public key that was used for encryption.



## Usage



```

NAME:

   s3backup - S3 backup script in a single binary



USAGE:

   s3backup [global options] [command [command options]]



COMMANDS:

   put        Upload file to S3 bucket using local credentials

   get        Download file from S3 bucket using local credentials

   vault-put  Upload file to S3 bucket using credentials from vault

   vault-get  Download file from S3 bucket using credentials from vault

   keygen     Generate RSA and AES backup keys

   encrypt    Encrypt a local file

   decrypt    Decrypt a local file

   version    Print version and exit

   help, h    Shows a list of commands or help for one command



GLOBAL OPTIONS:

   --help, -h     show help

   --version, -v  print the version

```



### AWS S3 Credentials



AWS S3 integration in `s3backup` can be configured from the command line, and/or an optional YAML configuration file provided by the `S3BACKUP_YAML` environment variable (in which its yaml key names match the option names), and using AWS environment variables and config files. [Click here](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html) for details on using default AWS credentials.



#### s3backup put



```

NAME:

   s3backup put - Upload file to S3 bucket using local credentials



USAGE:

   s3backup put [options] local_file_path s3://bucket/objectkey 



OPTIONS:

   --oldPass, --old, -o           Maintain password compatibility with older s3backup releases

   --symKey string, --sym string  Password or base64-encoded key to use for symmetric AES

                                  encryption. Use "ask" as the value to provide a password

                                  via an interactive prompt

   --pemKey FILE, --pem FILE      Path to PEM-encoded public key FILE

   --accessKey string             AWS Access Key ID (if not using default AWS credentials)

   --secretKey string             AWS Secret Key (required when accessKey is provided)

   --token string                 AWS Token (effective only when accessKey is provided &

                                  only if required by your AWS setup)

   --region string                AWS Region (we use AWS defaults if not provided)

   --endpoint URL                 Custom AWS Endpoint URL (optional)

   --nocheck                      Do not create backup checksums

   --help, -h                     show help

```



#### s3backup get



```

NAME:

   s3backup get - Download file from S3 bucket using local credentials



USAGE:

   s3backup get [options] s3://bucket/objectkey local_file_path 



OPTIONS:

   --symKey string, --sym string  Password or base64-encoded key to use for symmetric AES

                                  decryption. Use "ask" as the value to provide a password

                                  via an interactive prompt

   --pemKey FILE, --pem FILE      Path to PEM-encoded private key FILE

   --accessKey string             AWS Access Key ID (if not using default AWS credentials)

   --secretKey string             AWS Secret Key (required when accessKey is provided)

   --token string                 AWS Token (effective only when accessKey is provided &

                                  only if required by your AWS setup)

   --region string                AWS Region (we use AWS defaults if not provided)

   --endpoint URL                 Custom AWS Endpoint URL (optional)

   --nocheck                      Do not verify backup checksums

   --help, -h                     show help

```



### HashiCorp Vault



`s3backup` provides `vault-put` and `vault-get` commands that allow it to be configured using secrets held by a [vault](https://www.vaultproject.io/) instance so that you can store encryption keys and AWS credentials in a secure manner. The secrets that you need to hold in vault for `s3backup` are described [here](https://github.com/tomcz/s3backup/blob/master/config/config.go).



Vault integration in `s3backup` can be configured from the command line, and/or an optional YAML configuration file provided by the `S3BACKUP_YAML` environment variable (in which its yaml key names match the option names), and using vault's own [environment variables](https://www.vaultproject.io/docs/commands/environment.html).



#### s3backup vault-put



```

NAME:

   s3backup vault-put - Upload file to S3 bucket using credentials from vault



USAGE:

   s3backup vault-put [options] local_file_path s3://bucket/objectkey 



OPTIONS:

   --path string    Vault secret path containing backup credentials (required)

   --kv2            Vault secret path represents a key/value version 2 secrets engine

   --mount string   Vault approle mount path (default: approle)

   --role string    Vault role_id to retrieve backup credentials

                    (either role & secret, or token) [$VAULT_ROLE_ID]

   --secret string  Vault secret_id to retrieve backup credentials

                    (either role & secret, or token) [$VAULT_SECRET_ID]

   --token string   Vault token to retrieve backup credentials

                    (either role & secret, or token) [$VAULT_TOKEN]

   --caCert FILE    Vault root certificate FILE (optional, or use one of VAULT_CACERT,

                    VAULT_CACERT_BYTES, VAULT_CAPATH env vars)

   --vault URL      Vault service URL (or use VAULT_ADDR env var)

   --nocheck        Do not create backup checksums

   --help, -h       show help

```



#### s3backup vault-get



```

NAME:

   s3backup vault-get - Download file from S3 bucket using credentials from vault



USAGE:

   s3backup vault-get [options] s3://bucket/objectkey local_file_path 



OPTIONS:

   --path string    Vault secret path containing backup credentials (required)

   --kv2            Vault secret path represents a key/value version 2 secrets engine

   --mount string   Vault approle mount path (default: approle)

   --role string    Vault role_id to retrieve backup credentials

                    (either role & secret, or token) [$VAULT_ROLE_ID]

   --secret string  Vault secret_id to retrieve backup credentials

                    (either role & secret, or token) [$VAULT_SECRET_ID]

   --token string   Vault token to retrieve backup credentials

                    (either role & secret, or token) [$VAULT_TOKEN]

   --caCert FILE    Vault root certificate FILE (optional, or use one of VAULT_CACERT,

                    VAULT_CACERT_BYTES, VAULT_CAPATH env vars)

   --vault URL      Vault service URL (or use VAULT_ADDR env var)

   --nocheck        Do not verify backup checksums

   --help, -h       show help

```



## Backup key generation



To make things easier, `s3backup` also provides `keygen` commands to create 256-bit symmetric keys and 4096-bit RSA private/public key pairs suitable for use by `s3backup`.



```

NAME:

   s3backup keygen - Generate RSA and AES backup keys



USAGE:

   s3backup keygen [command [command options]]



COMMANDS:

   aes  Generate and print AES key

   rsa  Generate RSA key pair files



OPTIONS:

   --help, -h  show help

```



## Build



1. Install Go 1.26 from https://golang.org/

2. Build the binaries: `make build`