{"id":15173837,"url":"https://github.com/capcom6/mariadb-backup-s3","last_synced_at":"2026-05-27T02:05:39.056Z","repository":{"id":250273071,"uuid":"833895742","full_name":"capcom6/mariadb-backup-s3","owner":"capcom6","description":"Automate MariaDB backups, compress them, and upload to S3-compatible storage. Easy setup with environment variables and command-line flags.","archived":false,"fork":false,"pushed_at":"2024-10-01T08:12:36.000Z","size":15,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-12-30T21:44:52.084Z","etag":null,"topics":["automation","aws","backup","cloud","compression","database","devops","go","golang","mariadb","s3"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/capcom6.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-26T01:57:34.000Z","updated_at":"2024-11-04T09:21:53.000Z","dependencies_parsed_at":"2024-10-11T02:40:56.858Z","dependency_job_id":null,"html_url":"https://github.com/capcom6/mariadb-backup-s3","commit_stats":{"total_commits":3,"total_committers":1,"mean_commits":3.0,"dds":0.0,"last_synced_commit":"6dc6be102c72644a8cd547aff207cacc6d4b7750"},"previous_names":["capcom6/mariadb-backup-s3"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/capcom6%2Fmariadb-backup-s3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/capcom6%2Fmariadb-backup-s3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/capcom6%2Fmariadb-backup-s3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/capcom6%2Fmariadb-backup-s3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/capcom6","download_url":"https://codeload.github.com/capcom6/mariadb-backup-s3/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239808534,"owners_count":19700454,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","aws","backup","cloud","compression","database","devops","go","golang","mariadb","s3"],"created_at":"2024-09-27T11:03:32.857Z","updated_at":"2026-05-27T02:05:39.044Z","avatar_url":"https://github.com/capcom6.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🗄️ MariaDB Backup to S3\n\n![Go Version](https://img.shields.io/github/go-mod/go-version/capcom6/mariadb-backup-s3)\n![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)\n\n🔁 Automated MariaDB database backups with S3-compatible storage integration\n\n## Table of Contents\n- [🗄️ MariaDB Backup to S3](#️-mariadb-backup-to-s3)\n  - [Table of Contents](#table-of-contents)\n  - [🚀 Quick Start](#-quick-start)\n  - [✨ Features](#-features)\n  - [🛠️ How It Works](#️-how-it-works)\n    - [Temporary Working Directory](#temporary-working-directory)\n    - [Backup Registry](#backup-registry)\n  - [📋 Prerequisites](#-prerequisites)\n  - [📦 Installation](#-installation)\n    - [Binary Installation (Recommended)](#binary-installation-recommended)\n    - [Using Go Install](#using-go-install)\n    - [Docker](#docker)\n    - [From Source (Advanced)](#from-source-advanced)\n  - [⚙️ Configuration](#️-configuration)\n    - [Logging Configuration](#logging-configuration)\n  - [🚀 Usage](#-usage)\n    - [Backup](#backup)\n    - [Restore](#restore)\n    - [Retention](#retention)\n    - [Registry](#registry)\n  - [📂 Storage Types](#-storage-types)\n    - [S3 Storage](#s3-storage)\n    - [FTP Storage](#ftp-storage)\n    - [Filesystem Storage](#filesystem-storage)\n  - [🔐 Encryption](#-encryption)\n    - [Security Considerations](#security-considerations)\n      - [Key Management](#key-management)\n      - [Key Generation](#key-generation)\n  - [📝 Examples](#-examples)\n  - [🤝 Contributing](#-contributing)\n  - [👥 Contributors](#-contributors)\n  - [📄 License](#-license)\n\n\n## 🚀 Quick Start\n\n```shell\n# Using pre-built binary (check Releases page for latest version)\ncurl -LO https://github.com/capcom6/mariadb-backup-s3/releases/latest/download/mariadb-backup-s3_Linux_x86_64.tar.gz\ntar -xzf mariadb-backup-s3_Linux_x86_64.tar.gz\nchmod +x mariadb-backup-s3\n./mariadb-backup-s3 --help\n\n# Or via go install\ngo install github.com/capcom6/mariadb-backup-s3@latest\n\n# Configure \u0026 run\ncp .env.example .env\nnano .env  # Edit with your credentials\n./mariadb-backup-s3\n```\n\n## ✨ Features\n\n- 🛡️ Full database backups using `mariabackup`\n- 🗜️ Compression to `.tar.gz` format\n- 🔑 Optional encryption using AES-256-GCM\n- ☁️ Multiple storage backends (S3-compatible, FTP, filesystem)\n- 🔌 Pluggable storage interface for extensibility\n- 🔄 Automatic backup rotation with configurable retention policies\n- 📋 Backup registry for tracking and managing backups\n- 🐳 Docker container support\n\n## 🛠️ How It Works\n\nThe backup process follows these steps:\n\n1. 📂 Create temporary working directory\n2. 💾 Perform MariaDB backup using `mariabackup --backup`\n3. 🔧 Prepare backup for consistency using `mariabackup --prepare`\n4. 🗜️ Compress backup to `.tar.gz` archive\n5. 🔑 Encrypt archive using AES-256-GCM\n6. 🚀 Upload archive to configured storage backend\n7. 📋 Update backup registry with new backup metadata\n8. 🧹 Clean up old backups based on retention policy (unless `--skip-retention` is set)\n\nThe restore process follows these steps:\n\n1. 📥 Download the backup file from the specified storage backend\n2. 🔑 Decrypt the backup using AES-256-GCM\n3. 🗜️ Decompress the `.tar.gz` archive\n4. 🔄 Restore the database files to specified directory\n\n### Temporary Working Directory\n\nThe tool uses a temporary working directory to store intermediate files. By default, it uses the system's default temporary directory (e.g., `/tmp`). If for some reason you need to use a different directory, you can specify it via the `TMPDIR` environment variable.\n\n```bash\nexport TMPDIR=/mnt/data/backup\n./mariadb-backup-s3 backup\n```\n\n### Backup Registry\n\nThe tool maintains a backup registry (`.backup-registry.json`) in the storage backend to track all backups. This registry enables:\n\n- **Backup tracking**: Each backup is recorded with metadata including ID, filename, creation time, size, SHA256 hash, encryption status, and tool version\n- **Retention management**: The registry is used to apply retention policies and clean up old backups\n- **Backup listing**: View all available backups with their status and metadata\n\nThe registry is automatically updated when backups are created or deleted. Each backup entry includes:\n\n| Field        | Description                                   |\n| ------------ | --------------------------------------------- |\n| `id`         | Unique identifier (timestamp + SHA256 prefix) |\n| `filename`   | Backup filename                               |\n| `created_at` | Creation timestamp                            |\n| `size_bytes` | File size in bytes                            |\n| `sha256`     | SHA256 hash of the backup file                |\n| `status`     | Status: `ready`, `failed`, or `deleted`       |\n| `encrypted`  | Whether the backup is encrypted               |\n| `encryption` | Encryption metadata (algorithm)               |\n| `tool`       | Tool name and version                         |\n\n## 📋 Prerequisites\n\n- Go 1.23+ (for building from source)\n- MariaDB server\n- At least 2x the actual database size in free space (for successful backup)\n- Storage backend credentials (depending on chosen storage type)\n\n## 📦 Installation\n\n### Binary Installation (Recommended)\n1. Visit the [Releases page](https://github.com/capcom6/mariadb-backup-s3/releases/latest)\n2. Download the appropriate binary for your OS\n3. Make executable: `chmod +x mariadb-backup-s3`\n4. Move to PATH: `sudo mv mariadb-backup-s3 /usr/local/bin/`\n\n### Using Go Install\n```shell\ngo install github.com/capcom6/mariadb-backup-s3@latest\n```\n\n### Docker\n```shell\ndocker pull ghcr.io/capcom6/mariadb-backup-s3:latest\n```\n\n\u003e **Note**\n\u003e The Docker image uses MariaDB's `lts` version. For specific versions:\n\u003e 1. Clone the repository\n\u003e 2. Modify `Dockerfile` base image\n\u003e 3. Build custom image: `docker build -t custom-backup-image .`\n\n### From Source (Advanced)\n```shell\ngit clone https://github.com/capcom6/mariadb-backup-s3.git\ncd mariadb-backup-s3\ngo build -o mariadb-backup-s3\n```\n\n## ⚙️ Configuration\n\nThe tool supports loading configuration from multiple sources:\n\n1. `.env` file in the current directory\n2. Environment variables\n3. Command-line flags\n\nThe priority order is: `.env` \u003e Environment variables \u003e Command-line flags\n\n### Logging Configuration\n\nThe logging system can be configured via environment variables:\n\n- `LOG_LEVEL`: Set log level (debug, info, warn, error, fatal). Default: info\n- `LOG_FORMAT`: Set format (human, json). Default: human\n- `LOG_OUTPUT`: Set output destination:\n  - `stdout` (default): Standard output\n  - `stderr`: Standard error\n  - Any file path: Write logs to the specified file\n- `NO_COLOR`: When set (any non-empty value), disables colored output for human format\n\n## 🚀 Usage\n\n```bash\nmariadb-backup-s3 [global options] command [command options] [arguments...]\n```\n\nThe tool offers the following commands:\n\n| Command     | Description                                       |\n| ----------- | ------------------------------------------------- |\n| `backup`    | Perform a backup of the MariaDB database          |\n| `restore`   | Restore the database files to specified directory |\n| `retention` | Apply retention policies to backups               |\n| `registry`  | Manage backup registry                            |\n\n### Backup\n\n```bash\nmariadb-backup-s3 backup [options]\n```\n\n**Options:**\n\n| Option                        | Env Var                   | Description                                       | Default value    |\n| ----------------------------- | ------------------------- | ------------------------------------------------- | ---------------- |\n| **Database**                  |                           |                                                   |                  |\n| `--db-host`, `--host`         | `MARIADB__HOST`           | MariaDB hostname                                  | `localhost`      |\n| `--db-port`, `--port`         | `MARIADB__PORT`           | MariaDB port                                      | `3306`           |\n| `--db-user`, `--user`         | `MARIADB__USER`           | MariaDB username                                  | `root`           |\n| `--db-password`, `--password` | `MARIADB__PASSWORD`       | MariaDB password                                  | `\"\"`             |\n| **Storage**                   |                           |                                                   |                  |\n| `--storage`, `--storage-url`  | `STORAGE__URL`            | Storage URL, see [Storage Types](#-storage-types) | **required**     |\n| **Encryption**                |                           |                                                   |                  |\n| `--encryption-key`            | `ENCRYPTION__KEY`         | Encryption key                                    | `\"\"`             |\n| **mariadb-backup**            |                           |                                                   |                  |\n| `--db-backup-binary`          | `MARIADB__BACKUP_BINARY`  | MariaDB backup binary path                        | `mariadb-backup` |\n| `--db-backup-options`         | `MARIADB__BACKUP_OPTIONS` | MariaDB backup options                            | `\"\"`             |\n| **Retention**                 |                           |                                                   |                  |\n| `--retention-count`           | `RETENTION__COUNT`        | Number of backups to retain, 0 = unlimited        | `0`              |\n| `--max-age`                   | `RETENTION__MAX_AGE`      | Maximum age of backups to keep (e.g. 24h, 168h)   | unlimited        |\n| `--keep-daily`                | `RETENTION__KEEP_DAILY`   | Number of daily backups to keep                   | unlimited        |\n| `--keep-weekly`               | `RETENTION__KEEP_WEEKLY`  | Number of weekly backups to keep                  | unlimited        |\n| `--keep-monthly`              | `RETENTION__KEEP_MONTHLY` | Number of monthly backups to keep                 | unlimited        |\n| `--skip-retention`            | `BACKUP__SKIP_RETENTION`  | Skip retention policy after backup                | `false`          |\n\n**Example:**\n\n```shell\n./mariadb-backup-s3 backup \\\n  --db-host=mariadb.example.com \\\n  --db-user=backup \\\n  --storage-url=\"file:///var/backups/mariadb\"\n```\n\n### Restore\n\n```bash\nmariadb-backup-s3 restore [options] [backup_name.tar.gz | --latest | --backup-id=\u003cid\u003e]\n```\n\n**Options:**\n\n| Option                       | Env Var               | Description                                       | Default value |\n| ---------------------------- | --------------------- | ------------------------------------------------- | ------------- |\n| **Storage**                  |                       |                                                   |               |\n| `--storage`, `--storage-url` | `STORAGE__URL`        | Storage URL, see [Storage Types](#-storage-types) | **required**  |\n| **Encryption**               |                       |                                                   |               |\n| `--encryption-key`           | `ENCRYPTION__KEY`     | Encryption key                                    | `\"\"`          |\n| **Restore**                  |                       |                                                   |               |\n| `--target-dir`               | `RESTORE__TARGET_DIR` | Target directory to restore files to              | **required**  |\n| `--latest`                   |                       | Restore latest ready backup from registry         | `false`       |\n| `--backup-id`                |                       | Restore backup by registry backup ID              | `\"\"`          |\n\n\u003e **Note**\n\u003e Exactly one backup selector must be specified: either a filename argument, `--latest`, or `--backup-id`.\n\n**Arguments:**\n\n| Argument   | Description      |\n| ---------- | ---------------- |\n| `filename` | Backup file name |\n\n**Example:**\n\n```shell\n./mariadb-backup-s3 restore \\\n  --storage-url=\"file:///var/backups/mariadb\" \\\n  --target-dir=/var/lib/mariadb \\\n  backup_name.tar.gz\n```\n\n### Retention\n\nThe `retention` command applies retention policies to backups stored in the configured storage backend. This allows you to clean up old backups based on various criteria.\n\n```bash\nmariadb-backup-s3 retention [options]\n```\n\n**Options:**\n\n| Option              | Env Var                   | Description                                          | Default value |\n| ------------------- | ------------------------- | ---------------------------------------------------- | ------------- |\n| **Storage**         |                           |                                                      |               |\n| `--storage-url`     | `STORAGE__URL`            | Storage URL, see [Storage Types](#-storage-types)    | **required**  |\n| **Retention**       |                           |                                                      |               |\n| `--retention-count` | `RETENTION__COUNT`        | Number of backups to retain, 0 = unlimited           | `0`           |\n| `--max-age`         | `RETENTION__MAX_AGE`      | Maximum age of backups to keep (e.g. 24h, 168h)      | unlimited     |\n| `--keep-daily`      | `RETENTION__KEEP_DAILY`   | Number of daily backups to keep                      | unlimited     |\n| `--keep-weekly`     | `RETENTION__KEEP_WEEKLY`  | Number of weekly backups to keep                     | unlimited     |\n| `--keep-monthly`    | `RETENTION__KEEP_MONTHLY` | Number of monthly backups to keep                    | unlimited     |\n| **Options**         |                           |                                                      |               |\n| `--dry-run`         | `RETENTION__DRY_RUN`      | Show what would be deleted without actually deleting | `false`       |\n| `--force`           | `RETENTION__FORCE`        | Continue even if errors occur                        | `false`       |\n\n**Retention Policy Examples:**\n\n```shell\n# Keep only the 7 most recent backups\n./mariadb-backup-s3 retention \\\n  --storage-url=\"s3://my-bucket/backups\" \\\n  --retention-count=7\n\n# Keep backups from the last 7 days\n./mariadb-backup-s3 retention \\\n  --storage-url=\"s3://my-bucket/backups\" \\\n  --max-age=168h\n\n# Keep 1 daily backup for 7 days, 1 weekly for 4 weeks, 1 monthly for 12 months\n./mariadb-backup-s3 retention \\\n  --storage-url=\"s3://my-bucket/backups\" \\\n  --keep-daily=7 \\\n  --keep-weekly=4 \\\n  --keep-monthly=12\n\n# Preview what would be deleted without actually deleting\n./mariadb-backup-s3 retention \\\n  --storage-url=\"s3://my-bucket/backups\" \\\n  --retention-count=3 \\\n  --dry-run\n```\n\n\u003e **Note**\n\u003e At least one retention policy must be enabled. The retention command will fail if no policies are specified.\n\n### Registry\n\nThe `registry` command allows you to manage and view the backup registry.\n\n```bash\nmariadb-backup-s3 registry [command] [options]\n```\n\n**Subcommands:**\n\n| Command | Alias | Description                |\n| ------- | ----- | -------------------------- |\n| `list`  | `ls`  | List backups from registry |\n\n**List Command Options:**\n\n| Option          | Env Var        | Description                                       | Default value |\n| --------------- | -------------- | ------------------------------------------------- | ------------- |\n| `--storage-url` | `STORAGE__URL` | Storage URL, see [Storage Types](#-storage-types) | **required**  |\n\n**Example:**\n\n```shell\n./mariadb-backup-s3 registry list \\\n  --storage-url=\"s3://my-bucket/backups\"\n```\n\n**Output:**\n\nThe list command displays all backups in the registry with the following information:\n- **ID**: Unique backup identifier\n- **Created At**: Backup creation timestamp\n- **Status**: Backup status (`ready`, `failed`, or `deleted`)\n- **Encrypted**: Whether the backup is encrypted\n- **Size**: Backup file size in bytes\n- **Filename**: Backup filename\n\n## 📂 Storage Types\n\n### S3 Storage\nFor S3-compatible storage (including AWS S3, MinIO, DigitalOcean Spaces, etc.):\n\n```dotenv\nSTORAGE__URL=s3://bucket-name/path?endpoint=https://s3.example.com\n```\n\n**Required for S3:**\n- `AWS_ACCESS_KEY`: Your access key\n- `AWS_SECRET_KEY`: Your secret key\n- `AWS_REGION`: AWS region (or any region for non-AWS S3)\n\n**Query Parameters:**\n- `endpoint`: S3 endpoint URL\n- `s3-force-path-style`: Set to \"true\" to use path-style URLs\n- `part-size`: Multipart upload part size in bytes (default: 10485760 = 10 MB, minimum: 5242880 = 5 MB)\n\n### FTP Storage\nFor FTP servers:\n\n```dotenv\nSTORAGE__URL=ftp://username:password@host:port/path\n```\n\n**Required for FTP:**\n- `username`: FTP username (defaults to \"anonymous\" if not provided)\n- `password`: FTP password (optional for anonymous)\n- `host`: FTP server hostname\n- `port`: FTP port (defaults to 21)\n\n### Filesystem Storage\nFor local or mounted filesystem storage:\n\n```dotenv\nSTORAGE__URL=file:///absolute/path/to/backup/directory\n```\n\n**Examples:**\n- Linux/macOS: `file:///var/backups/mariadb`\n- Windows: `file://C:/backups/mariadb`\n- Docker volume: `file:///data/backups`\n\n## 🔐 Encryption\n\nThe backup system supports client-side encryption using AES-256 in Galois/Counter Mode (GCM) to ensure your database backups remain confidential and secure. This method provides both confidentiality and integrity verification.\n\n**Features:**\n- 256-bit key strength\n- Authenticated encryption with additional data (AEAD)\n- Automatic nonce generation for each backup\n\n**Configuration:**\n```dotenv\n# base64-encoded encryption key\nENCRYPTION__KEY=Av2cfWJ3enCHTyzPdzowfAXshvJtbEsvwgPjV46wnjc=\n```\n\n### Security Considerations\n\n#### Key Management\n- **Never commit encryption keys to version control**\n- Store keys in secure environment variables or dedicated secret management systems\n- Implement proper access controls for key storage\n\n#### Key Generation\nGenerate secure encryption keys using cryptographically secure methods:\n\n```bash\n# Generate a 32-byte (256-bit) key for AES256-GCM\nopenssl rand -base64 32\n\n# Alternative method using /dev/urandom\nhead -c 32 /dev/urandom | base64\n```\n\n## 📝 Examples\n\n- **systemd Service Example**: [examples/systemd-service](./examples/systemd-service/)\n- **Docker Swarm CRON Example**: [examples/docker-cron-backup](./examples/docker-cron-backup/)\n- **Simple CRON Example**: [examples/simple-cron-backup](./examples/simple-cron-backup/)\n- **Advanced CRON Example**: [examples/advanced-cron-backup](./examples/advanced-cron-backup/)\n- **Encryption Example**: [examples/encryption-example](./examples/encryption-example/)\n\n## 🤝 Contributing\n\nWe welcome contributions! Please follow these steps:\n\n1. 🍴 Fork the repository\n2. 🌿 Create a feature branch: `git checkout -b feat/amazing-feature`\n3. 💾 Commit changes: `git commit -m 'Add amazing feature'`\n4. 🚀 Push to branch: `git push origin feat/amazing-feature`\n5. 🔀 Create a Pull Request\n\n## 👥 Contributors\n\nA big thank you to everyone who has contributed to this project!\n\n- [gslongo](https://github.com/gslongo)\n\nSee the full [CONTRIBUTORS.md](CONTRIBUTORS.md) file for more information.\n\n## 📄 License\n\nApache 2.0 - See [LICENSE](LICENSE) for details.\n\n---\n\n💡 **Need Help?** Open an [issue](https://github.com/capcom6/mariadb-backup-s3/issues) for support.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcapcom6%2Fmariadb-backup-s3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcapcom6%2Fmariadb-backup-s3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcapcom6%2Fmariadb-backup-s3/lists"}