{"id":26279564,"url":"https://github.com/devopsvanilla/datashelter","last_synced_at":"2025-03-14T14:15:27.670Z","repository":{"id":281745089,"uuid":"946155130","full_name":"devopsvanilla/datashelter","owner":"devopsvanilla","description":"K8S automation for database backups","archived":false,"fork":false,"pushed_at":"2025-03-13T04:29:20.000Z","size":40,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-13T05:27:02.441Z","etag":null,"topics":["backup","helm","k8s","kubernetes","mysql"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/devopsvanilla.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-10T17:39:42.000Z","updated_at":"2025-03-13T04:29:24.000Z","dependencies_parsed_at":"2025-03-13T05:27:02.873Z","dependency_job_id":null,"html_url":"https://github.com/devopsvanilla/datashelter","commit_stats":null,"previous_names":["devopsvanilla/mysql-k8s-backup","devopsvanilla/datashelter"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devopsvanilla%2Fdatashelter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devopsvanilla%2Fdatashelter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devopsvanilla%2Fdatashelter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devopsvanilla%2Fdatashelter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/devopsvanilla","download_url":"https://codeload.github.com/devopsvanilla/datashelter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243589329,"owners_count":20315471,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","helm","k8s","kubernetes","mysql"],"created_at":"2025-03-14T14:15:26.898Z","updated_at":"2025-03-14T14:15:27.572Z","avatar_url":"https://github.com/devopsvanilla.png","language":"Shell","readme":"# DataShelter\n\n![DataShelter](./images/DataShelter.png)\n\nMade with ❤️ by [DevOps Vanilla](https://devopsvanilla.guru)\n\n[![Versão em Português](https://img.shields.io/badge/README-pt--BR-green)](README_pt-BR.md)\n\nThis MySQL backup automation solution using Helm offers an automated and flexible way to manage MySQL database backups in a Kubernetes environment. Designed with efficiency in mind, it delivers high performance while maintaining a lightweight footprint. With support for different types of backups, flexible scheduling, email notifications, multiple backup destinations, data encryption, and retention management, the solution ensures the security, availability, and optimized resource utilization of database data\n\n## About\n\n- [DataShelter](#datashelter)\n  - [About](#about)\n    - [Solution Architecture](#solution-architecture)\n    - [Features](#features)\n    - [Configuration Details](#configuration-details)\n    - [Use Cases](#use-cases)\n    - [How to Deploy the Solution](#how-to-deploy-the-solution)\n      - [Prerequisites](#prerequisites)\n      - [Deployment Steps](#deployment-steps)\n    - [Deployment Verification](#deployment-verification)\n    - [Monitoring Backup Execution](#monitoring-backup-execution)\n    - [Deleting the Solution](#deleting-the-solution)\n    - [Dependency Configuration](#dependency-configuration)\n      - [Configuring Buckets on AWS and Digital Ocean](#configuring-buckets-on-aws-and-digital-ocean)\n      - [Generating Encryption Keys](#generating-encryption-keys)\n    - [Other Tips and References](#other-tips-and-references)\n\n### Solution Architecture\n\nThe MySQL backup solution consists of the following components:\n\n```mermaid\ngraph TD\n    A[Kubernetes Cluster] --\u003e|Deploy| B[Helm Chart]\n    B --\u003e|Creates| C[Job]\n    B --\u003e|Creates| D[CronJob]\n    B --\u003e|Creates| E[ConfigMap]\n    B --\u003e|Creates| F[Secret]\n    B --\u003e|Creates| G[PersistentVolumeClaim]\n    B --\u003e|Creates| H[Notification Job]\n    C --\u003e|Executes| I[Backup Script]\n    D --\u003e|Schedules| I\n    I --\u003e|Stores| G\n    I --\u003e|Upload| J[Digital Ocean Spaces]\n    I --\u003e|Upload| K[AWS S3]\n    I --\u003e|Encrypt| L[RSA Public Key]\n    I --\u003e|Send| H\n    H --\u003e|Uses| F\n    H --\u003e|Uses| E\n```\n\n### Features\n\nThe solution offers the following features:\n\n- **Backup Types**: Support for full schema, incremental schema, full server, and incremental server backups.\n- **Scheduling**: Allows configuring multiple backup schedules with different frequencies and execution times.\n- **Email Notifications**: Sends email notifications about backup completion and errors, with support for multiple recipients.\n- **Backup Destinations**: Support for storing backups in Digital Ocean Spaces and AWS S3.\n- **Encryption**: Encrypts backups using RSA public keys.\n- **Retention**: Manages backup retention, automatically deleting old backups after a specified period.\n\n### Configuration Details\n\nThe MySQL backup solution configuration is managed through the `values.yaml` file. Here are the main configuration sections:\n\n- **MySQL Connection Details**: Configure the MySQL host, port, username, and password.\n  \n  ```yaml\n  mysql:\n    host: localhost\n    port: 3306\n    username: root\n    password: password\n  ```\n\n- **Backup Types**: Enable or disable different types of backups.\n  \n  ```yaml\n  backupTypes:\n    schemaTotal: true\n    schemaIncremental: true\n    serverTotal: true\n    serverIncremental: true\n  ```\n\n- **Scheduling**: Configure the scheduling parameters for each type of backup.\n\n  ```yaml\n  scheduling:\n    schemaTotal:\n      frequency: daily\n      time: \"02:00\"\n      maxExecutionTime: 60\n    schemaIncremental:\n      frequency: daily\n      time: \"04:00\"\n      maxExecutionTime: 30\n    serverTotal:\n      frequency: weekly\n      dayOfWeek: \"Sunday\"\n      time: \"03:00\"\n      maxExecutionTime: 120\n    serverIncremental:\n      frequency: daily\n      time: \"06:00\"\n      maxExecutionTime: 45\n  ```\n\n- **Email Notifications**: Configure the email notification parameters.\n\n  ```yaml\n  notifications:\n    email:\n      enabled: true\n      smtp:\n        host: smtp.example.com\n        port: 587\n        username: user@example.com\n        password: password\n      recipients: recipient1@example.com,recipient2@example.com\n  ```\n\n- **Backup Destinations**: Configure the settings for Digital Ocean Spaces and AWS S3.\n\n  ```yaml\n  backupDestinations:\n    digitalOceanSpaces:\n      enabled: true\n      accessKey: DO_ACCESS_KEY\n      secretKey: DO_SECRET_KEY\n      region: nyc3\n      bucket: my-backups\n    awsS3:\n      enabled: false\n      accessKey: AWS_ACCESS_KEY\n      secretKey: AWS_SECRET_KEY\n      region: us-west-2\n      bucket: my-backups\n  ```\n\n- **Encryption**: Configure the RSA public key for encryption.\n\n  ```yaml\n  encryption:\n    rsaPublicKeyPath: /path/to/public.key\n  ```\n\n- **Retention**: Configure the backup retention period.\n\n  ```yaml\n  retention:\n    days: 30\n  ```\n\n### Use Cases\n\nThe MySQL backup solution can be used in various scenarios to ensure reliable and automated backups of MySQL databases. Here are some use cases:\n\n- **Daily Incremental Backups**: Configure daily incremental backups to capture changes made to the database throughout the day.\n- **Weekly Full Backups**: Schedule weekly full backups to create a complete snapshot of the database, including data and structure.\n- **Email Notifications**: Configure email notifications to receive alerts about backup completion and errors, ensuring timely awareness of backup status.\n- **Backups to Multiple Destinations**: Store backups in multiple destinations, such as Digital Ocean Spaces and AWS S3, for redundancy and disaster recovery.\n- **Encrypted Backups**: Encrypt backups using RSA public keys to ensure data security and compliance with regulatory requirements.\n- **Backup Retention**: Manage backup retention by automatically deleting old backups after a specified period, optimizing storage usage.\n\n### How to Deploy the Solution\n\n#### Prerequisites\n\n- Configured Kubernetes cluster.\n- Installed Helm.\n\n#### Deployment Steps\n\n1. Clone the repository:\n\n   ```sh\n   git clone https://github.com/devopsvanilla/mysql-k8s-backup.git\n   cd mysql-k8s-backup\n   ```\n\n2. Configure the `values.yaml` file as needed.\n\n3. Deploy the solution using Helm:\n\n   ```sh\n   helm install mysql-backup ./helm/mysql-backup\n   ```\n\n### Deployment Verification\n\nTo verify if the deployment was successful, run the following command:\n\n```sh\nkubectl get pods\n```\n\nCheck if the pods related to the MySQL backup are running.\n\n### Monitoring Backup Execution\n\nTo monitor the execution of backups, you can check the logs of the pods:\n\n```sh\nkubectl logs \u003cpod-name\u003e\n```\n\n### Deleting the Solution\n\nTo delete the solution, run the following command:\n\n```sh\nhelm uninstall mysql-backup\n```\n\n### Dependency Configuration\n\n#### Configuring Buckets on AWS and Digital Ocean\n\n- **AWS S3**: Create a bucket on AWS S3 and configure the access credentials in the `values.yaml` file.\n- **Digital Ocean Spaces**: Create a bucket on Digital Ocean Spaces and configure the access credentials in the `values.yaml` file.\n\n#### Generating Encryption Keys\n\nTo generate RSA keys for encryption, run the following command:\n\n```sh\nopenssl genrsa -out private.key 2048\nopenssl rsa -in private.key -pubout -out public.key\n```\n\nConfigure the path to the public key in the `values.yaml` file.\n\n### Other Tips and References\n\n- Ensure that the Kubernetes cluster has sufficient resources to run the backup jobs.\n- Regularly check the logs of the pods to ensure that backups are being executed as expected.\n- Refer to the official Helm and Kubernetes documentation for more information on managing and monitoring resources in the cluster.\n\n___\n\nMade with 💙 by [DevOps Vanilla.guru](https://DevOpsVanilla.guru)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevopsvanilla%2Fdatashelter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevopsvanilla%2Fdatashelter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevopsvanilla%2Fdatashelter/lists"}