{"id":19096574,"url":"https://github.com/sapcc/maria-back-me-up","last_synced_at":"2025-04-30T14:14:52.144Z","repository":{"id":36989629,"uuid":"208272137","full_name":"sapcc/maria-back-me-up","owner":"sapcc","description":"Mariadb incremental backup and restore tool for k8s","archived":false,"fork":false,"pushed_at":"2025-04-17T00:40:55.000Z","size":14265,"stargazers_count":16,"open_issues_count":18,"forks_count":0,"subscribers_count":48,"default_branch":"master","last_synced_at":"2025-04-17T13:42:06.711Z","etag":null,"topics":["backup","binlog","database","incremental-backups","k8s","mariadb","restore","s3-storage","swift"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sapcc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-09-13T13:39:21.000Z","updated_at":"2025-04-09T10:14:44.000Z","dependencies_parsed_at":"2024-01-18T03:42:40.822Z","dependency_job_id":"0e2383b2-8c39-49e9-9142-80c681828488","html_url":"https://github.com/sapcc/maria-back-me-up","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sapcc%2Fmaria-back-me-up","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sapcc%2Fmaria-back-me-up/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sapcc%2Fmaria-back-me-up/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sapcc%2Fmaria-back-me-up/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sapcc","download_url":"https://codeload.github.com/sapcc/maria-back-me-up/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249514448,"owners_count":21284546,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","binlog","database","incremental-backups","k8s","mariadb","restore","s3-storage","swift"],"created_at":"2024-11-09T03:37:07.599Z","updated_at":"2025-04-18T15:33:49.084Z","avatar_url":"https://github.com/sapcc.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# maria-back-me-up\n\nMariaDB backup tool\n\n## Features\n\nList of features currently available:\n\n- Full dump via MyDumper or mysqldump (invertal can be configured)\n- Incremental backups via binlog (invertal can be configured)\n- Supported backup storage\n  - S3\n  - Swift\n  - Disk\n- Automatic verification of existing backups (can be run as separate service)\n- UI to see and select an available backup to restore to\n- UI shows status of backup verification\n- UI/API can be secured via OAuth openID\n- Replication of full dump and binlog events to another MariaDB\n  - only QueryEvents are supported\n  - restore from this replicas MariaDB is not supported\n  - verification of this replica is not supported\n  \n## Architecture\n\u003cimg width=\"605\" alt=\"maria-backup\" src=\"https://user-images.githubusercontent.com/566649/215467649-ac049663-c6fa-4e0e-b21f-25b9af6c187c.png\"\u003e\n\n## UI\n\nThe UI is available via localhost:8081/\nIt shows a list of available full backups in S3. Any full backup contains 1 or more incremental backups, which can be selected to perform a complete restore!\\\nThe color of an incremental backups shows the state of the backup verification:\\\n\n```text\n# backup verfication not yet executed\n- backup verfication failed\n! Backup verfication partly succeeded. A restore was successful, however the table checksum failed\n+ backup verfication successful. A restore is save to perform!\n```\n\n## Full logical backups\n\nAre done either via the mysql_dump **(default)** or the [MyDumper tool](https://github.com/maxbube/mydumper).\nMyDumper can use multiple threads to dump and restore tables, makes it therefore suitable for databases with a huge number of tables.\n\n```text\nfull_dump_tool=mysqldump/mydumper\n```\n\n## Incremental backups via binlogs\nThis backup tool uses binlogs to make incremental backups.\\\nTherefore the binlog needs to be enabled in the MariaDB config\n\n```text\nlog-bin=bin.log      # Binlog folder and name\nbinlog_format=MIXED  # Formant, described below\nexpire_logs_days=3   # After x days binlog files get purged. Important! Otherwise volume could be filling up fast\nserver_id=1          # Unique server id. Used for replication\n```\n\n## Binlog\n\nThe tool acts like a replication slave and receives all the binlog events. This allows it to check if any real changes have been made to the database.\\\nIf no changes have been detected, no incremental backup will be created and saved to S3.\n\n## Binlog Format\n\nBy default MariaDB will use the MIXED format (since 10.2.4). It is a mix of ROW and STATEMENT.\\\nROW will capture the actual change made to a table. The binlog files therefor can get very large.\\\ne.g. an update to a table column of 1000 rows will create 1000 row changes.\nWith STATEMENT only the update statement will be recorded in the binlog.\n\n## Config\n\n``` yaml\nservice_name: # Name of your MariaDB (also used as the s3 folder name)\nnamespace: # k8s namespace name\nsidecar: # boolean\nbackup:\n  full_backup_interval_in_hours: # Interval for full MariaDB dumps in hours\n  incremental_backup_interval_in_minutes: # Interval for saving incremental backups, one continous increment if \u003c 0\n  purge_binlog_after_minutes: # if \u003e 0 binlog files are kept on the primary db until they are older\n  enable_init_restore: # Enables a automatic restore if one of the databases (in MariaDB.databases) are missing.\n  enable_restore_on_db_failure: # Enables automatic restore if the db is unhealthy.\\\n  disable_binlog_purge_on_rotate: # Boolean to disable binlog purging. Purging is enabled by detault\n  binlog_max_reconnect_attempts: # Number of reconnect attempts by the binlog syncer, default is 10\n  outh:\n    enabled: # enables OAuth to access the API (openID)\\\n    provider_url: # Url of the openID provider (e.g. Dex)\\\n    redirect_url: # OAuth redirect url (this is the url of your mariabackup service)\\\ndatabase: # database config\n    type: # either 'mariadb' or 'postgres'\n    version: # MariaDB version e.g.: \"10.4.0\"\n    full_dump_tool:\n    log_name_format: # prefix of the binlog files\n    user: # user with admin rights (to drop and restart MariaDB)\n    password: # user password\n    host: # host of the MariaDB instance. If running as a sidecar within the MariaDB pod: 127.0.0.1\n    port: # MariaDB port number\n    server_id: # server_uuid/server_id of the binlog syncer connecting to the host\n    data_dir: # data directory of the MariaDB instance\n    databases: # list of databases (used for health checks and restores)\n      - database_name\n      - ...\n    verify_tables: # list of tables for the backup verification check. If none are provided the checksum verification is skipped!\n      - database_name.table_name\n      - ...\nstorages:\n  s3:\n    - name: # name of the storage\n      aws_access_key_id: # s3 access key\n      aws_secret_access_key: # s3 secret access key\n      aws_endpoint:\n      sse_customer_algorithm:\n      s3_force_path_style:\n      sse_customer_key:\n      region: # s3 region\n      bucket_name: # bucket name to save the backup to\n  swift:\n    - name: # name of the storage\n      auth_version: # OpenStack auth version\n      auth_url: # OpenStack auth url (keystone url)\n      user_name: # os user name\n      user_domain_name: # os user domain name\n      project_name: # os project name\n      project_domain_name: # os project domain name\n      password: # os password\n      region: # region name\n      container_name: # name of the container the backups should be store in\n      chunk_size: # default 200mb\n      slo_size: # default 600mb\n  maria_db:\n    - name: # name of the storage\n      host: # host of the MariaDB instance\n      port: # MariaDB port number\n      user: # MariaDB user with admin rights\n      password: # user password\n      full_dump_tool: # dump tool used to restore the full dump\n      databases: # if specified, only the listed databases are replicated\n      parse_schema: # if true, the schema is parsed from the SQL Statement of a QueryEvent\n      dump_filter_buffer_size_mb: # buffer used for reading from dump, default 2mb\n  disk:\n    - name: # name of the storage\n      base_path: # root folder for the backups\n      retention: # backup retention in number of full backups\nverification:\n  interval_in_minutes: # how often are the backups verified\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsapcc%2Fmaria-back-me-up","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsapcc%2Fmaria-back-me-up","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsapcc%2Fmaria-back-me-up/lists"}