{"id":19693863,"url":"https://github.com/aiven-open/myhoard","last_synced_at":"2025-04-05T10:09:03.807Z","repository":{"id":38118218,"uuid":"188801364","full_name":"Aiven-Open/myhoard","owner":"Aiven-Open","description":"MySQL Backup and Point-in-time Recovery service","archived":false,"fork":false,"pushed_at":"2025-03-27T15:27:32.000Z","size":936,"stargazers_count":95,"open_issues_count":15,"forks_count":20,"subscribers_count":64,"default_branch":"master","last_synced_at":"2025-03-29T09:11:39.443Z","etag":null,"topics":["backup","cloud-object-storage","mysql","pitr","restore"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Aiven-Open.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-27T08:18:04.000Z","updated_at":"2025-03-17T09:01:43.000Z","dependencies_parsed_at":"2023-08-09T13:42:52.802Z","dependency_job_id":"41e051d2-92e3-4d6b-8525-e2b4475d554f","html_url":"https://github.com/Aiven-Open/myhoard","commit_stats":{"total_commits":280,"total_committers":30,"mean_commits":9.333333333333334,"dds":0.6892857142857143,"last_synced_commit":"8045f018e05361978d05965243f535b8652af0a0"},"previous_names":["aiven-open/myhoard"],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aiven-Open%2Fmyhoard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aiven-Open%2Fmyhoard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aiven-Open%2Fmyhoard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aiven-Open%2Fmyhoard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Aiven-Open","download_url":"https://codeload.github.com/Aiven-Open/myhoard/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247318745,"owners_count":20919484,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","cloud-object-storage","mysql","pitr","restore"],"created_at":"2024-11-11T19:18:30.589Z","updated_at":"2025-04-05T10:09:03.787Z","avatar_url":"https://github.com/Aiven-Open.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MyHoard [![Build Status](https://github.com/aiven/myhoard/workflows/Build%20MyHoard/badge.svg?branch=master)](https://github.com/aiven/myhoard/actions) [![Codecov](https://codecov.io/gh/aiven/myhoard/branch/main/graph/badge.svg?token=nLr7M7hvCx)](https://codecov.io/gh/aiven/myhoard)\n\nMyHoard is a daemon for creating, managing and restoring MySQL backups.\nThe backup data can be stored in any of the supported cloud object storages.\nIt is functionally similar to [pghoard](https://github.com/aiven/pghoard)\nbackup daemon for PostgreSQL.\n\n# Features\n\n- Automatic periodic full backup\n- Automatic binary log backup in near real-time\n- Cloud object storage support (AWS S3, Google Cloud Storage, Azure)\n- Encryption and compression\n- Backup restoration from object storage\n- Point-in-time-recovery (PITR)\n- Automatic backup history cleanup based on number of backups and/or backup age\n- Purging local binary logs once they're backed up and not needed by other\n  MySQL servers (requires external system to provide executed GTID info for the\n  standby servers)\n- Almost no extra local disk space requirements for creating and restoring\n  backups\n- Incremental backups\n\nFault-resilience and monitoring:\n\n- Handles temporary object storage connectivity issues by retrying all\n  operations\n- Metrics via statsd using [Telegraf® tag extensions](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/statsd)\n- Unexpected exception reporting via Sentry\n- State reporting via HTTP API\n- Full internal state stored on local file system to cope with process and\n  server restarts\n\n# Overview\n\nThere are a number existing tools and scripts for managing MySQL backups so\nwhy have yet another tool? As far as taking a full (or incremental) snapshot of\nMySQL goes, [Percona XtraBackup](https://www.percona.com) does a very\ngood job and is in fact what MyHoard is using internally as well. Where things usually get\nmore complicated is when you want to back up and restore binary logs so that\nyou can do point-in-time recovery and reduce data loss window. Also, as good\nas Percona XtraBackup is for taking and restoring the backup you still need\nall sorts of scripts and timers added around it to actually execute it and if\nanything goes wrong, e.g. because of network issues, it's up to you to retry.\n\nOften binary log backup is based on just uploading the binary log files using\nsome simple scheduled file copying mechanism and restoring them is left as an\nafterthought, usually just comprising of \"download all the binlogs and then\nuse mysqlbinlog to replay them\". In addition to not having proper automation\nto do this to ensure it is repeatable and safe this approach also does not work\nin some cases: In order for binary log restoration with mysqlbinlog to be safe\nyou need to have all binary logs on local disk. For change heavy environments\nthis may be much more than the size of the actual database and if server disk\nis adjusted based on the database size the binary logs may simply not fit on\nthe disk.\n\nMyHoard uses an alternative approach for binary log restoration, which is based\non presenting the backed up binary logs as relay logs in batches via direct\nrelay log index manipulation and having the regular SQL slave thread apply them\nas if they were replicated from another MySQL server. This allows applying them\nin batches so there's very little extra disk space required during restoration\nand this would also allow applying them in parallel (though that requires more\nwork, currently there are known issues with using `slave-parallel-workers`\nvalue other than 0, i.e. multithreading must currently be disabled).\n\nExisting tooling also doesn't pay much attention to real life HA environments\nand failovers where the backup responsibilities need to be switched from one\nserver to another and getting uninterrupted sequence of backed up transactions\nthat can be restored to any point in time, including the time around the\nfailover. This requires something much more sophisticated than just blindly\nuploading all local binary logs.\n\nMyHoard aims to provide a single solution daemon that takes care of all of your\nMySQL backup and restore needs. It handles creating, managing and restoring\nbackups in multi-node setups where master nodes may frequently be going away\n(either because of rolling forward updates or actual server failure). You just\nneed to create a fairly simple configuration file, start the systemd service on\nthe master and any standby servers and make one or two HTTP requests to get the\ndaemon into correct state and it will start automatically doing the right\nthings.\n\n# Basic usage\n\nOn the very first master after you've initialized MySQL database and started up\nMyHoard you'd do this:\n\n```\ncurl -XPUT -H \"Content-Type: application/json\" -d '{\"mode\": \"active\"}' \\\n  http://localhost:16001/status\n```\n\nThis tells MyHoard to switch to active mode where it starts backing up data on\nthis server. If there are no existing backups it will immediately create the\nfirst one.\n\nOn a new standby server you'd first install MySQL and MyHoard but not start or\ninitialize MySQL (i.e. don't do `mysqld --initialize`). After starting the\nMyHoard service you'd do this:\n\n```\ncurl http://localhost:16001/backup  # lists all available backups\ncurl -XPUT -H \"Content-Type: application/json\" \\\n  -d '{\"mode\": \"restore\", \"site\": \"mybackups\", \"stream_id\": \"backup_id\", \"target_time\": null}' \\\n  http://localhost:16001/status\n```\n\nThis tells MyHoard to fetch the given backup, restore it, start the MySQL\nserver once finished, and switch to observe mode where it keeps on observing\nwhat backups are available and what transactions have been backed up but\ndoesn't do any backups itself. Because binary logging is expected to be\nenabled also on the standby server MyHoard does take care of purging any local\nbinary logs that contain only transactions that have been backed up. If you\nwanted to restore to a specific point in time you'd just give a timestamp like\n`\"2019-05-22T11:19:02Z\"` and restoration will be performed up until the last\ntransaction before the target time.\n\nIf the master server fails for any reason you'd do this on one of the standby\nservers:\n\n```\ncurl -XPUT -H \"Content-Type: application/json\" -d '{\"mode\": \"promote\"}' \\\n  http://localhost:16001/status\n```\n\nThis updates the object storage to indicate this server is now the master and\nany updates from the old master should be ignored by any other MyHoard\ninstances. (The old master could still be alive at this point but e.g.\nresponding so slowly that it is considered to be unavailable yet it might be\nable to accept writes and back those up before going totally away and those\ntransactions must be ignored when restoring backups in the future because they\nhave not been replicated to the new master server.) After the initial object\nstorage state update is complete MyHoard switches itself to `active` mode and\nresumes uploading binary logs to the currently active backup stream starting\nfrom the first binary log that contains transactions that have not yet been\nbacked up.\n\n# Requirements\n\nMyHoard requires Python 3.10 or later and some additional components to operate:\n\n- [percona-xtrabackup](https://github.com/percona/percona-xtrabackup)\n- [python3-PyMySQL](https://github.com/PyMySQL/PyMySQL)\n- [python3-rohmu](https://github.com/aiven/rohmu)\n- [MySQL server 8.x+](https://www.oracle.com/mysql/)\n\nCurrently MyHoard only works on Linux and expects MySQL service to be managed\nvia systemd.\n\nMyHoard requires MySQL to be used and configured in a specific manner in order\nfor it to work properly:\n\n- Single writable master, N read only standbys\n- Binary logging enabled both on master and on standbys\n- `binlog_format` set to `ROW`\n- Global transaction identifiers (GTIDs) enabled\n- Use of only InnoDB databases\n\n# Configuration options\n\n`myhoard.json` has an example configuration that shows the structure of the\nconfig file and has reasonable default values for many of the settings. Below\nis full list the settings and the effect of each.\n\n**backup_settings.backup_age_days_max**\n\nMaximum age of backups. Any backup that has been closed (marked as final with\nno more binary logs being uploaded to it) more than this number of days ago\nwill be deleted from storage, unless total number of backups is below the\nminimum number of backups.\n\n**backup_settings.backup_count_max**\n\nMaximum number of backups to keep. Because new backups can be requested\nmanually it is possible to end up with a large number backups. If the total\nnumber goes above this backups will be deleted even if they are not older than\n`backup_age_days_max` days.\n\n**backup_settings.backup_count_min**\n\nMinimum number of backups to keep. If for example the server is powered off and\nthen back on a month later, all existing backups would be very old. However,\nin that case it is usually not desirable to immediately delete all old backups.\nThis setting allows specifying a minimum number of backups that should always\nbe preserved regardless of their age.\n\n**backup_hour**\n\nThe hour of day at which to take new full backup. If backup interval is less\nthan 24 hours this is used as base for calculating the backup times. E.g. if\nbackup interval was 6 hours and backup hour was 4, backups would be taken at\nhours 4, 10, 16 and 22.\n\n**backup_minute**\nThe minute of hour at which to take new full backup.\n\n**backup_interval_minutes**\n\nThe interval in minutes at which to take new backups. Individual binary logs\nare backed up as soon as they're created so there's usually no need to have\nvery frequent full backups.\nNote: If this value is not does not have a factor of 1440 (1 day) then the backup_hour and backup_minute\nsettings cannot be changed once the first backup has been taken, as having a cycle not as a multiple of days\nmeans that the hour and minute of the backup will not be the same each day.\n\n**forced_binlog_rotation_interval**\n\nHow frequently, in seconds, to force creation of new binary log if one hasn't\nbeen created otherwise. This setting ensures that environments with low rate of\nchanges so that new binary logs are not created because the size limit is\nexceeded also get all data backed up frequently enough.\n\n**upload_site**\n\nName of the backup site to which new backups should be created to. See\n`backup_sites` for more information. Only needs to be defined if multiple\nnon-recovery backup sites are present.\n\n**incremental.enabled**\n\nBoolean setting, which controls periodic incremental backups, according to the schedule\n`incremental.full_backup_week_schedule`\n\n**incremental.full_backup_week_schedule**\n\nA string of comma-separated days of the week: `mon`, `tue`, `wed`, `thu`, `fri`, `sat`, `sun`.\nDefines on which days full backups should be taken, thus other days incremental backups will be done.\nE.g. if the value of this setting is `sun,wed`, then full backup is taken on Sundays and Wednesdays, other days of the\nweek incremental backups will be scheduled. Requires `\"incremental.enabled\": true`.\n\n**backup_sites**\n\nObject storage configurations and encryption keys. This is an object with\n`\"site_name\": {\u003csite_config\u003e}` entries. Typically there is only a single\nbackup site but in cases where new server needs to fetch a backup from a\ndifferent location than where it should start writing its own backups there\ncould be two. The backup site name has no relevance for MyHoard and you can\npick whatever names you like.\n\nEach site has the following configuration options:\n\n**backup_sites.{name}.compression**\n\nThe compression method and option to use for binary logs uploaded to this\nsite:\n\n**backup_sites.{name}.compression.algorithm**\n\nOne of the supported compression algorithms: `lzma`, `snappy`, `zstd`.\nDefaults to `snappy`.\n\n**backup_sites.{name}.compression.level**\n\nCompression level for `lzma` or `zstd`.\n\n**backup_sites.{name}.encryption_keys**\n\nThis is an object containing two keys, `public` and `private`. These define\nthe RSA master key used for encrypting/decrypting individual encryption keys\nused for actual data encryption/decryption. The values must be in PEM format.\n\n**backup_sites.{name}.object_storage**\n\nThe object storage configuration for this backup site. Please refer to the\n[PGHoard readme](https://github.com/aiven/pghoard) for details regarding these\nsettings.\n\n**backup_sites.{name}.recovery_only**\n\nThis site is not considered for new backups, it can only be used for recovering\nan existing backup.\n\n**binlog_purge_settings.enabled**\n\nIf `true` MyHoard purges binary logs that are no longer required. The\nrecommended configuration is to have MySQL keep binary logs around for a\nlongish period of time (several days) but have this setting enabled so that\nMyHoard removes the binary logs as soon as they aren't needed anymore.\n\nNote that in order to consider replication to other MySQL servers that are\npart of the cluster the `PUT /replication_state` API must be used to\nperiodically tell MyHoard what transactions other cluster nodes have applied\nto avoid MyHoard purging binary logs that haven't been replicated. This state\nupdate is not strictly required since MySQL will not allow purging binary logs\nthat connected standbys don't yet have but in case standbys get temporarily\ndisconnected relevant binlogs could get purged. Also, in case a standby is\npromoted as new master it should still have any binary log that any other\nstandby hasn't yet received so that when the other standbys start to replicate\nfrom the standby that gets promoted as master they are able to get in sync.\nThis requires MyHoard on the standby to know what transactions other standbys\nhave applied.\n\n**binlog_purge_settings.min_binlog_age_before_purge**\n\nMinimum age of a binary log before purging it is considered. It is advisable to\nset this to some minutes to avoid any race conditions where e.g. new standby\njoins the cluster but MyHoard hasn't yet been informed of its replication\nposition and could end up purging binary log that the new standby could end up\nneeding.\n\n**binlog_purge_settings.purge_interval**\n\nNumber of seconds between checks for binary logs that could be removed.\n\n**binlog_purge_settings.purge_when_observe_no_streams**\n\nAllow purging binary logs when no active backups exist. This setting is mostly\nrelevant for detached read-only replicas that are not configured to take\nbackups of their own. In this case the read replica would see no active backups\nbecause the backup site used by the source service has been specified as\nrecovery only site for the replica. For any other nodes than detached read-only\nreplicas this setting can be set to `false`, for the replicas this should be\n`true` or else MyHoard cannot do any purging at all.\n\n**http_address**\n\nThe IP address to which MyHoard binds its HTTP server. It is highly recommended\nto use local loopback address. MyHoard provides no authentication or TLS\nsupport for the HTTP requests and they should only be made from localhost\nunless you use something like HAProxy to add authentication and encryption.\n\n**http_port**\n\nThe TCP port to which MyHoard binds its HTTP server.\n\n**mysql.binlog_prefix**\n\nThe full path and file name prefix of binary logs. This must be the same as the\ncorresponding MySQL configuration option except full path is always required\nhere.\n\n**mysql.client_params**\n\nThe parameters MyHoard uses to connect to MySQL. Because MyHoard needs to\nperform certain low level operations like manually patch GTID executed value in\n`mysql.gtid_executed` table while restoring data the user account must have\nhigh level of privileges.\n\n**mysql.config_file_name**\n\nFull path of the MySQL server configuration file. This is passed to Percona\nXtraBackup while creating or restoring full database snapshot.\n\n**mysql.data_directory**\n\nFull path of the MySQL data directory. This is currently only used for getting\nfile sizes for reporting and progress tracking purposes.\n\n**mysql.relay_log_index_file**\n\nFull path of the MySQL relay log index file. This must be the same as the\ncorresponding MySQL configuration option except full path is always required\nhere.\n\n**mysql.relay_log_prefix**\n\nThe full path and file name prefix of relay logs. This must be the same as the\ncorresponding MySQL configuration option except full path is always required\nhere.\n\n**restore_auto_mark_backups_broken**\n\nBoolean value. Enables a mechanism to automatically mark backups as broken if restoration fails for\nmultiple times in a row. Defaults to `false`\n\n**restore_free_memory_percentage**\n\nMaximum percentage of system memory to allow xtrabackup to use while\npreparing a basebackup for restoration. If not defined, use xtrabackup's\ndefault value.\n\n**restore_max_binlog_bytes**\n\nMaximum amount of disk space to use for binary logs including pre-fetched logs\nthat are not yet being applied and the binary logs that MySQL is currently\napplying. When Percona XtraBackup is restoring the full database snapshot\nMyHoard starts prefetching binary logs that are needed on top of the full\nsnapshot. Up to this number of bytes are fetched. When the snapshot has been\nrestored MyHoard tells MySQL to start applying the binary logs it has managed\nto pre-fetch thus far and keep on pre-fetching files as long as the total size\nof pre-fetched and files being applied is below the limit (files are deleted as\nsoon as they have been fully applied).\n\nThis should be set to something like 1% of all disk space or at least a few\nhundred MiBs (depending on individual binary log max size] so that sufficient\nnumber of binary logs can be fetched.\n\n**sentry_dsn**\n\nSet this value to Sentry Data Source Name (DSN) string to have any unexpected\nexceptions sent to Sentry.\n\n**server_id**\n\nServer identifier of this node (integer in the range 1 - 0x7FFFFFFF). This must\nmatch the corresponding MySQL configuration option for this node.\n\n**start_command**\n\nCommand used for starting MySQL. This is mostly used for tests. In any\nproduction setups using systemd to manage MySQL server daemon is highly\nrecommended.\n\n**state_directory**\n\nDirectory where to store MyHoard state files. MyHoard stores its full state\ninto a number of separate JSON files.\n\n**statsd**\n\nStatsd configuration used for sending metrics data points. The implementation\nwill send tags along the data points so Telegraf Statsd or other similar\nimplementation that can handle the tags is expected.\n\nThe tags specified here are also reused for Sentry.\n\n**systemctl_command**\n\nThe `systemctl` base command to invoke when MyHoard needs to start or stop\nthe MySQL server. This is only used when restoring a backup where MySQL needs\nto be started after full database snapshot has been recovered and restarted\na couple of times with slightly different settings to allow patching GTID\nexecuted information appropriately.\n\n**systemd_env_update_command**\n\nA command to invoke before `systemctl` to configure MySQL server to use the\ndesired configuration options. This is typically just the built-in\n`myhoard_mysql_env_update` command that writes to MySQL systemd environment\nfile. Separate command is needed to allow running the update as root user.\n\n**systemd_service**\n\nName of the MySQL systemd service.\n\n**temporary_directory**\n\nTemporary directory to use for backup and restore operations. This is currently\nnot used directly by MyHoard but instead passed on to Percona XtraBackup. It is\nrecommended not to use `/tmp` for this because that is an in-memory file\nsystem on many distributions and the exact space requirements for this\ndirectory are not well defined.\n\n**xtrabackup.copy_threads**\n\nNumber of worker threads created by XtraBackup for parallel copying data files when taking a backup. The default value is ``1``.\n\nNote: It is recommended to use more threads for copying than to compress or encrypt.\n\n**xtrabackup.compress_threads**\n\nNumber of worker threads created by XtraBackup for parallel compression when taking a backup. The default value is ``1``.\n\n**xtrabackup.encrypt_threads**\n\nNumber of worker threads created by XtraBackup for parallel encryption when taking a backup. The default value is ``1``.\n\n**xtrabackup.register_redo_log_consumer**\n\nLets XtraBackup register as a redo log consumer at the start of the backup.\nThe server does not remove a redo log that Percona XtraBackup (the consumer) has not yet copied.\nThe consumer reads the redo log and manually advances the log sequence number (LSN).\nThe server blocks the writes during the process. Based on the redo log consumption,\nthe server determines when it can purge the log. It is disabled by default.\n\n# HTTP API\n\nMyHoard provides an HTTP API for managing the service. The various entry points\nare described here.\n\nAll APIs return a response like this on error:\n\n```\n{\n  \"message\": \"details regarding what went wrong\"\n}\n```\n\n## GET /backup\n\nLists all available backups. This call takes no request parameters. Response\nformat is as follows:\n\n```\n{\n  \"backups\": [\n    {\n      \"basebackup_info\": {\n        ...\n      },\n      \"closed_at\": \"2019-05-23T06:29:10.041489Z\",\n      \"completed_at\": \"2019-05-22T06:29:20.582302Z\",\n      \"recovery_site\": false,\n      \"resumable\": true,\n      \"site\": \"{backup_site_name}\",\n      \"stream_id\": \"{backup_identifier}\"\n    }\n  ]\n}\n```\n\nIn case MyHoard has not yet finished fetching the list of backups the root\nlevel `\"backups\"` value will be `null`.\n\n**basebackup_info**\n\nThis contains various details regarding the full snapshot that is the basis of\nthis backup.\n\n**closed_at**\n\nThe time at which last binary log to this stream was uploaded and no more\nuploads are expected. The backup can resume to any point in time between\n`closed_at` and `completed_at`. If `closed_at` is `null` the backup is\nactive and new binary logs are still being uploaded to it.\n\n**recovery_site**\n\nTells whether the backup site this backup is stored into is recovery only.\n\n**resumable**\n\nTells whether the backup is in a state that another server can continue backing\nup data to it. When current master server starts a new backup it first needs to\nupload the initial full snapshot and some associated metadata. Before that is\ndone no other server could possibly do anything useful with that backup. Once\nthese steps are completed then if the master fails for any reason and one of\nthe standbys are promoted as new master the newly promoted master can continue\nuploading binary logs to the existing active backup. If the backup is not\nresumable the new master needs to discard it and start new backup from scratch.\n\n**site**\n\nName of the backup site this backup is stored into.\n\n**stream_id**\n\nIdentifier of this backup.\n\n## POST /backup\n\nCreate new full backup or force binary log rotation and back up the latest\nbinary log file. Request body must be like this:\n\n```\n{\n  \"backup_type\": \"{basebackup|binlog}\",\n  \"wait_for_upload\": 3.0,\n  \"incremental\": false\n}\n```\n\n**backup_type**\n\nThis specifies the kind of backup to perform. If set to `basebackup` a new\nfull backup is created and old backup is closed once that is complete. If set\nto `binlog` this rotates currently active binary log so that a finished\nbinary log file with all current transactions is created and that file is then\nbacked up.\n\n**wait_for_upload**\n\nThis is only valid in case `backup_type` is `binlog`. In that case the\noperation will block for up to as many seconds as specified by this parameter\nfor the binary log upload to complete before returning.\n\n**incremental**\n\nBoolean value, when `true` - requests incremental base backup if possible.\nSetting is used only in combination with `basebackup` backup_type.\nDefaults to `false`.\n\nResponse on success looks like this:\n\n```\n{\n  \"success\": true\n}\n```\n\n\n## PUT /backup/{stream_id}/preserve\n\nCreate a request for updating the preservation status of a backup. Request body must be like this:\n\n```\n{\n  \"preserve_until\": \"2023-09-01T00:00:0000\",\n  \"wait_for_applied_preservation\": 3.0,\n}\n```\n\n**stream_id**\n\nIdentifier of this backup.\n\n**preserve_until**\n\nOptional datetime value in ISO format for keeping the backup from being deleted.\nIf a valid value is provided, the backup will be preserved until the specified datetime.\nIf not provided or has a null value, the backup can be deleted due to old age.\n\n**wait_for_applied_preservation**\n\nOptional amount of time to the wait for the preservation request to be effectively executed. The\noperation will block for up to as many seconds as specified by this parameter.\n\n\nResponse on success looks like this:\n\n```\n{\n  \"success\": true\n}\n```\n\n\n## PUT /replication_state\n\nThis call can be used to inform MyHoard of the executed GTIDs on other servers\nin the cluster to allow MyHoard to only purge binlogs that have been fully\napplied on all cluster nodes. Request body must be like this:\n\n```\n{\n  \"server1\": {\n    \"206375d9-ec5a-46b7-bb26-b621812e7471\": [[1, 100]],\n    \"131a1f4d-fb7a-44fe-94b9-5508445aa126\": [[1, 5858]],\n  },\n  \"server2\": {\n    \"206375d9-ec5a-46b7-bb26-b621812e7471\": [[1, 100]],\n    \"131a1f4d-fb7a-44fe-94b9-5508445aa126\": [[1, 5710]],\n  }\n}\n```\n\nThe main level object must have an entry for each server in the cluster. The\nnames of the servers are not relevant as long as they are used consistently.\nFor each of the servers the value is an object that contains server UUID as the\nkey and list of GNO start end ranges as the value. The server UUID is the\noriginal server from which the transactions originated, not the UUID of the\nserver reporting the numbers. In the example both servers `server1` and\n`server2` have executed transactions 206375d9-ec5a-46b7-bb26-b621812e7471:1-100\nand both have executed some part of transactions from server\n131a1f4d-fb7a-44fe-94b9-5508445aa126 but `server1` is further ahead, having\nexecuted GNOs up until 5858 while `server2` is only at 5710.\n\nNote that it is not expected to have multiple masters active at the same time.\nMultiple server UUIDs exist when old master servers have been replaced.\n\nResponse on success echoes back the same data sent in the request.\n\n## GET /status\n\nReturns current main mode of MyHoard. Response looks like this:\n\n```\n{\n  \"mode\": \"{active|idle|observe|promote|restore}\"\n}\n```\n\nSee the status update API for more information regarding the different modes.\n\n## PUT /status\n\nUpdates current main mode of MyHoard. Request must be like this:\n\n```\n{\n  \"force\": false,\n  \"mode\": \"{active|idle|observe|promote|restore}\",\n  \"site\": \"{site_name}\",\n  \"stream_id\": \"{backup_id}\",\n  \"target_time\": null,\n  \"target_time_approximate_ok\": false\n}\n```\n\n**force**\n\nThis value can only be passed when switching mode to active.\n\nWhen binary log restoration is ongoing setting this true will cause mode to\nbe forcibly switched to promote without waiting for all binary logs to get\napplied and the promote phase will skip the step of ensuring all binary logs\nare applied. If mode is already promote and binary logs are being applied in\nthat state, the binary logs sync is considered to be immediately complete.\nIf the server is not currently applying binary logs passing `\"force\": true`\nwill cause the operation to fail with error 400.\n\nThis parameter is only intended for exceptional situations. For example\nbroken binary logs that cannot be applied and it is preferable to promote the\nserver in it's current state. Another possible case is binary logs containing\nchanges to large tables without primary keys with row format in use and the\noperation being so slow that it will not complete in reasonable amount of time.\nData loss will incur when using this option!\n\n**mode**\n\nMyHoard initially starts in mode `idle`. In this state it only fetches the\navailable backups but doesn't actively do anything else. From `idle` state it\nis possible to switch to `restore` or `active` states. Only the very first\nserver in a new cluster should be switched to `active` state directly from\n`idle` state. All other servers must first be switched to `restore` and\nonly after restoration has finished should other state changes be performed.\n\nWhen restore operation completes MyHoard automatically transitions to mode\n`observe`, in which it keeps track of backups managed by other servers in the\ncluster but doesn't actively back up anything. If this node should be the new\nmaster (or new separate forked service) then mode must be switched to\n`promote` once MyHoard has changed it from `restore` to `observe`. This\nwill make MyHoard update metadata in object storage appropriately before\nautomatically transitioning to state `active`.\n\nServers in state `active` cannot be transitioned to other states. They are\nthe active master node and MyHoard on the node must just be deactivated if the\nserver should stop acting in that role.\n\n**site**\n\nThis is only applicable when new mode is `restore`. Identifies the site\ncontaining the backup to restore. Use `GET /backup` to list all available\nbackups.\n\n**stream_id**\n\nThis is only applicable when new mode is `restore`. Identifies the backup\nto restore. Use `GET /backup` to list all available backups.\n\n**target_time**\n\nThis is only applicable when new mode is `restore`. If this is omitted or\n`null` the last available transaction for the given backup is restored. When\nthis is defined restoration is performed up until the last transaction before\nthis time. Must be ISO 8601 timestamp. If the requested time is not available\nin the given timestamp (time is not between the `completed_at` and\n`closed_at` timestamps) the request will fail.\n\n**target_time_approximate_ok**\n\nThis is only applicable when new mode is `restore` and `target_time` has\nbeen specified. If this is set to `true` then `target_time` is only used to\nrestrict results on individual binary log level. That is, the restore process\nis guaranteed not to restore binary logs whose first transaction is later than\nthe given target time but the last file that is picked for restoration is fully\napplied even if that means applying some transactions that are more recent than\nthe target time.\n\nThis mode is useful when restoring potentially large number of binary logs and\nthe exact target time is not relevant. Enabling this mode avoids having to use\nthe `UNTIL SQL_AFTER_GTIDS = x` parameter for the SQL thread. The `UNTIL`\nmodifier forces single threaded apply and on multi-core machines makes the\nrestoration slower. The single threaded mode only applies for the last batch\nbut that too can be very large and setting this value can significantly reduce\nthe restoration time.\n\n## GET /status/restore\n\nIf current mode is `restore` this API can be used to get details regarding\nrestore progress. If mode is something else the request will fail with HTTP\nstatus 400. For successful requests the response body looks like this:\n\n```\n{\n  \"basebackup_compressed_bytes_downloaded\": 8489392354,\n  \"basebackup_compressed_bytes_total\": 37458729461,\n  \"binlogs_being_restored\": 0,\n  \"binlogs_pending\": 73,\n  \"binlogs_restored\": 0,\n  \"phase\": \"{current_phase}\"\n}\n```\n\n**basebackup_compressed_bytes_downloaded**\n\nNumber of compressed bytes of the full snapshot that have been downloaded so\nfar. Decryption and decompression is performed on the fly so these bytes have\nalso been processed.\n\n**basebackup_compressed_bytes_total**\n\nTotal number of (compressed) bytes in the full snapshot.\n\n**binlogs_being_restored**\n\nNumber of binary logs currently passed on to MySQL to restore.\n\n**binlogs_pending**\n\nBinary logs that are pending to be restored. Note that this number may be\ngoing up if there is currently an active master node that is uploading new\nbinary logs to the backup being restored and there is no recovery target time\ngiven.\n\n**binlogs_restored**\n\nNumber of binary logs that have been successfully applied.\n\n**phase**\n\nCurrent phase of backup restoration. Possible options are these:\n\n- getting_backup_info: Backup metadata is being fetched to determine what\n  exactly needs to be restored.\n- initiating_binlog_downloads: Binary log prefetch operations are being\n  scheduled so that progress with those can be made while the full snapshot is\n  being restored.\n- restoring_basebackup: The full snapshot is being downloaded and prepared.\n- rebuilding_tables: Rebuilding tables before restoring binary logs. This can\n  avoid data corruption when updating from older MySQL versions.\n- refreshing_binlogs: Refreshing binary log info to see if new binary logs have\n  been uploaded to object storage from current master. This and the other\n  binlog related phases are typically entered multiple times as the binlogs are\n  handled in batches.\n- applying_binlogs: Refreshing the list of binary logs MySQL should be\n  restoring.\n- waiting_for_apply_to_finish: Waiting for MySQL to finish applying current\n  subset of binary logs.\n- finalizing: Performing final steps to complete backup restoration.\n- completed: The operation has completed. This is typically not returned via\n  the API because MyHoard will automatically switch to `observe` mode when\n  restoration completes and the restoration status is not available in that\n  mode.\n- failed: Restoring the backup failed. The operation will be retried\n  automatically, but it may fail repeatedly and analyzing logs to get more\n  details regarding the failure is advisable.\n- failed_basebackup: Terminal state for a RestoreCoordinator instance but\n  restoring an earlier backup may be an option.\n\n# Metrics\n\nThe following metrics are exported by myhoard:\n\n**myhoard.backup_stream.basebackup_bytes_uploaded**\n**myhoard.backup_stream.basebackup_requested**\n**myhoard.backup_stream.basebackup_upload_rate**\n**myhoard.backup_stream.binlog_upload_rate**\n**myhoard.backup_stream.errors**\n**myhoard.basebackup.bytes_compressed**\n**myhoard.basebackup.bytes_uncompressed**\n**myhoard.basebackup.compression_ratio**\n**myhoard.basebackup.errors**\n**myhoard.basebackup.estimated_progress**\n**myhoard.basebackup.optimize_table**\n**myhoard.basebackup.xtrabackup_backup**\n**myhoard.basebackup_broken**\n**myhoard.basebackup_restore.xbstream_extract**\n**myhoard.basebackup_restore.xtrabackup_move**\n**myhoard.basebackup_restore.xtrabackup_prepare**\n**myhoard.binlog.count**\n**myhoard.binlog.count_new**\n**myhoard.binlog.remote_copy**\n**myhoard.binlog.removed**\n**myhoard.binlog.size**\n**myhoard.binlog.size_new**\n**myhoard.binlog.upload**\n**myhoard.binlog.upload_errors**\n**myhoard.disk_full_errors**\n**myhoard.generic_errors**\n**myhoard.http.{name}**\n**myhoard.network_errors**\n**myhoard.pending_binlog_bytes_max**\n**myhoard.pending_binlog_bytes_min**\n**myhoard.pending_binlog_count_max**\n**myhoard.pending_binlog_count_min**\n**myhoard.ratetracker.errors**\n**myhoard.remote_read_errors**\n**myhoard.remote_write_errors**\n**myhoard.restore.basebackup_bytes_downloaded**\n**myhoard.restore.binlogs_restored**\n**myhoard.restore.cannot_reset**\n**myhoard.restore.change_master_to_failed**\n**myhoard.restore.pending_binlogs**\n**myhoard.restore.unexpected_extra_relay_log**\n**myhoard.restore_errors**\n\n# Running container-based tests\n\nMake sure docker is installed (podman currently untested) and just run:\n\n```\nmake PYTHON_VERSION=3.11 PERCONA_VERSION=8.0.30-23-1.bullseye MYSQL_VERSION=8.0.30 build-setup-specific-image\n\nmake dockertest\n```\n\nIf you don't need to change percona or mysql or python version, but you want to change myhoard source code and re-test,\nrun:\n\n```\nmake dockertest-resync\n```\n\nto re-sync the source code from the host and re-run the tests.\n\nTake a look at `.github/workflows/build.yaml` for possible version values.\n\nIn order to locally launch a single test (again while resyncing from current source code), you can use `pytest-quick` e.g.\n\n```\nmake PYTEST_ARGS=\"-k test_3_node_service_failover_and_restore\" dockertest-pytest\n```\n\n# Running tests natively\n\nRunning native tests must NOT be performed as root (requires additional options for mysql)\n\n# Test environment setup: Debian/Ubuntu\n\nRun:\n\n```bash\nMYSQL_VERSION=8.0.30\nPERCONA_VERSION=8.0.30-23-1.bullseye\nmake build-dep-ubuntu\nmake clean\nscripts/remove-default-mysql\nscripts/install-mysql-packages ${MYSQL_VERSION}\nscripts/setup-percona-repo\nscripts/install-percona-package ${PERCONA_VERSION}\nscripts/install-python-deps\npip3 install -e .\n```\n\nNote that since Percona-xtrabackup does not work with a version of MySQL newer\nthan Percona-xtrabackup, the version strings should match.\n\nThis command will install all the required package version. Please note: the state of your environment\nWILL change with this command. Both native and Python packages will be installed.\n\n# Test environment setup: Fedora\n\nrun:\n\n`make build-dep-fedora`\n\n(this can install or change packages on your host system)\n\n# Running tests\n\nOnce the environment setup is over, you can execute\n\n`make PYTHON_VERSION=3.11 coverage`\n\nAnd have all test run, or just\n\n`python${PYTHON_VERSION} -m pytest \"$@\"`\n\nSetting `PYTHON_VERSION` is optional, but make sure you're using the same interpreter that was employed during setup, otherwise\nyou may encounter runtime errors.\n\n# License\n\nMyHoard is licensed under the Apache License, Version 2.0. Full license text\nis available in the ``LICENSE`` file and at\nhttp://www.apache.org/licenses/LICENSE-2.0.txt\n\n# Contact\n\nBug reports and patches are very welcome, please post them as GitHub issues\nand pull requests at https://github.com/aiven/myhoard. Any possible\nvulnerabilities or other serious issues should be reported directly to the\nmaintainers \u003copensource@aiven.io\u003e.\n\n# Trademarks\n\nMySQL is a registered trademark of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.\n\nTelegraf® is a trademark [registered / owned] by InfluxData, which is not affiliated with, and does not endorse, this product.\n\nPercona Xtrabackup is a trademark and property of its respective owners. All product and service names used in this website are for identification purposes only and do not imply endorsement.\n\n# Credits\n\nMyHoard was created by, and is maintained by, [Aiven](https://aiven.io) cloud\ndata hub developers.\n\nRecent contributors are listed on the GitHub project page,\nhttps://github.com/aiven/myhoard/graphs/contributors\n\nMyHoard uses [Percona Xtrabackup](https://www.percona.com) for creating and\nrestoring database snapshot excluding binary logs.\n\nCopyright ⓒ 2019 Aiven Ltd.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faiven-open%2Fmyhoard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faiven-open%2Fmyhoard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faiven-open%2Fmyhoard/lists"}