{"id":13570994,"url":"https://github.com/Altinity/clickhouse-backup","last_synced_at":"2025-04-04T07:32:31.557Z","repository":{"id":37492883,"uuid":"150444746","full_name":"Altinity/clickhouse-backup","owner":"Altinity","description":"Tool for easy backup and restore for ClickHouse® using object storage for backup files. ","archived":false,"fork":false,"pushed_at":"2025-04-02T17:07:01.000Z","size":6019,"stargazers_count":1349,"open_issues_count":40,"forks_count":232,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-04-03T06:43:42.688Z","etag":null,"topics":["azblob","backup","clickhouse","clickhouse-backup","clickhousedump","dump","ftp","gcs","s3","sftp"],"latest_commit_sha":null,"homepage":"https://altinity.com","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Altinity.png","metadata":{"files":{"readme":"ReadMe.md","changelog":"ChangeLog.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-09-26T15:00:57.000Z","updated_at":"2025-04-02T16:39:41.000Z","dependencies_parsed_at":"2023-10-03T18:43:51.246Z","dependency_job_id":"f208a18a-0271-4fa3-942c-c1161aea6d0c","html_url":"https://github.com/Altinity/clickhouse-backup","commit_stats":null,"previous_names":["altinity/clickhouse-backup","alexakulov/clickhouse-backup"],"tags_count":175,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclickhouse-backup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclickhouse-backup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclickhouse-backup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclickhouse-backup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Altinity","download_url":"https://codeload.github.com/Altinity/clickhouse-backup/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247139261,"owners_count":20890196,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azblob","backup","clickhouse","clickhouse-backup","clickhousedump","dump","ftp","gcs","s3","sftp"],"created_at":"2024-08-01T14:00:57.324Z","updated_at":"2025-04-04T07:32:31.549Z","avatar_url":"https://github.com/Altinity.png","language":"Go","funding_links":[],"categories":["Go","Ops"],"sub_categories":["ETL and Data Processing"],"readme":"\n# Altinity Backup for ClickHouse®\n\n[![Build](https://github.com/Altinity/clickhouse-backup/actions/workflows/build.yaml/badge.svg?branch=master)](https://github.com/Altinity/clickhouse-backup/actions/workflows/build.yaml)\n[![GoDoc](https://godoc.org/github.com/Altinity/clickhouse-backup?status.svg)](http://godoc.org/github.com/Altinity/clickhouse-backup)\n[![Telegram](https://img.shields.io/badge/telegram-join%20chat-3796cd.svg)](https://t.me/clickhousebackup)\n[![Docker Image](https://img.shields.io/docker/pulls/altinity/clickhouse-backup.svg)](https://hub.docker.com/r/altinity/clickhouse-backup)\n[![Downloads](https://img.shields.io/github/downloads/Altinity/clickhouse-backup/total.svg)](http://github.com/Altinity/clickhouse-backup/releases)\n[![Coverage Status](https://coveralls.io/repos/github/Altinity/clickhouse-backup/badge.svg)](https://coveralls.io/github/Altinity/clickhouse-backup)\n\u003ca href=\"https://altinity.com/slack\"\u003e\n  \u003cimg src=\"https://img.shields.io/static/v1?logo=slack\u0026logoColor=959DA5\u0026label=Slack\u0026labelColor=333a41\u0026message=join%20conversation\u0026color=3AC358\" alt=\"AltinityDB Slack\" /\u003e\n\u003c/a\u003e\n\nA tool for easy backup and restore utility for ClickHouse databases with support for many cloud and non-cloud storage types.\n\n### Don't run `clickhouse-backup` remotely\nTo backup data, `clickhouse-backup` requires access to the same files as `clickhouse-server` in `/var/lib/clickhouse` folders.\nFor that reason, it's required to run `clickhouse-backup` on the same host or same Kubernetes Pod or the neighbor container on the same host where `clickhouse-server` ran.\n**WARNING** You can backup only schema when connect to remote `clickhouse-server` hosts.\n\n## Features\n\n- Easy creating and restoring backups of all or specific tables\n- Efficient storing of multiple backups on the file system\n- Uploading and downloading with streaming compression\n- Works with AWS, GCS, Azure, Tencent COS, FTP, SFTP\n- **Support for Atomic Database Engine**\n- **Support for multi disks installations**\n- **Support for custom remote storage types via `rclone`, `kopia`, `restic`, `rsync` etc**\n- Support for incremental backups on remote storage\n\n## Limitations\n\n- ClickHouse above 1.1.54394 is supported\n- Only MergeTree family tables engines (more table types for `clickhouse-server` 22.7+ and `USE_EMBEDDED_BACKUP_RESTORE=true`)\n\n## Community\n\nAltinity Backup for ClickHouse is a community effort sponsored by Altinity. The best way to reach us or ask questions is:\n\n* Join the [Altinity Slack](https://altinity.com/slack) - Chat with the developers and other users\n* Log an [issue on GitHub](https://github.com/Altinity/clickhouse-backup/issues) - Ask questions, log bugs and feature requests\n\n## Support \n\nAltinity is the primary maintainer of clickhouse-backup. We offer a range of software and \nservices related to ClickHouse. \n\n- [Official website](https://altinity.com/) - Get a high level overview of Altinity and our offerings.\n- [Altinity.Cloud](https://altinity.com/cloud-database/) - Run ClickHouse in our cloud or yours.\n- [Altinity Support](https://altinity.com/support/) - Get Enterprise-class support for ClickHouse.\n- [Slack](https://altinity.com/slack) - Talk directly with ClickHouse users and Altinity devs.\n- [Contact us](https://hubs.la/Q020sH3Z0) - Contact Altinity with your questions or issues.\n- [Free consultation](https://hubs.la/Q020sHkv0) - Get a free consultation with a ClickHouse expert today.\n\n## Installation\n\nDownload the latest binary from the [releases](https://github.com/Altinity/clickhouse-backup/releases) page and decompress with:\n\n```shell\ntar -zxvf clickhouse-backup.tar.gz\n```\n\nUse the official tiny Docker image and run it on a host with `clickhouse-server` installed:\n\n```shell\ndocker run -u $(id -u clickhouse) --rm -it --network host -v \"/var/lib/clickhouse:/var/lib/clickhouse\" \\\n   -e CLICKHOUSE_PASSWORD=\"password\" \\\n   -e S3_BUCKET=\"clickhouse-backup\" \\\n   -e S3_ACCESS_KEY=\"access_key\" \\\n   -e S3_SECRET_KEY=\"secret\" \\\n   altinity/clickhouse-backup --help\n```\n\nBuild from the sources (required go 1.21+):\n\n```shell\nGO111MODULE=on go install github.com/Altinity/clickhouse-backup/v2/cmd/clickhouse-backup@latest\n```\n\n## Brief description of how clickhouse-backup works\n\nData files are immutable in the `clickhouse-server`.\nDuring a backup operation, `clickhouse-backup` creates file system hard links to existing `clickhouse-server` data parts via executing the `ALTER TABLE ... FREEZE` query.\nDuring the restore operation, `clickhouse-backup` copies the hard links to the `detached` folder and executes the `ALTER TABLE ... ATTACH PART` query for each data part and each table in the backup.\nA more detailed description is available here: https://www.youtube.com/watch?v=megsNh9Q-dw\n\n## Default Config File\n\nBy default, the config file is located at `/etc/clickhouse-backup/config.yml`, but it can be redefined via the `CLICKHOUSE_BACKUP_CONFIG` environment variable or via `--config` command line parameter.\nAll options can be overwritten via environment variables.\nUse `clickhouse-backup default-config` to print the default config.\n\n## Configurable Parameters\n\nUse `clickhouse-backup print-config` to print the current config.\nEnvironment variables can override each config parameter defined in the config file. Their names should be UPPERCASE, and exact names are provided after the comment character `#.`\nThe following values are not defaults; they explain what each config parameter with an example.\n\n```yaml\ngeneral:\n  remote_storage: none           # REMOTE_STORAGE, choice from: `azblob`,`gcs`,`s3`, etc; if `none` then `upload` and `download` commands will fail.\n  max_file_size: 1073741824      # MAX_FILE_SIZE, 1G by default, useless when upload_by_part is true, use to split data parts files by archives\n  backups_to_keep_local: 0       # BACKUPS_TO_KEEP_LOCAL, how many latest local backup should be kept, 0 means all created backups will be stored on local disk\n                                 # -1 means backup will keep after `create` but will delete after `create_remote` command\n                                 # You can run `clickhouse-backup delete local \u003cbackup_name\u003e` command to remove temporary backup files from the local disk\n  backups_to_keep_remote: 0      # BACKUPS_TO_KEEP_REMOTE, how many latest backup should be kept on remote storage, 0 means all uploaded backups will be stored on remote storage.\n                                 # If old backups are required for newer incremental backup then it won't be deleted. Be careful with long incremental backup sequences.\n  log_level: info                # LOG_LEVEL, a choice from `debug`, `info`, `warning`, `error`\n  allow_empty_backups: false     # ALLOW_EMPTY_BACKUPS\n  # Concurrency means parallel tables and parallel parts inside tables\n  # For example, 4 means max 4 parallel tables and 4 parallel parts inside one table, so equals 16 concurrent streams\n  download_concurrency: 1        # DOWNLOAD_CONCURRENCY, max 255, by default, the value is round(sqrt(AVAILABLE_CPU_CORES / 2))\n  upload_concurrency: 1          # UPLOAD_CONCURRENCY, max 255, by default, the value is round(sqrt(AVAILABLE_CPU_CORES / 2))\n  \n  # Throttling speed for upload and download, calculates on part level, not the socket level, it means short period for high traffic values and then time to sleep \n  download_max_bytes_per_second: 0  # DOWNLOAD_MAX_BYTES_PER_SECOND, 0 means no throttling \n  upload_max_bytes_per_second: 0    # UPLOAD_MAX_BYTES_PER_SECOND, 0 means no throttling\n  \n  # when table data contains in system.disks with type=ObjectStorage, then we need execute remote copy object in object storage service provider, this parameter can restrict how many files will copied in parallel  for each table \n  object_disk_server_side_copy_concurrency: 32\n  # when CopyObject failure or object disk storage and backup destination have incompatible, will warning about possible high network traffic \n  allow_object_disk_streaming: false\n  \n  # RESTORE_SCHEMA_ON_CLUSTER, execute all schema related SQL queries with `ON CLUSTER` clause as Distributed DDL.\n  # Check `system.clusters` table for the correct cluster name, also `system.macros` can be used.\n  # This isn't applicable when `use_embedded_backup_restore: true`\n  restore_schema_on_cluster: \"\"\n  upload_by_part: true           # UPLOAD_BY_PART\n  download_by_part: true         # DOWNLOAD_BY_PART\n  use_resumable_state: true      # USE_RESUMABLE_STATE, allow resume upload and download according to the \u003cbackup_name\u003e.resumable file. Resumable state is not supported for custom method in remote storage.\n\n  # RESTORE_DATABASE_MAPPING, restore rules from backup databases to target databases, which is useful when changing destination database, all atomic tables will be created with new UUIDs.\n  # The format for this env variable is \"src_db1:target_db1,src_db2:target_db2\". For YAML please continue using map syntax\n  restore_database_mapping: {}\n\n  # RESTORE_TABLE_MAPPING, restore rules from backup tables to target tables, which is useful when changing destination tables.\n  # The format for this env variable is \"src_table1:target_table1,src_table2:target_table2\". For YAML please continue using map syntax\n  restore_table_mapping: {}\n\n  retries_on_failure: 3          # RETRIES_ON_FAILURE, how many times to retry after a failure during upload or download\n  retries_pause: 5s              # RETRIES_PAUSE, duration time to pause after each download or upload failure\n\n  watch_interval: 1h       # WATCH_INTERVAL, use only for `watch` command, backup will create every 1h\n  full_interval: 24h       # FULL_INTERVAL, use only for `watch` command, full backup will create every 24h\n  watch_backup_name_template: \"shard{shard}-{type}-{time:20060102150405}\" # WATCH_BACKUP_NAME_TEMPLATE, used only for `watch` command, macros values will apply from `system.macros` for time:XXX, look format in https://go.dev/src/time/format.go\n\n  sharded_operation_mode: none       # SHARDED_OPERATION_MODE, how different replicas will shard backing up data for tables. Options are: none (no sharding), table (table granularity), database (database granularity), first-replica (on the lexicographically sorted first active replica). If left empty, then the \"none\" option will be set as default.\n  \n  cpu_nice_priority: 15    # CPU niceness priority, to allow throttling CPU intensive operation, more details https://manpages.ubuntu.com/manpages/xenial/man1/nice.1.html\n  io_nice_priority: \"idle\" # IO niceness priority, to allow throttling DISK intensive operation, more details https://manpages.ubuntu.com/manpages/xenial/man1/ionice.1.html\n  \n  rbac_backup_always: true # always, backup RBAC objects\n  rbac_resolve_conflicts: \"recreate\"  # action, when RBAC object with the same name already exists, allow \"recreate\", \"ignore\", \"fail\" values\nclickhouse:\n  username: default                # CLICKHOUSE_USERNAME\n  password: \"\"                     # CLICKHOUSE_PASSWORD\n  host: localhost                  # CLICKHOUSE_HOST, To make backup data `clickhouse-backup` requires access to the same file system as clickhouse-server, so `host` should localhost or address of another docker container on the same machine, or IP address bound to some network interface on the same host.\n  port: 9000                       # CLICKHOUSE_PORT, don't use 8123, clickhouse-backup doesn't support HTTP protocol\n  # CLICKHOUSE_DISK_MAPPING, use this mapping when your `system.disks` are different between the source and destination clusters during backup and restore process.\n  # The format for this env variable is \"disk_name1:disk_path1,disk_name2:disk_path2\". For YAML please continue using map syntax.\n  # If destination disk is different from source backup disk then you need to specify the destination disk in the config file:\n\n  # disk_mapping:\n  #  disk_destination: /var/lib/clickhouse/disks/destination\n  \n  # `disk_destination`  needs to be referenced in backup (source config), and all names from this map (`disk:path`) shall exist in `system.disks` on destination server.\n  # During download of the backup from remote location (s3), if `name` is not present in `disk_mapping` (on the destination server config too) then `default` disk path will used for download.\n  # `disk_mapping` is used to understand during download where downloaded parts shall be unpacked (which disk) on destination server and where to search for data parts directories during restore.\n  disk_mapping: {}\n  # CLICKHOUSE_SKIP_TABLES, the list of tables (pattern are allowed) which are ignored during backup and restore process\n  # The format for this env variable is \"pattern1,pattern2,pattern3\". For YAML please continue using list syntax\n  skip_tables:\n    - system.*\n    - INFORMATION_SCHEMA.*\n    - information_schema.*\n  # CLICKHOUSE_SKIP_TABLE_ENGINES, the list of tables engines which are ignored during backup, upload, download, restore process\n  # The format for this env variable is \"Engine1,Engine2,engine3\". For YAML please continue using list syntax\n  skip_table_engines: []\n  # CLICKHOUSE_SKIP_DISKS, list of disk names which are ignored during create, upload, download and restore command\n  # The format for this env variable is \"Engine1,Engine2,engine3\". For YAML please continue using list syntax\n  skip_disks: []\n  # CLICKHOUSE_SKIP_DISK_TYPES, list of disk types which are ignored during create, upload, download and restore command\n  # The format for this env variable is \"Engine1,Engine2,engine3\". For YAML please continue using list syntax\n  skip_disk_types: []\n  timeout: 5m                  # CLICKHOUSE_TIMEOUT\n  freeze_by_part: false        # CLICKHOUSE_FREEZE_BY_PART, allow freezing by part instead of freezing the whole table\n  freeze_by_part_where: \"\"     # CLICKHOUSE_FREEZE_BY_PART_WHERE, allow parts filtering during freezing when freeze_by_part: true\n  secure: false                # CLICKHOUSE_SECURE, use TLS encryption for connection\n  skip_verify: false           # CLICKHOUSE_SKIP_VERIFY, skip certificate verification and allow potential certificate warnings\n  sync_replicated_tables: true # CLICKHOUSE_SYNC_REPLICATED_TABLES\n  tls_key: \"\"                  # CLICKHOUSE_TLS_KEY, filename with TLS key file\n  tls_cert: \"\"                 # CLICKHOUSE_TLS_CERT, filename with TLS certificate file\n  tls_ca: \"\"                   # CLICKHOUSE_TLS_CA, filename with TLS custom authority file\n  log_sql_queries: true        # CLICKHOUSE_LOG_SQL_QUERIES, logging `clickhouse-backup` SQL queries on `info` level, when true, `debug` level when false\n  debug: false                 # CLICKHOUSE_DEBUG\n  config_dir:      \"/etc/clickhouse-server\"              # CLICKHOUSE_CONFIG_DIR\n  # CLICKHOUSE_RESTART_COMMAND, use this command when restoring with --rbac, --rbac-only or --configs, --configs-only options\n  # will split command by ; and execute one by one, all errors will logged and ignore\n  # available prefixes\n  # - sql: will execute SQL query\n  # - exec: will execute command via shell\n  restart_command: \"exec:systemctl restart clickhouse-server\" \n  ignore_not_exists_error_during_freeze: true # CLICKHOUSE_IGNORE_NOT_EXISTS_ERROR_DURING_FREEZE, helps to avoid backup failures when running frequent CREATE / DROP tables and databases during backup, `clickhouse-backup` will ignore `code: 60` and `code: 81` errors during execution of `ALTER TABLE ... FREEZE`\n  check_replicas_before_attach: true # CLICKHOUSE_CHECK_REPLICAS_BEFORE_ATTACH, helps avoiding concurrent ATTACH PART execution when restoring ReplicatedMergeTree tables\n  default_replica_path: \"/clickhouse/tables/{cluster}/{shard}/{database}/{table}\" # CLICKHOUSE_DEFAULT_REPLICA_PATH, will use during restore Replicated tables without macros in replication_path if replica already exists, to avoid restoring conflicts\n  default_replica_name: \"{replica}\" # CLICKHOUSE_DEFAULT_REPLICA_NAME, will use during restore Replicated tables without macros in replica_name if replica already exists, to avoid restoring conflicts\n  use_embedded_backup_restore: false # CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE, use BACKUP / RESTORE SQL statements instead of regular SQL queries to use features of modern ClickHouse server versions\n  embedded_backup_disk: \"\"  # CLICKHOUSE_EMBEDDED_BACKUP_DISK - disk from system.disks which will use when `use_embedded_backup_restore: true` \n  backup_mutations: true # CLICKHOUSE_BACKUP_MUTATIONS, allow backup mutations from system.mutations WHERE is_done=0 and apply it during restore\n  restore_as_attach: false # CLICKHOUSE_RESTORE_AS_ATTACH, allow restore tables which have inconsistent data parts structure and mutations in progress\n  check_parts_columns: true # CLICKHOUSE_CHECK_PARTS_COLUMNS, check data types from system.parts_columns during create backup to guarantee mutation is complete\n  max_connections: 0 # CLICKHOUSE_MAX_CONNECTIONS, how many parallel connections could be opened during operations\nazblob:\n  endpoint_suffix: \"core.windows.net\" # AZBLOB_ENDPOINT_SUFFIX\n  account_name: \"\"               # AZBLOB_ACCOUNT_NAME\n  account_key: \"\"                # AZBLOB_ACCOUNT_KEY\n  sas: \"\"                        # AZBLOB_SAS\n  use_managed_identity: false    # AZBLOB_USE_MANAGED_IDENTITY\n  container: \"\"                  # AZBLOB_CONTAINER\n  assume_container_exists: false # AZBLOB_ASSUME_CONTAINER_EXISTS, enables assignment of rights granting clickhouse-backup access only to blobs in the container\n  path: \"\"                       # AZBLOB_PATH, `system.macros` values can be applied as {macro_name}\n  object_disk_path: \"\"           # AZBLOB_OBJECT_DISK_PATH, path for backup of part from clickhouse object disks, if object disks present in clickhouse, then shall not be zero and shall not be prefixed by `path`\n  compression_level: 1           # AZBLOB_COMPRESSION_LEVEL\n  compression_format: tar        # AZBLOB_COMPRESSION_FORMAT, allowed values tar, lz4, bzip2, gzip, sz, xz, brortli, zstd, `none` for upload data part folders as is\n  sse_key: \"\"                    # AZBLOB_SSE_KEY\n  max_parts_count: 256           # AZBLOB_MAX_PARTS_COUNT, number of parts for AZBLOB uploads, for properly calculate buffer size\n  max_buffers: 3                 # AZBLOB_MAX_BUFFERS, similar with S3_CONCURRENCY\n  debug: false                   # AZBLOB_DEBUG\ns3:\n  access_key: \"\"                   # S3_ACCESS_KEY\n  secret_key: \"\"                   # S3_SECRET_KEY\n  bucket: \"\"                       # S3_BUCKET\n  endpoint: \"\"                     # S3_ENDPOINT\n  region: us-east-1                # S3_REGION\n  # AWS changed S3 defaults in April 2023 so that all new buckets have ACL disabled: https://aws.amazon.com/blogs/aws/heads-up-amazon-s3-security-changes-are-coming-in-april-of-2023/\n  # They also recommend that ACLs are disabled: https://docs.aws.amazon.com/AmazonS3/latest/userguide/ensure-object-ownership.html\n  # use `acl: \"\"` if you see \"api error AccessControlListNotSupported: The bucket does not allow ACLs\"\n  acl: private                     # S3_ACL \n  assume_role_arn: \"\"              # S3_ASSUME_ROLE_ARN\n  force_path_style: false          # S3_FORCE_PATH_STYLE\n  path: \"\"                         # S3_PATH, `system.macros` values can be applied as {macro_name}\n  object_disk_path: \"\"             # S3_OBJECT_DISK_PATH, path for backup of part from clickhouse object disks, if object disks present in clickhouse, then shall not be zero and shall not be prefixed by `path`\n  disable_ssl: false               # S3_DISABLE_SSL\n  compression_level: 1             # S3_COMPRESSION_LEVEL\n  compression_format: tar          # S3_COMPRESSION_FORMAT, allowed values tar, lz4, bzip2, gzip, sz, xz, brortli, zstd, `none` for upload data part folders as is\n  # look at details in https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html\n  sse: \"\"                          # S3_SSE, empty (default), AES256, or aws:kms\n  sse_customer_algorithm: \"\"       # S3_SSE_CUSTOMER_ALGORITHM, encryption algorithm, for example, AES256\n  sse_customer_key: \"\"             # S3_SSE_CUSTOMER_KEY, customer-provided encryption key use `openssl rand 32 \u003e aws_sse.key` and `cat aws_sse.key | base64` \n  sse_customer_key_md5: \"\"         # S3_SSE_CUSTOMER_KEY_MD5, 128-bit MD5 digest of the encryption key according to RFC 1321 use `cat aws_sse.key |  openssl dgst -md5 -binary | base64`\n  sse_kms_key_id: \"\"               # S3_SSE_KMS_KEY_ID, if S3_SSE is aws:kms then specifies the ID of the Amazon Web Services Key Management Service\n  sse_kms_encryption_context: \"\"   # S3_SSE_KMS_ENCRYPTION_CONTEXT, base64-encoded UTF-8 string holding a JSON with the encryption context\n                                   # Specifies the Amazon Web Services KMS Encryption Context to use for object encryption.\n                                   # This is a collection of non-secret key-value pairs that represent additional authenticated data.\n                                   # When you use an encryption context to encrypt data, you must specify the same (an exact case-sensitive match)\n                                   # encryption context to decrypt the data. An encryption context is supported only on operations with symmetric encryption KMS keys\n  disable_cert_verification: false # S3_DISABLE_CERT_VERIFICATION\n  use_custom_storage_class: false  # S3_USE_CUSTOM_STORAGE_CLASS\n  storage_class: STANDARD          # S3_STORAGE_CLASS, by default allow only from list https://github.com/aws/aws-sdk-go-v2/blob/main/service/s3/types/enums.go#L787-L799\n  concurrency: 1                   # S3_CONCURRENCY\n  max_parts_count: 4000            # S3_MAX_PARTS_COUNT, number of parts for S3 multipart uploads\n  allow_multipart_download: false  # S3_ALLOW_MULTIPART_DOWNLOAD, allow faster multipart download speed, but will require additional disk space, download_concurrency * part size in worst case\n  checksum_algorithm: \"\"           # S3_CHECKSUM_ALGORITHM, use it when you use object lock which allow to avoid delete keys from bucket until some timeout after creation, use CRC32 as fastest\n\n  # S3_OBJECT_LABELS, allow setup metadata for each object during upload, use {macro_name} from system.macros and {backupName} for current backup name\n  # The format for this env variable is \"key1:value1,key2:value2\". For YAML please continue using map syntax\n  object_labels: {}\n  # S3_CUSTOM_STORAGE_CLASS_MAP, allow setup storage class depending on the backup name regexp pattern, format nameRegexp \u003e className\n  custom_storage_class_map: {}\n  # S3_REQUEST_PAYER, define who will pay to request, look https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html for details, possible values requester, if empty then bucket owner\n  request_payer: \"\"\n  debug: false                     # S3_DEBUG\ngcs:\n  credentials_file: \"\"         # GCS_CREDENTIALS_FILE\n  credentials_json: \"\"         # GCS_CREDENTIALS_JSON\n  credentials_json_encoded: \"\" # GCS_CREDENTIALS_JSON_ENCODED\n  # look https://cloud.google.com/storage/docs/authentication/managing-hmackeys#create how to get HMAC keys for access to bucket\n  embedded_access_key: \"\"      # GCS_EMBEDDED_ACCESS_KEY, use it when `use_embedded_backup_restore: true`, `embedded_backup_disk: \"\"`, `remote_storage: gcs`\n  embedded_secret_key: \"\"      # GCS_EMBEDDED_SECRET_KEY, use it when `use_embedded_backup_restore: true`, `embedded_backup_disk: \"\"`, `remote_storage: gcs`\n  skip_credentials: false      # GCS_SKIP_CREDENTIALS, skip add credentials to requests to allow anonymous access to bucket\n  endpoint: \"\"                 # GCS_ENDPOINT, use it for custom GCS endpoint/compatible storage. For example, when using custom endpoint via private service connect\n  bucket: \"\"                   # GCS_BUCKET\n  path: \"\"                     # GCS_PATH, `system.macros` values can be applied as {macro_name}\n  object_disk_path: \"\"         # GCS_OBJECT_DISK_PATH, path for backup of part from clickhouse object disks, if object disks present in clickhouse, then shall not be zero and shall not be prefixed by `path`\n  compression_level: 1         # GCS_COMPRESSION_LEVEL\n  compression_format: tar      # GCS_COMPRESSION_FORMAT, allowed values tar, lz4, bzip2, gzip, sz, xz, brortli, zstd, `none` for upload data part folders as is\n  storage_class: STANDARD      # GCS_STORAGE_CLASS\n  chunk_size: 0                # GCS_CHUNK_SIZE, default 16 * 1024 * 1024 (16MB)\n  client_pool_size: 500        # GCS_CLIENT_POOL_SIZE, default max(upload_concurrency, download concurrency) * 3, should be at least 3 times bigger than `UPLOAD_CONCURRENCY` or `DOWNLOAD_CONCURRENCY` in each upload and download case to avoid stuck\n  # GCS_OBJECT_LABELS, allow setup metadata for each object during upload, use {macro_name} from system.macros and {backupName} for current backup name\n  # The format for this env variable is \"key1:value1,key2:value2\". For YAML please continue using map syntax\n  object_labels: {}\n  # GCS_CUSTOM_STORAGE_CLASS_MAP, allow setup storage class depends on backup name regexp pattern, format nameRegexp \u003e className\n  custom_storage_class_map: {}\n  debug: false                 # GCS_DEBUG\n  force_http: false            # GCS_FORCE_HTTP\ncos:\n  url: \"\"                      # COS_URL\n  timeout: 2m                  # COS_TIMEOUT\n  secret_id: \"\"                # COS_SECRET_ID\n  secret_key: \"\"               # COS_SECRET_KEY\n  path: \"\"                     # COS_PATH, `system.macros` values can be applied as {macro_name}\n  object_disk_path: \"\"         # GOS_OBJECT_DISK_PATH, path for backup of part from clickhouse object disks, if object disks present in clickhouse, then shall not be zero and shall not be prefixed by `path`\n  compression_format: tar      # COS_COMPRESSION_FORMAT, allowed values tar, lz4, bzip2, gzip, sz, xz, brortli, zstd, `none` for upload data part folders as is\n  compression_level: 1         # COS_COMPRESSION_LEVEL\nftp:\n  address: \"\"                  # FTP_ADDRESS in format `host:port`\n  timeout: 2m                  # FTP_TIMEOUT\n  username: \"\"                 # FTP_USERNAME\n  password: \"\"                 # FTP_PASSWORD\n  tls: false                   # FTP_TLS\n  tls_skip_verify: false       # FTP_TLS_SKIP_VERIFY\n  path: \"\"                     # FTP_PATH, `system.macros` values can be applied as {macro_name}\n  object_disk_path: \"\"         # FTP_OBJECT_DISK_PATH, path for backup of part from clickhouse object disks, if object disks present in clickhouse, then shall not be zero and shall not be prefixed by `path`\n  compression_format: tar      # FTP_COMPRESSION_FORMAT, allowed values tar, lz4, bzip2, gzip, sz, xz, brortli, zstd, `none` for upload data part folders as is\n  compression_level: 1         # FTP_COMPRESSION_LEVEL\n  debug: false                 # FTP_DEBUG\nsftp:\n  address: \"\"                  # SFTP_ADDRESS\n  username: \"\"                 # SFTP_USERNAME\n  password: \"\"                 # SFTP_PASSWORD\n  port: 22                     # SFTP_PORT\n  key: \"\"                      # SFTP_KEY\n  path: \"\"                     # SFTP_PATH, `system.macros` values can be applied as {macro_name}\n  object_disk_path: \"\"         # SFTP_OBJECT_DISK_PATH, path for backup of part from clickhouse object disks, if object disks present in clickhouse, then shall not be zero and shall not be prefixed by `path`\n  concurrency: 1               # SFTP_CONCURRENCY\n  compression_format: tar      # SFTP_COMPRESSION_FORMAT, allowed values tar, lz4, bzip2, gzip, sz, xz, brortli, zstd, `none` for upload data part folders as is\n  compression_level: 1         # SFTP_COMPRESSION_LEVEL\n  debug: false                 # SFTP_DEBUG\ncustom:\n  upload_command: \"\"           # CUSTOM_UPLOAD_COMMAND\n  download_command: \"\"         # CUSTOM_DOWNLOAD_COMMAND\n  delete_command: \"\"           # CUSTOM_DELETE_COMMAND\n  list_command: \"\"             # CUSTOM_LIST_COMMAND\n  command_timeout: \"4h\"        # CUSTOM_COMMAND_TIMEOUT\napi:\n  listen: \"localhost:7171\"     # API_LISTEN\n  enable_metrics: true         # API_ENABLE_METRICS\n  enable_pprof: false          # API_ENABLE_PPROF\n  username: \"\"                 # API_USERNAME, basic authorization for API endpoint\n  password: \"\"                 # API_PASSWORD\n  secure: false                # API_SECURE, use TLS for listen API socket\n  ca_cert_file: \"\"             # API_CA_CERT_FILE\n                               # openssl genrsa -out /etc/clickhouse-backup/ca-key.pem 4096\n                               # openssl req -subj \"/O=altinity\" -x509 -new -nodes -key /etc/clickhouse-backup/ca-key.pem -sha256 -days 365 -out /etc/clickhouse-backup/ca-cert.pem\n  private_key_file: \"\"         # API_PRIVATE_KEY_FILE, openssl genrsa -out /etc/clickhouse-backup/server-key.pem 4096\n  certificate_file: \"\"         # API_CERTIFICATE_FILE,\n                               # openssl req -subj \"/CN=localhost\" -addext \"subjectAltName = DNS:localhost,DNS:*.cluster.local\" -new -key /etc/clickhouse-backup/server-key.pem -out /etc/clickhouse-backup/server-req.csr\n                               # openssl x509 -req -days 365000 -extensions SAN -extfile \u003c(printf \"\\n[SAN]\\nsubjectAltName=DNS:localhost,DNS:*.cluster.local\") -in /etc/clickhouse-backup/server-req.csr -out /etc/clickhouse-backup/server-cert.pem -CA /etc/clickhouse-backup/ca-cert.pem -CAkey /etc/clickhouse-backup/ca-key.pem -CAcreateserial\n  integration_tables_host: \"\"  # API_INTEGRATION_TABLES_HOST, allow using DNS name to connect in `system.backup_list` and `system.backup_actions`\n  allow_parallel: false        # API_ALLOW_PARALLEL, enable parallel operations, this allows for significant memory allocation and spawns go-routines, don't enable it if you are not sure\n  create_integration_tables: false # API_CREATE_INTEGRATION_TABLES, create `system.backup_list` and `system.backup_actions`\n  complete_resumable_after_restart: true # API_COMPLETE_RESUMABLE_AFTER_RESTART, after API server startup, if `/var/lib/clickhouse/backup/*/(upload|download).state2` present, then operation will continue in the background\n  watch_is_main_process: false # WATCH_IS_MAIN_PROCESS, treats 'watch' command as a main api process, if it is stopped unexpectedly, api server is also stopped. Does not stop api server if 'watch' command canceled by the user. \n\n```\n\n## Concurrency, CPU and Memory usage recommendation\n\n`upload_concurrency` and `download_concurrency` define how many parallel download / upload go-routines will start independently of the remote storage type.\nIn 1.3.0+ it means how many parallel data parts will be uploaded, assuming `upload_by_part` and `download_by_part` are `true` (which is the default value).\n\n`concurrency` in the `s3` section means how many concurrent `upload` streams will run during multipart upload in each upload go-routine.\nA high value for `S3_CONCURRENCY` will allocate more memory for buffers inside the AWS golang SDK.\n\n`concurrency` in the `sftp` section means how many concurrent request will be used for `upload` and `download` for each file.\n\nFor `compression_format`, a good default is `tar`, which uses less CPU. In most cases the data in clickhouse is already compressed, so you may not get a lot of space savings when compressing already-compressed data.\n\n## remote_storage: custom\n\nAll custom commands use the go-template language. For example, you can use `{{ .cfg.* }}` `{{ .backupName }}` `{{ .diffFromRemote }}`.\nA custom `list_command` returns JSON which is compatible with the `metadata.BackupMetadata` type with [JSONEachRow](https://clickhouse.com/docs/en/interfaces/formats/#jsoneachrow) format.\nFor examples, see [restic](https://github.com/Altinity/clickhouse-backup/tree/master/test/integration/restic/), [rsync](https://github.com/Altinity/clickhouse-backup/tree/master/test/integration/rsync/) and [kopia](https://github.com/Altinity/clickhouse-backup/tree/master/test/integration/kopia/). Feel free to add yours custom storage.\n\n## ATTENTION!\n\n**Never change file permissions in `/var/lib/clickhouse/backup`.**\nThis path contains hard links. Permissions on all hard links to the same data on disk are always identical.\nThat means that if you change the permissions/owner/attributes on a hard link in backup path, permissions on files with which ClickHouse works will be changed too.\nThat can lead to data corruption.\n\n## API\n\nUse the `clickhouse-backup server` command to run as a REST API server. In general, the API attempts to mirror the CLI commands.\n\n### GET /\n\nList all current applicable HTTP routes\n\n### POST /\n\n### POST /restart\n\nRestart HTTP server, close all current connections, close listen socket, open listen socket again, all background go-routines breaks with contexts\n\n### POST /backup/kill\n\nKill selected command from `GET /backup/actions` command list, kill process should be near immediate, but some go-routines (upload one data part) could continue to run.\n\n- Optional query argument `command` may contain the command name to kill, or if it is omitted then kill the first \"in progress\" command.\n\n### GET /backup/tables\n\nPrint list of tables: `curl -s localhost:7171/backup/tables | jq .`, exclude pattern matched tables from `skip_tables` configuration parameters\n\n- Optional query argument `table` works the same as the `--table=pattern` CLI argument.\n- Optional query argument `remote_backup` or `remote-backup` works the same as `--remote-backup=name` CLI argument.\n\n### GET /backup/tables/all\n\nPrint list of tables: `curl -s localhost:7171/backup/tables/all | jq .`, ignore `skip_tables` configuration parameters.\n\n- Optional query argument `table` works the same as the `--table=pattern` CLI argument.\n- Optional query argument `remote_backup`or `remote-backup` works the same as `--remote-backup=name` CLI argument.\n\n### POST /backup/create\n\nCreate new backup: `curl -s localhost:7171/backup/create -X POST | jq .`\n\n- Optional string query argument `table` works the same as the `--table=pattern` CLI argument.\n- Optional string query argument `partitions` works the same as the `--partitions=value` CLI argument.\n- Optional string query argument `diff-from-remote` or `diff_from_remote` works the same as the `--diff-from-remote=backup_name` CLI argument (will calculate increment for object disks).\n- Optional string query argument `name` works the same as specifying a backup name with the CLI.\n- Optional boolean query argument `schema` works the same as the `--schema` CLI argument (backup schema only).\n- Optional boolean query argument `rbac` works the same as the `--rbac` CLI argument (backup RBAC).\n- Optional boolean query argument `rbac-only` or `rbac_only` works the same as the `--rbac-only` CLI argument (backup only RBAC).\n- Optional boolean query argument `configs` works the same as the `--configs` CLI argument (backup configs).\n- Optional boolean query argument `configs-only` or `configs_only` works the same as the `--configs-only` CLI argument (backup only configs).\n- Optional boolean query argument `skip-check-parts-columns` or `skip_check_parts_columns` works the same as the `--skip-check-parts-columns` CLI argument (allow backup inconsistent column types for data parts).\n- Optional boolean query argument `resume` works the same as the `--resume` CLI argument (resume upload for object disk data).\n- Optional string query argument `callback` allow pass callback URL which will call with POST with `application/json` with payload `{\"status\":\"error|success\",\"error\":\"not empty when error happens\", \"operation_id\" : \"\u003crandom_uuid\u003e\"}`.\n\nAdditional example: `curl -s 'localhost:7171/backup/create?table=default.billing\u0026name=billing_test' -X POST`\n\nNote: this operation is asynchronous, so the API will return once the operation has started.\n\n### POST /backup/watch\n\nRun background watch process and create full+incremental backups sequence: `curl -s localhost:7171/backup/watch -X POST | jq .`\nYou can't run watch twice with the same parameters even when `allow_parallel: true`\n\n- Optional string query argument `watch_interval` or `watch-interval` works the same as the `--watch-interval value` CLI argument.\n- Optional string query argument `full_interval` or `full-interval` works the same as the `--full-interval value` CLI argument.\n- Optional string query argument `watch_backup_name_template` or `watch-backup-name-template` works the same as the `--watch-backup-name-template value` CLI argument.\n- Optional string query argument `table` works the same as the `--table value` CLI argument (backup only selected tables).\n- Optional string query argument `partitions` works the same as the `--partitions value` CLI argument (backup only selected partitions).\n- Optional boolean query argument `schema` works the same as the `--schema` CLI argument (backup schema only).\n- Optional boolean query argument `rbac` works the same as the `--rbac` CLI argument (backup RBAC).\n- Optional boolean query argument `configs` works the same as the `--configs` CLI argument (backup configs).\n- Optional boolean query argument `skip-check-parts-columns` or `skip_check_parts_columns` works the same as the `--skip-check-parts-columns` CLI argument (allow backup inconsistent column types for data parts).\n- Optional boolean query argument `delete-source` or `delete_source` works the same as the `--delete-source` CLI argument (delete source files during upload backup).\n- Additional example: `curl -s 'localhost:7171/backup/watch?table=default.billing\u0026watch_interval=1h\u0026full_interval=24h' -X POST`\n\nNote: this operation is asynchronous and can only be stopped with `kill -s SIGHUP $(pgrep -f clickhouse-backup)` or call `/restart`, `/backup/kill`. The API will return immediately once the operation has started.\n\n### POST /backup/clean\n\nClean the `shadow` folders using all available paths from `system.disks`\n\n### POST /backup/clean/remote_broken\n\nRemove\nNote: this operation is sync, and could take a lot of time, increase http timeouts during call\n\n### POST /backup/upload\n\nUpload backup to remote storage: `curl -s localhost:7171/backup/upload/\u003cBACKUP_NAME\u003e -X POST | jq .`\n\n- Optional boolean query argument `delete-source` or `delete_source` works the same as the `--delete-source` CLI argument.\n- Optional string query argument `diff-from` or `diff_from` works the same as the `--diff-from` CLI argument.\n- Optional string query argument `diff-from-remote` or `diff_from_remote` works the same as the `--diff-from-remote` CLI argument.\n- Optional string query argument `table` works the same as the `--table value` CLI argument.\n- Optional string query argument `partitions` works the same as the `--partitions value` CLI argument.\n- Optional boolean query argument `schema` works the same as the `--schema` CLI argument (upload schema only).\n- Optional boolean query argument `rbac-only` works the same as the `--rbac-only` CLI argument (upload rbac only).\n- Optional boolean query argument `configs-only` works the same as the `--configs-only` CLI argument (upload configs\n  only).\n- Optional boolean query argument `resumable` works the same as the `--resumable` CLI argument (save intermediate upload state and resume upload if data already exists on remote storage).\n- Optional string query argument `callback` allow pass callback URL which will call with POST with `application/json` with payload `{\"status\":\"error|success\",\"error\":\"not empty when error happens\", \"operation_id\" : \"\u003crandom_uuid\u003e\"}`.\n\nNote: this operation is asynchronous, so the API will return once the operation has started.\n\n### GET /backup/list/{where}\n\nPrint a list of backups: `curl -s localhost:7171/backup/list | jq .`\nPrint a list of only local backups: `curl -s localhost:7171/backup/list/local | jq .`\nPrint a list of only remote backups: `curl -s localhost:7171/backup/list/remote | jq .`\n\nNote: The `Size` field will not be set for the local backups that have just been created or are in progress.\nNote: The `Size` field will not be set for the remote backups with upload status in progress.\n\n### POST /backup/download\n\nDownload backup from remote storage: `curl -s localhost:7171/backup/download/\u003cBACKUP_NAME\u003e -X POST | jq .`\n\n- Optional string query argument `table` works the same as the `--table value` CLI argument.\n- Optional string query argument `partitions` works the same as the `--partitions value` CLI argument.\n- Optional boolean query argument `schema` works the same as the `--schema` CLI argument (download schema only).\n- Optional boolean query argument `rbac-only` works the same as the `--rbac-only` CLI argument (download rbac only).\n- Optional boolean query argument `configs-only` works the same as the `--configs-only` CLI argument (download configs\n  only).\n- Optional boolean query argument `resumable` works the same as the `--resumable` CLI argument (save intermediate download state and resume download if it already exists on local storage).\n- Optional string query argument `callback` allow pass callback URL which will call with POST with `application/json` with payload `{\"status\":\"error|success\",\"error\":\"not empty when error happens\", \"operation_id\" : \"\u003crandom_uuid\u003e\"}`.\n\nNote: this operation is asynchronous, so the API will return once the operation has started.\n\n### POST /backup/restore\n\nCreate schema and restore data from backup: `curl -s localhost:7171/backup/restore/\u003cBACKUP_NAME\u003e -X POST | jq .`\n\n- Optional string query argument `table` works the same as the `--table value` CLI argument.\n- Optional string query argument `partitions` works the same as the `--partitions value` CLI argument.\n- Optional boolean query argument `schema` works the same as the `--schema` CLI argument (restore schema only).\n- Optional boolean query argument `data` works the same as the `--data` CLI argument (restore data only).\n- Optional boolean query argument `rm` works the same as the `--rm` CLI argument (drop tables before restore).\n- Optional boolean query argument `ignore_dependencies` or `ignore-dependencies` works the as same the `--ignore-dependencies` CLI argument.\n- Optional boolean query argument `rbac` works the same as the `--rbac` CLI argument (restore RBAC).\n- Optional boolean query argument `rbac-only` works the same as the `--rbac` CLI argument (restore only RBAC).\n- Optional boolean query argument `configs` works the same as the `--configs` CLI argument (restore configs).\n- Optional boolean query argument `configs-only` works the same as the `--configs-only` CLI argument (restore configs).\n- Optional string query argument `restore_database_mapping` or `restore-database-mapping` works the same as the `--restore-database-mapping=old_db:new_db` CLI argument.\n- Optional string query argument `restore_table_mapping` or `restore-table-mapping` works the same as the `--restore-table-mapping=old_table:new_table` CLI argument.\n- Optional string query argument `restore_schema_as_attach` or `restore-schema-as-attach` works the same as the `--restore-schema-as-attach` CLI argument.\n- Optional boolean query argument `resume` works the same as the `--resume` CLI argument (resume download for object disk data).\n- Optional string query argument `callback` allow pass callback URL which will call with POST with `application/json` with payload `{\"status\":\"error|success\",\"error\":\"not empty when error happens\", \"operation_id\" : \"\u003crandom_uuid\u003e\"}`.\n\n### POST /backup/delete\n\nDelete specific remote backup: `curl -s localhost:7171/backup/delete/remote/\u003cBACKUP_NAME\u003e -X POST | jq .`\n\nDelete specific local backup: `curl -s localhost:7171/backup/delete/local/\u003cBACKUP_NAME\u003e -X POST | jq .`\n\n### GET /backup/status\n\nDisplay list of currently running asynchronous operations: `curl -s localhost:7171/backup/status | jq .`\nOr latest command result if no backup operations executed.\n\n### POST /backup/actions\n\nExecute multiple backup actions: `curl -X POST -d '{\"command\":\"create test_backup\"}' -s localhost:7171/backup/actions`\nYou could pass multi line json each row in POST body\nWill return result for each command as separate json string in each line.\n\n### GET /backup/actions\n\nDisplay a list of all operations from start of API server: `curl -s localhost:7171/backup/actions | jq .`\n\n- Optional string query argument `filter` to filter actions on server side.\n- Optional string query argument `last` to show only the last `N` actions.\n\n## Examples\n\n- [Simple cron script for daily backups and remote upload](Examples.md#simple-cron-script-for-daily-backups-and-remote-upload)\n- [How to convert MergeTree to ReplicatedMergeTree](Examples.md#how-to-convert-mergetree-to-replicatedmergetree)\n- [How to store backups on NFS or another server](Examples.md#how-to-store-backups-on-nfs-backup-drive-or-another-server-via-sftp)\n- [How to move data to another clickhouse server](Examples.md#how-to-move-data-to-another-clickhouse-server)\n- [How to monitor that backups created and uploaded correctly](Examples.md#how-to-monitor-that-backups-were-created-and-uploaded-correctly)\n- [How to back up / restore a sharded cluster](Examples.md#how-to-back-up--restore-a-sharded-cluster)\n- [How to back up a sharded cluster with Ansible](Examples.md#how-to-back-up-a-sharded-cluster-with-ansible)\n- [How to back up a database with several terabytes of data](Examples.md#how-to-back-up-a-database-with-several-terabytes-of-data)\n- [How to use clickhouse-backup in Kubernetes](Examples.md#how-to-use-clickhouse-backup-in-kubernetes)\n- [How to back up object disks to s3 with s3:CopyObject](Examples.md#how-to-back-up-object-disks-to-s3-with-s3copyobject)\n- [How to restore object disks to s3 with s3:CopyObject](Examples.md#how-to-restore-object-disks-to-s3-with-s3copyobject)\n- [How to use AWS IRSA and IAM to allow S3 backup without Explicit credentials](Examples.md#how-to-use-aws-irsa-and-iam-to-allow-s3-backup-without-explicit-credentials)\n- [How to do incremental backups work to remote storage](Examples.md#how-incremental-backups-work-with-remote-storage)\n- [How to watch backups work](Examples.md#how-to-watch-backups-work)\n\n## Original Author\nAltinity wants to thank [@AlexAkulov](https://github.com/AlexAkulov) for creating this tool and for his valuable contributions.\n\n## Common CLI Usage\n\n### CLI command - tables\n```\nNAME:\n   clickhouse-backup tables - List of tables, exclude skip_tables\n\nUSAGE:\n   clickhouse-backup tables [--tables=\u003cdb\u003e.\u003ctable\u003e] [--remote-backup=\u003cbackup-name\u003e] [--all]\n\nOPTIONS:\n   --config value, -c value                   Config 'FILE' name. (default: \"/etc/clickhouse-backup/config.yml\") [$CLICKHOUSE_BACKUP_CONFIG]\n   --environment-override value, --env value  override any environment variable via CLI parameter\n   --all, -a                                  Print table even when match with skip_tables pattern\n   --table value, --tables value, -t value    List tables only match with table name patterns, separated by comma, allow ? and * as wildcard\n   --remote-backup value                      List tables from remote backup\n   \n```\n### CLI command - create\n```\nNAME:\n   clickhouse-backup create - Create new backup\n\nUSAGE:\n   clickhouse-backup create [-t, --tables=\u003cdb\u003e.\u003ctable\u003e] [--partitions=\u003cpartition_names\u003e] [--diff-from-remote=\u003cbackup-name\u003e] [-s, --schema] [--rbac] [--configs] [--skip-check-parts-columns] [--resume] \u003cbackup_name\u003e\n\nDESCRIPTION:\n   Create new backup\n\nOPTIONS:\n   --config value, -c value                   Config 'FILE' name. (default: \"/etc/clickhouse-backup/config.yml\") [$CLICKHOUSE_BACKUP_CONFIG]\n   --environment-override value, --env value  override any environment variable via CLI parameter\n   --table value, --tables value, -t value    Create backup only matched with table name patterns, separated by comma, allow ? and * as wildcard\n   --diff-from-remote value                   Create incremental embedded backup or upload incremental object disk data based on other remote backup name\n   --partitions partition_id                  Create backup only for selected partition names, separated by comma\nIf PARTITION BY clause returns numeric not hashed values for partition_id field in system.parts table, then use --partitions=partition_id1,partition_id2 format\nIf PARTITION BY clause returns hashed string values, then use --partitions=('non_numeric_field_value_for_part1'),('non_numeric_field_value_for_part2') format\nIf PARTITION BY clause returns tuple with multiple fields, then use --partitions=(numeric_value1,'string_value1','date_or_datetime_value'),(...) format\nIf you need different partitions for different tables, then use --partitions=db.table1:part1,part2 --partitions=db.table?:*\nValues depends on field types in your table, use single quotes for String and Date/DateTime related types\nLook at the system.parts partition and partition_id fields for details https://clickhouse.com/docs/en/operations/system-tables/parts/\n   --schema, -s                                                                               Backup schemas only, will skip data\n   --rbac, --backup-rbac, --do-backup-rbac                                                    Backup RBAC related objects\n   --configs, --backup-configs, --do-backup-configs                                           Backup 'clickhouse-server' configuration files\n   --rbac-only                                                                                Backup RBAC related objects only, will skip backup data, will backup schema only if --schema added\n   --configs-only                                                                             Backup 'clickhouse-server' configuration files only, will skip backup data, will backup schema only if --schema added\n   --skip-check-parts-columns                                                                 Skip check system.parts_columns to allow backup inconsistent column types for data parts\n   --skip-projections db_pattern.table_pattern:projections_pattern                            Skip make hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax\n   --resume use_embedded_backup_restore: true, --resumable use_embedded_backup_restore: true  Will resume upload for object disk data, hard links on local disk still continue to recreate, not work when use_embedded_backup_restore: true\n   \n```\n### CLI command - create_remote\n```\nNAME:\n   clickhouse-backup create_remote - Create and upload new backup\n\nUSAGE:\n   clickhouse-backup create_remote [-t, --tables=\u003cdb\u003e.\u003ctable\u003e] [--partitions=\u003cpartition_names\u003e] [--diff-from=\u003clocal_backup_name\u003e] [--diff-from-remote=\u003clocal_backup_name\u003e] [--schema] [--rbac] [--configs] [--resumable] [--skip-check-parts-columns] \u003cbackup_name\u003e\n\nDESCRIPTION:\n   Create and upload\n\nOPTIONS:\n   --config value, -c value                   Config 'FILE' name. (default: \"/etc/clickhouse-backup/config.yml\") [$CLICKHOUSE_BACKUP_CONFIG]\n   --environment-override value, --env value  override any environment variable via CLI parameter\n   --table value, --tables value, -t value    Create and upload backup only matched with table name patterns, separated by comma, allow ? and * as wildcard\n   --partitions partition_id                  Create and upload backup only for selected partition names, separated by comma\nIf PARTITION BY clause returns numeric not hashed values for partition_id field in system.parts table, then use --partitions=partition_id1,partition_id2 format\nIf PARTITION BY clause returns hashed string values, then use --partitions=('non_numeric_field_value_for_part1'),('non_numeric_field_value_for_part2') format\nIf PARTITION BY clause returns tuple with multiple fields, then use --partitions=(numeric_value1,'string_value1','date_or_datetime_value'),(...) format\nIf you need different partitions for different tables, then use --partitions=db.table1:part1,part2 --partitions=db.table?:*\nValues depends on field types in your table, use single quotes for String and Date/DateTime related types\nLook at the system.parts partition and partition_id fields for details https://clickhouse.com/docs/en/operations/system-tables/parts/\n   --diff-from value                                                Local backup name which used to upload current backup as incremental\n   --diff-from-remote value                                         Remote backup name which used to upload current backup as incremental\n   --schema, -s                                                     Backup and upload metadata schema only, will skip data backup\n   --rbac, --backup-rbac, --do-backup-rbac                          Backup and upload RBAC related objects\n   --configs, --backup-configs, --do-backup-configs                 Backup and upload 'clickhouse-server' configuration files\n   --rbac-only                                                      Backup RBAC related objects only, will skip backup data, will backup schema only if --schema added\n   --configs-only                                                   Backup 'clickhouse-server' configuration files only, will skip backup data, will backup schema only if --schema added\n   --resume, --resumable                                            Save intermediate upload state and resume upload if backup exists on remote storage, ignore when 'remote_storage: custom' or 'use_embedded_backup_restore: true'\n   --skip-check-parts-columns                                       Skip check system.parts_columns to allow backup inconsistent column types for data parts\n   --skip-projections db_pattern.table_pattern:projections_pattern  Skip make and upload hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax\n   --delete, --delete-source, --delete-local                        explicitly delete local backup during upload\n   \n```\n### CLI command - upload\n```\nNAME:\n   clickhouse-backup upload - Upload backup to remote storage\n\nUSAGE:\n   clickhouse-backup upload [-t, --tables=\u003cdb\u003e.\u003ctable\u003e] [--partitions=\u003cpartition_names\u003e] [-s, --schema] [--diff-from=\u003clocal_backup_name\u003e] [--diff-from-remote=\u003cremote_backup_name\u003e] [--resumable] \u003cbackup_name\u003e\n\nOPTIONS:\n   --config value, -c value                   Config 'FILE' name. (default: \"/etc/clickhouse-backup/config.yml\") [$CLICKHOUSE_BACKUP_CONFIG]\n   --environment-override value, --env value  override any environment variable via CLI parameter\n   --diff-from value                          Local backup name which used to upload current backup as incremental\n   --diff-from-remote value                   Remote backup name which used to upload current backup as incremental\n   --table value, --tables value, -t value    Upload data only for matched table name patterns, separated by comma, allow ? and * as wildcard\n   --partitions partition_id                  Upload backup only for selected partition names, separated by comma\nIf PARTITION BY clause returns numeric not hashed values for partition_id field in system.parts table, then use --partitions=partition_id1,partition_id2 format\nIf PARTITION BY clause returns hashed string values, then use --partitions=('non_numeric_field_value_for_part1'),('non_numeric_field_value_for_part2') format\nIf PARTITION BY clause returns tuple with multiple fields, then use --partitions=(numeric_value1,'string_value1','date_or_datetime_value'),(...) format\nIf you need different partitions for different tables, then use --partitions=db.table1:part1,part2 --partitions=db.table?:*\nValues depends on field types in your table, use single quotes for String and Date/DateTime related types\nLook at the system.parts partition and partition_id fields for details https://clickhouse.com/docs/en/operations/system-tables/parts/\n   --schema, -s                                                     Upload schemas only\n   --rbac-only, --rbac                                              Upload RBAC related objects only, will skip upload data, will backup schema only if --schema added\n   --configs-only, --configs                                        Upload 'clickhouse-server' configuration files only, will skip upload data, will backup schema only if --schema added\n   --skip-projections db_pattern.table_pattern:projections_pattern  Skip make and upload hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax\n   --resume, --resumable                                            Save intermediate upload state and resume upload if backup exists on remote storage, ignored with 'remote_storage: custom' or 'use_embedded_backup_restore: true'\n   --delete, --delete-source, --delete-local                        explicitly delete local backup during upload\n   \n```\n### CLI command - list\n```\nNAME:\n   clickhouse-backup list - List of backups\n\nUSAGE:\n   clickhouse-backup list [all|local|remote] [latest|previous]\n\nOPTIONS:\n   --config value, -c value                   Config 'FILE' name. (default: \"/etc/clickhouse-backup/config.yml\") [$CLICKHOUSE_BACKUP_CONFIG]\n   --environment-override value, --env value  override any environment variable via CLI parameter\n   \n```\n### CLI command - download\n```\nNAME:\n   clickhouse-backup download - Download backup from remote storage\n\nUSAGE:\n   clickhouse-backup download [-t, --tables=\u003cdb\u003e.\u003ctable\u003e] [--partitions=\u003cpartition_names\u003e] [-s, --schema] [--resumable] \u003cbackup_name\u003e\n\nOPTIONS:\n   --config value, -c value                   Config 'FILE' name. (default: \"/etc/clickhouse-backup/config.yml\") [$CLICKHOUSE_BACKUP_CONFIG]\n   --environment-override value, --env value  override any environment variable via CLI parameter\n   --table value, --tables value, -t value    Download objects which matched with table name patterns, separated by comma, allow ? and * as wildcard\n   --partitions partition_id                  Download backup data only for selected partition names, separated by comma\nIf PARTITION BY clause returns numeric not hashed values for partition_id field in system.parts table, then use --partitions=partition_id1,partition_id2 format\nIf PARTITION BY clause returns hashed string values, then use --partitions=('non_numeric_field_value_for_part1'),('non_numeric_field_value_for_part2') format\nIf PARTITION BY clause returns tuple with multiple fields, then use --partitions=(numeric_value1,'string_value1','date_or_datetime_value'),(...) format\nIf you need different partitions for different tables, then use --partitions=db.table1:part1,part2 --partitions=db.table?:*\nValues depends on field types in your table, use single quotes for String and Date/DateTime related types\nLook at the system.parts partition and partition_id fields for details https://clickhouse.com/docs/en/operations/system-tables/parts/\n   --schema, --schema-only, -s  Download schema only\n   --rbac-only, --rbac          Download RBAC related objects only, will skip download data, will backup schema only if --schema added\n   --configs-only, --configs    Download 'clickhouse-server' configuration files only, will skip download data, will backup schema only if --schema added\n   --resume, --resumable        Save intermediate download state and resume download if backup exists on local storage, ignored with 'remote_storage: custom' or 'use_embedded_backup_restore: true'\n   \n```\n### CLI command - restore\n```\nNAME:\n   clickhouse-backup restore - Create schema and restore data from backup\n\nUSAGE:\n   clickhouse-backup restore  [-t, --tables=\u003cdb\u003e.\u003ctable\u003e] [-m, --restore-database-mapping=\u003coriginDB\u003e:\u003ctargetDB\u003e[,\u003c...\u003e]] [--tm, --restore-table-mapping=\u003coriginTable\u003e:\u003ctargetTable\u003e[,\u003c...\u003e]] [--partitions=\u003cpartitions_names\u003e] [-s, --schema] [-d, --data] [--rm, --drop] [-i, --ignore-dependencies] [--rbac] [--configs] [--resume] \u003cbackup_name\u003e\n\nOPTIONS:\n   --config value, -c value                    Config 'FILE' name. (default: \"/etc/clickhouse-backup/config.yml\") [$CLICKHOUSE_BACKUP_CONFIG]\n   --environment-override value, --env value   override any environment variable via CLI parameter\n   --table value, --tables value, -t value     Restore only database and objects which matched with table name patterns, separated by comma, allow ? and * as wildcard\n   --restore-database-mapping value, -m value  Define the rule to restore data. For the database not defined in this struct, the program will not deal with it.\n   --restore-table-mapping value, --tm value   Define the rule to restore data. For the table not defined in this struct, the program will not deal with it.\n   --partitions partition_id                   Restore backup only for selected partition names, separated by comma\nIf PARTITION BY clause returns numeric not hashed values for partition_id field in system.parts table, then use --partitions=partition_id1,partition_id2 format\nIf PARTITION BY clause returns hashed string values, then use --partitions=('non_numeric_field_value_for_part1'),('non_numeric_field_value_for_part2') format\nIf PARTITION BY clause returns tuple with multiple fields, then use --partitions=(numeric_value1,'string_value1','date_or_datetime_value'),(...) format\nIf you need different partitions for different tables, then use --partitions=db.table1:part1,part2 --partitions=db.table?:*\nValues depends on field types in your table, use single quotes for String and Date/DateTime related types\nLook at the system.parts partition and partition_id fields for details https://clickhouse.com/docs/en/operations/system-tables/parts/\n   --schema, -s                                                     Restore schema only\n   --data, -d                                                       Restore data only\n   --rm, --drop                                                     Drop exists schema objects before restore\n   -i, --ignore-dependencies                                        Ignore dependencies when drop exists schema objects\n   --rbac, --restore-rbac, --do-restore-rbac                        Restore RBAC related objects\n   --configs, --restore-configs, --do-restore-configs               Restore 'clickhouse-server' CONFIG related files\n   --rbac-only                                                      Restore RBAC related objects only, will skip backup data, will backup schema only if --schema added\n   --configs-only                                                   Restore 'clickhouse-server' configuration files only, will skip backup data, will backup schema only if --schema added\n   --skip-projections db_pattern.table_pattern:projections_pattern  Skip make hardlinks to *.proj/* files during backup restoring, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax\n   --resume, --resumable                                            Will resume download for object disk data\n   --replicated-copy-to-detached                                    Copy data to detached folder for Replicated*MergeTree tables but skip ATTACH PART step\n   \n```\n### CLI command - restore_remote\n```\nNAME:\n   clickhouse-backup restore_remote - Download and restore\n\nUSAGE:\n   clickhouse-backup restore_remote [--schema] [--data] [-t, --tables=\u003cdb\u003e.\u003ctable\u003e] [-m, --restore-database-mapping=\u003coriginDB\u003e:\u003ctargetDB\u003e[,\u003c...\u003e]] [--tm, --restore-table-mapping=\u003coriginTable\u003e:\u003ctargetTable\u003e[,\u003c...\u003e]] [--partitions=\u003cpartitions_names\u003e] [--rm, --drop] [-i, --ignore-dependencies] [--rbac] [--configs] [--skip-rbac] [--skip-configs] [--resumable] \u003cbackup_name\u003e\n\nOPTIONS:\n   --config value, -c value                    Config 'FILE' name. (default: \"/etc/clickhouse-backup/config.yml\") [$CLICKHOUSE_BACKUP_CONFIG]\n   --environment-override value, --env value   override any environment variable via CLI parameter\n   --table value, --tables value, -t value     Download and restore objects which matched with table name patterns, separated by comma, allow ? and * as wildcard\n   --restore-database-mapping value, -m value  Define the rule to restore data. For the database not defined in this struct, the program will not deal with it.\n   --restore-table-mapping value, --tm value   Define the rule to restore data. For the database not defined in this struct, the program will not deal with it.\n   --partitions partition_id                   Download and restore backup only for selected partition names, separated by comma\nIf PARTITION BY clause returns numeric not hashed values for partition_id field in system.parts table, then use --partitions=partition_id1,partition_id2 format\nIf PARTITION BY clause returns hashed string values, then use --partitions=('non_numeric_field_value_for_part1'),('non_numeric_field_value_for_part2') format\nIf PARTITION BY clause returns tuple with multiple fields, then use --partitions=(numeric_value1,'string_value1','date_or_datetime_value'),(...) format\nIf you need different partitions for different tables, then use --partitions=db.table1:part1,part2 --partitions=db.table?:*\nValues depends on field types in your table, use single quotes for String and Date/DateTime related types\nLook at the system.parts partition and partition_id fields for details https://clickhouse.com/docs/en/operations/system-tables/parts/\n   --schema, -s                                                     Download and Restore schema only\n   --data, -d                                                       Download and Restore data only\n   --rm, --drop                                                     Drop schema objects before restore\n   -i, --ignore-dependencies                                        Ignore dependencies when drop exists schema objects\n   --rbac, --restore-rbac, --do-restore-rbac                        Download and Restore RBAC related objects\n   --configs, --restore-configs, --do-restore-configs               Download and Restore 'clickhouse-server' CONFIG related files\n   --rbac-only                                                      Restore RBAC related objects only, will skip backup data, will backup schema only if --schema added\n   --configs-only                                                   Restore 'clickhouse-server' configuration files only, will skip backup data, will backup schema only if --schema added\n   --skip-projections db_pattern.table_pattern:projections_pattern  Skip make hardlinks to *.proj/* files during backup restoring, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax\n   --resume, --resumable                                            Save intermediate download state and resume download if backup exists on remote storage, ignored with 'remote_storage: custom' or 'use_embedded_backup_restore: true'\n   --replicated-copy-to-detached                                    Copy data to detached folder for Replicated*MergeTree tables but skip ATTACH PART step\n   \n```\n### CLI command - delete\n```\nNAME:\n   clickhouse-backup delete - Delete specific backup\n\nUSAGE:\n   clickhouse-backup delete \u003clocal|remote\u003e \u003cbackup_name\u003e\n\nOPTIONS:\n   --config value, -c value                   Config 'FILE' name. (default: \"/etc/clickhouse-backup/config.yml\") [$CLICKHOUSE_BACKUP_CONFIG]\n   --environment-override value, --env value  override any environment variable via CLI parameter\n   \n```\n### CLI command - default-config\n```\nNAME:\n   clickhouse-backup default-config - Print default config\n\nUSAGE:\n   clickhouse-backup default-config [command options] [arguments...]\n\nOPTIONS:\n   --config value, -c value                   Config 'FILE' name. (default: \"/etc/clickhouse-backup/config.yml\") [$CLICKHOUSE_BACKUP_CONFIG]\n   --environment-override value, --env value  override any environment variable via CLI parameter\n   \n```\n### CLI command - print-config\n```\nNAME:\n   clickhouse-backup print-config - Print current config merged with environment variables\n\nUSAGE:\n   clickhouse-backup print-config [command options] [arguments...]\n\nOPTIONS:\n   --config value, -c value                   Config 'FILE' name. (default: \"/etc/clickhouse-backup/config.yml\") [$CLICKHOUSE_BACKUP_CONFIG]\n   --environment-override value, --env value  override any environment variable via CLI parameter\n   \n```\n### CLI command - clean\n```\nNAME:\n   clickhouse-backup clean - Remove data in 'shadow' folder from all 'path' folders available from 'system.disks'\n\nUSAGE:\n   clickhouse-backup clean [command options] [arguments...]\n\nOPTIONS:\n   --config value, -c value                   Config 'FILE' name. (default: \"/etc/clickhouse-backup/config.yml\") [$CLICKHOUSE_BACKUP_CONFIG]\n   --environment-override value, --env value  override any environment variable via CLI parameter\n   \n```\n### CLI command - clean_remote_broken\n```\nNAME:\n   clickhouse-backup clean_remote_broken - Remove all broken remote backups\n\nUSAGE:\n   clickhouse-backup clean_remote_broken [command options] [arguments...]\n\nOPTIONS:\n   --config value, -c value                   Config 'FILE' name. (default: \"/etc/clickhouse-backup/config.yml\") [$CLICKHOUSE_BACKUP_CONFIG]\n   --environment-override value, --env value  override any environment variable via CLI parameter\n   \n```\n### CLI command - watch\n```\nNAME:\n   clickhouse-backup watch - Run infinite loop which create full + incremental backup sequence to allow efficient backup sequences\n\nUSAGE:\n   clickhouse-backup watch [--watch-interval=1h] [--full-interval=24h] [--watch-backup-name-template=shard{shard}-{type}-{time:20060102150405}] [-t, --tables=\u003cdb\u003e.\u003ctable\u003e] [--partitions=\u003cpartitions_names\u003e] [--schema] [--rbac] [--configs] [--skip-check-parts-columns]\n\nDESCRIPTION:\n   Execute create_remote + delete local, create full backup every `--full-interval`, create and upload incremental backup every `--watch-interval` use previous backup as base with `--diff-from-remote` option, use `backups_to_keep_remote` config option for properly deletion remote backups, will delete old backups which not have references from other backups\n\nOPTIONS:\n   --config value, -c value                   Config 'FILE' name. (default: \"/etc/clickhouse-backup/config.yml\") [$CLICKHOUSE_BACKUP_CONFIG]\n   --environment-override value, --env value  override any environment variable via CLI parameter\n   --watch-interval value                     Interval for run 'create_remote' + 'delete local' for incremental backup, look format https://pkg.go.dev/time#ParseDuration\n   --full-interval value                      Interval for run 'create_remote'+'delete local' when stop create incremental backup sequence and create full backup, look format https://pkg.go.dev/time#ParseDuration\n   --watch-backup-name-template value         Template for new backup name, could contain names from system.macros, {type} - full or incremental and {time:LAYOUT}, look to https://go.dev/src/time/format.go for layout examples\n   --table value, --tables value, -t value    Create and upload only objects which matched with table name patterns, separated by comma, allow ? and * as wildcard\n   --partitions partition_id                  Partitions names, separated by comma\nIf PARTITION BY clause returns numeric not hashed values for partition_id field in system.parts table, then use --partitions=partition_id1,partition_id2 format\nIf PARTITION BY clause returns hashed string values, then use --partitions=('non_numeric_field_value_for_part1'),('non_numeric_field_value_for_part2') format\nIf PARTITION BY clause returns tuple with multiple fields, then use --partitions=(numeric_value1,'string_value1','date_or_datetime_value'),(...) format\nIf you need different partitions for different tables, then use --partitions=db.table1:part1,part2 --partitions=db.table?:*\nValues depends on field types in your table, use single quotes for String and Date/DateTime related types\nLook at the system.parts partition and partition_id fields for details https://clickhouse.com/docs/en/operations/system-tables/parts/\n   --schema, -s                                                     Schemas only\n   --rbac, --backup-rbac, --do-backup-rbac                          Backup RBAC related objects only\n   --configs, --backup-configs, --do-backup-configs                 Backup `clickhouse-server' configuration files only\n   --skip-check-parts-columns                                       Skip check system.parts_columns to allow backup inconsistent column types for data parts\n   --skip-projections db_pattern.table_pattern:projections_pattern  Skip make and upload hardlinks to *.proj/* files during backup creation, format db_pattern.table_pattern:projections_pattern, use https://pkg.go.dev/path/filepath#Match syntax\n   \n```\n### CLI command - server\n```\nNAME:\n   clickhouse-backup server - Run API server\n\nUSAGE:\n   clickhouse-backup server [command options] [arguments...]\n\nOPTIONS:\n   --config value, -c value                   Config 'FILE' name. (default: \"/etc/clickhouse-backup/config.yml\") [$CLICKHOUSE_BACKUP_CONFIG]\n   --environment-override value, --env value  override any environment variable via CLI parameter\n   --watch                                    Run watch go-routine for 'create_remote' + 'delete local', after API server startup\n   --watch-interval value                     Interval for run 'create_remote' + 'delete local' for incremental backup, look format https://pkg.go.dev/time#ParseDuration\n   --full-interval value                      Interval for run 'create_remote'+'delete local' when stop create incremental backup sequence and create full backup, look format https://pkg.go.dev/time#ParseDuration\n   --watch-backup-name-template value         Template for new backup name, could contain names from system.macros, {type} - full or incremental and {time:LAYOUT}, look to https://go.dev/src/time/format.go for layout examples\n   \n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAltinity%2Fclickhouse-backup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAltinity%2Fclickhouse-backup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAltinity%2Fclickhouse-backup/lists"}