{"id":26652839,"url":"https://github.com/Imomoi/clickhouse-backup","last_synced_at":"2025-03-25T03:58:40.595Z","repository":{"id":38295807,"uuid":"169121099","full_name":"Imomoi/clickhouse-backup","owner":"Imomoi","description":"Backup tool for ClickHouse DB with restoration via bash scripts","archived":false,"fork":false,"pushed_at":"2022-06-07T21:23:27.000Z","size":64,"stargazers_count":2,"open_issues_count":3,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-03-11T00:52:50.963Z","etag":null,"topics":["backup","clickhouse","restore","ruby"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Imomoi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-02-04T17:48:13.000Z","updated_at":"2021-06-30T12:36:49.000Z","dependencies_parsed_at":"2022-09-08T07:33:04.672Z","dependency_job_id":null,"html_url":"https://github.com/Imomoi/clickhouse-backup","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Imomoi%2Fclickhouse-backup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Imomoi%2Fclickhouse-backup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Imomoi%2Fclickhouse-backup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Imomoi%2Fclickhouse-backup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Imomoi","download_url":"https://codeload.github.com/Imomoi/clickhouse-backup/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245394757,"owners_count":20608125,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","clickhouse","restore","ruby"],"created_at":"2025-03-25T03:58:39.182Z","updated_at":"2025-03-25T03:58:40.589Z","avatar_url":"https://github.com/Imomoi.png","language":"Ruby","funding_links":[],"categories":["By Language"],"sub_categories":["Ruby"],"readme":"# Clickhouse Backup Tool\n\n## Description\n\nThis application backup databases and tables from local clickhouse instance\n\n## Nice badges\n\n[![Maintainability](https://api.codeclimate.com/v1/badges/f4e295473446f1e13a64/maintainability)](https://codeclimate.com/github/Imomoi/clickhouse-backup/maintainability)\n[![Build Status](https://travis-ci.org/Imomoi/clickhouse-backup.svg?branch=master)](https://travis-ci.org/Imomoi/clickhouse-backup)\n\n## Usage\n\nTo run application:\n\n1. Follow installation instructions to install application itself and all required dependencies\n2. Configure permissions to access Clickhouse shadow directory (need read and write access) and metadata directory (need read access)\n3. Go to application directory\n4. Make `config.yml` file in application directory\n5. Run\n\n        ~/infrastructure.clickhouse_backup/$ ./bin/clickhouse_backup config.yml\n\n## Installation\n\nTo run this application you need to install MRI Ruby and other prerequisites.\n\nI'll recomend to use RVM (rvm.io) and follow installation instructions for your OS environment.\n\nCause of specific Clickhouse behaviour Multi-User installed RVM is required (not per user)\n\n\u003e Following instruction on RVM installation may be outdated!\n\n1. Install GPG v2\n\n    sudo apt-get install gnupg2      \n\n2. Get keys\n\n    curl -sSL https://rvm.io/mpapis.asc | sudo gpg2 --import -\n    curl -sSL https://rvm.io/pkuczynski.asc | sudo gpg2 --import -\n\n3. Install RVM\n\n    \\curl -sSL https://get.rvm.io | sudo bash -s stable\n    sudo usermod -a -G rvm `whoami`\n    sudo usermod -a -G rvm clickhouse\n\nNow download application from bitbucket and install dependencies\n\n    cd /usr/local/ \u0026\u0026 git clone https://github.com/Imomoi/clickhouse-backup.git infrastructure.clickhouse_backup\n    \n\nInstall and use latest stable Ruby 2.5 for clickhouse user. Run this commands\n\n    ~/$ source /usr/local/rvm/scripts/rvm\n    ~/$ rvm install 2.5\n    ~/$ rvm use 2.5 --default\n    ~/$ cd infrastructure.clickhouse_backup\n    ~/infrastructure.clickhouse_backup/$ bundle install\n\nLet's configure our application.\n\n## Configuration\n\n### Basic configuration\n\nAfter you finish installation step you can use basic configuration file from `examples` directory.\n\n    ~/infrastructure.clickhouse_backup/$ cp examples/config.yml.example config.yml\n\nWith this configuration application would use default options for Clickhouse connection and file locations, set backup location to current user HOME directory and will not upload backups to anywhere.\n\nTo fine configure application read next sections.\n\n### Configuration file overview\n\nConfiguration file sections:\n\n1. `clickhouse` - this section describes local clickhouse instance configuration\n2. `backup` - this section describes `.tar` file making rules\n3. `destinations` - this section describes backup remote storage options\n4. `log_level` - this parameter describes logging level. Can be one of:\n   - `DEBUG`\n   - `INFO`\n   - `WARN`\n   - `ERROR`\n5. `ignored_databases` - this parameter describes which databases should be ignored during buckup. Simple YAML array of strings\n\n### Section `clickhouse`\n\n\u003e Required section\n\n1. `connection` - connection parameters\n\n   1. `scheme` - connection protocol `http` or `https`\n   2. `host` - `localhost`, cause it cann't backup remote databases (may be nfs? ;)\n   3. `port` - Clickhouse JSON API port\n2. `shadow` - full path to shadow directory\n3. `metadata` - metadata files location. This option can be in following formats:\n   1. `REAL_PATH:CLICKHOUSE_INTERNAL_PATH` - this format will remap internal clickhouse path to real location on file system (for example in case you used symlinks)\n   2. `PATH` - must be equal to internal clickhouse path\n\n### Section `backup`\n\n\u003e Required section\n\n1. `archive-prefix` - prefix for archive name\n2. `temp-file-location` - path to location, where archive file would be stored. Can be relative path\n\n### Section `destinations`\n\n\u003e Optional section\n\nFor now applications only support AWS S3 storage.\n\n1. `s3` - configuration for AWS S3\n    1. `bucket` - where to store backups\n    2. `key` - AMI user API Access key ID\n    3. `secret` - AMI user API Secret access key\n    4. `region` - storage region\n\n### Example\n\nFollowing example describes config for Clickhouse running under docker on Windows with uploading to ASW S3\n\n    ---\n    clickhouse:\n      connection:\n        scheme: http\n        host: localhost\n        port: 32770\n      shadow: \"/mnt/c/Users/Backup/Documents/Kitematic/clickhouse-server/var/lib/clickhouse/shadow\"\n      metadata: \"/var/lib/clickhouse/metadata:/mnt/c/Users/Viacheslav/Documents/Kitematic/clickhouse-server/var/lib/clickhouse/metadata\"\n\n    backup:\n      archive-prefix: 'backup-local-'\n      temp-file-location: \"~/\"\n\n    destinations:\n        s3: # AMI user: clickhouse-backup\n          bucket: 'clickhouse-backup'\n          key: 'AKIAJ...'\n          secret: '6gVkcm...'\n          region: 'us-east-1'\n\n    log_level: DEBUG\n\n    ignored_databases:\n      - default\n      - system\n      - healthmon\n\n## CRON\n\nExample `crontab` configuration:\n\n    0 0 * * * /usr/local/infrastructure.clickhouse_backup/examples/crontask.sh\n\n## Errors and troubleshooting\n\n### Permission denied on cleanup stage\n\nIf during cleanup stage you see lots of messages like in example, possibly you run \nbackup not under Clickhouse DB user (by default - `clickhouse`)\n\n    rm: cannot remove '1/data/..': Permission denied\n\n### Unsecure world writable dir\n\nIf during cleanup stage you get error please follow instructions below.\n\n    parent directory is world writable, FileUtils#remove_entry_secure does not work; abort: (...) (parent directory mode ...)\n\nClickhouse shadow directory must not be world writable. To check it follow next instructions\n\n    clickhose-data-root$ ls --full\n\nIf you see something like this - it's OK (owner user: `clickhouse`, group: `clickhouse`, other - nonwritable)\n\n    drwxrwxrwx  11 clickhouse clickhouse       4096 2018-11-14 09:18:16.066608906 +0000 data\n    drwxrwxr-x 247 clickhouse clickhouse      12288 2018-12-07 08:40:46.097357747 +0000 shadow\n\nIf you see in output something like in next example:\n\n    drwxrwxrwx  11 clickhouse clickhouse       4096 2018-11-14 09:18:16.066608906 +0000 data\n    drwxrwxrwx 247 root       root            12288 2018-12-07 08:40:46.097357747 +0000 shadow\n\nFix permissions for clickhouse directory (here e assume that your Clickhouse running under `clickhouse` user)\n\n    clickhose-data-root$ sudo chown clickhouse:clickhouse shadow\n    clickhose-data-root$ sudo chmod o-w shadow/\n\n## Data restoration\n\nAfter unpacking backup archive you will find `restore.sh` scripts in root directory, each database directories and each table directories.\n\nTo make full restoration just run `restore.sh` in root directory.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FImomoi%2Fclickhouse-backup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FImomoi%2Fclickhouse-backup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FImomoi%2Fclickhouse-backup/lists"}