{"id":20102536,"url":"https://github.com/clickhouse/copier","last_synced_at":"2025-05-06T08:30:46.762Z","repository":{"id":226994194,"uuid":"770116360","full_name":"ClickHouse/copier","owner":"ClickHouse","description":"clickhouse-copier (obsolete)","archived":false,"fork":false,"pushed_at":"2024-03-17T10:32:11.000Z","size":1491,"stargazers_count":11,"open_issues_count":1,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-09T10:12:15.436Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ClickHouse.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-11T00:11:10.000Z","updated_at":"2025-02-16T00:44:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"51c5b6fc-c12d-461a-b81e-51406ca65d29","html_url":"https://github.com/ClickHouse/copier","commit_stats":null,"previous_names":["clickhouse/copier"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fcopier","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fcopier/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fcopier/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fcopier/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ClickHouse","download_url":"https://codeload.github.com/ClickHouse/copier/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252648448,"owners_count":21782391,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T17:31:35.369Z","updated_at":"2025-05-06T08:30:46.756Z","avatar_url":"https://github.com/ClickHouse.png","language":"C++","readme":"\u003e [!NOTE]\n\u003e This tool is no longer supported, but you can use the latest available version as is.\n\n# clickhouse-copier\n\nCopies data from the tables in one cluster to tables in another (or the same) cluster.\n\nTo get a consistent copy, the data in the source tables and partitions should not change during the entire process.\n\nYou can run multiple `clickhouse-copier` instances on different servers to perform the same job. ClickHouse Keeper, or ZooKeeper, is used for syncing the processes.\n\nAfter starting, `clickhouse-copier`:\n\n- Connects to ClickHouse Keeper and receives:\n\n    - Copying jobs.\n    - The state of the copying jobs.\n\n- It performs the jobs.\n\n    Each running process chooses the “closest” shard of the source cluster and copies the data into the destination cluster, resharding the data if necessary.\n\n`clickhouse-copier` tracks the changes in ClickHouse Keeper and applies them on the fly.\n\nTo reduce network traffic, we recommend running `clickhouse-copier` on the same server where the source data is located.\n\n## Download and Install\n\nDownload the binaries from the [final release](releases/tag/final).\n\n## Running Clickhouse-copier\n\nThe utility should be run manually:\n\n``` bash\n$ clickhouse-copier --daemon --config keeper.xml --task-path /task/path --base-dir /path/to/dir\n```\n\nParameters:\n\n- `daemon` — Starts `clickhouse-copier` in daemon mode.\n- `config` — The path to the `keeper.xml` file with the parameters for the connection to ClickHouse Keeper.\n- `task-path` — The path to the ClickHouse Keeper node. This node is used for syncing `clickhouse-copier` processes and storing tasks. Tasks are stored in `$task-path/description`.\n- `task-file` — Optional path to file with task configuration for initial upload to ClickHouse Keeper.\n- `task-upload-force` — Force upload `task-file` even if node already exists. Default is false.\n- `base-dir` — The path to logs and auxiliary files. When it starts, `clickhouse-copier` creates `clickhouse-copier_YYYYMMHHSS_\u003cPID\u003e` subdirectories in `$base-dir`. If this parameter is omitted, the directories are created in the directory where `clickhouse-copier` was launched.\n\n## Format of keeper.xml\n\n``` xml\n\u003cclickhouse\u003e\n    \u003clogger\u003e\n        \u003clevel\u003etrace\u003c/level\u003e\n        \u003csize\u003e100M\u003c/size\u003e\n        \u003ccount\u003e3\u003c/count\u003e\n    \u003c/logger\u003e\n\n    \u003czookeeper\u003e\n        \u003cnode index=\"1\"\u003e\n            \u003chost\u003e127.0.0.1\u003c/host\u003e\n            \u003cport\u003e2181\u003c/port\u003e\n        \u003c/node\u003e\n    \u003c/zookeeper\u003e\n\u003c/clickhouse\u003e\n```\n\n## Configuration of Copying Tasks\n\n``` xml\n\u003cclickhouse\u003e\n    \u003c!-- Configuration of clusters as in an ordinary server config --\u003e\n    \u003cremote_servers\u003e\n        \u003csource_cluster\u003e\n            \u003c!--\n                source cluster \u0026 destination clusters accept exactly the same\n                parameters as parameters for the usual Distributed table\n                see https://clickhouse.com/docs/en/engines/table-engines/special/distributed/\n            --\u003e\n            \u003cshard\u003e\n                \u003cinternal_replication\u003efalse\u003c/internal_replication\u003e\n                    \u003creplica\u003e\n                        \u003chost\u003e127.0.0.1\u003c/host\u003e\n                        \u003cport\u003e9000\u003c/port\u003e\n                        \u003c!--\n                        \u003cuser\u003edefault\u003c/user\u003e\n                        \u003cpassword\u003edefault\u003c/password\u003e\n                        \u003csecure\u003e1\u003c/secure\u003e\n                        --\u003e\n                    \u003c/replica\u003e\n            \u003c/shard\u003e\n            ...\n        \u003c/source_cluster\u003e\n\n        \u003cdestination_cluster\u003e\n        ...\n        \u003c/destination_cluster\u003e\n    \u003c/remote_servers\u003e\n\n    \u003c!-- How many simultaneously active workers are possible. If you run more workers superfluous workers will sleep. --\u003e\n    \u003cmax_workers\u003e2\u003c/max_workers\u003e\n\n    \u003c!-- Setting used to fetch (pull) data from source cluster tables --\u003e\n    \u003csettings_pull\u003e\n        \u003creadonly\u003e1\u003c/readonly\u003e\n    \u003c/settings_pull\u003e\n\n    \u003c!-- Setting used to insert (push) data to destination cluster tables --\u003e\n    \u003csettings_push\u003e\n        \u003creadonly\u003e0\u003c/readonly\u003e\n    \u003c/settings_push\u003e\n\n    \u003c!-- Common setting for fetch (pull) and insert (push) operations. Also, copier process context uses it.\n         They are overlaid by \u003csettings_pull/\u003e and \u003csettings_push/\u003e respectively. --\u003e\n    \u003csettings\u003e\n        \u003cconnect_timeout\u003e3\u003c/connect_timeout\u003e\n        \u003c!-- Sync insert is set forcibly, leave it here just in case. --\u003e\n        \u003cdistributed_foreground_insert\u003e1\u003c/distributed_foreground_insert\u003e\n    \u003c/settings\u003e\n\n    \u003c!-- Copying tasks description.\n         You could specify several table task in the same task description (in the same ZooKeeper node), they will be performed\n         sequentially.\n    --\u003e\n    \u003ctables\u003e\n        \u003c!-- A table task, copies one table. --\u003e\n        \u003ctable_hits\u003e\n            \u003c!-- Source cluster name (from \u003cremote_servers/\u003e section) and tables in it that should be copied --\u003e\n            \u003ccluster_pull\u003esource_cluster\u003c/cluster_pull\u003e\n            \u003cdatabase_pull\u003etest\u003c/database_pull\u003e\n            \u003ctable_pull\u003ehits\u003c/table_pull\u003e\n\n            \u003c!-- Destination cluster name and tables in which the data should be inserted --\u003e\n            \u003ccluster_push\u003edestination_cluster\u003c/cluster_push\u003e\n            \u003cdatabase_push\u003etest\u003c/database_push\u003e\n            \u003ctable_push\u003ehits2\u003c/table_push\u003e\n\n            \u003c!-- Engine of destination tables.\n                 If destination tables have not be created, workers create them using columns definition from source tables and engine\n                 definition from here.\n\n                 NOTE: If the first worker starts insert data and detects that destination partition is not empty then the partition will\n                 be dropped and refilled, take it into account if you already have some data in destination tables. You could directly\n                 specify partitions that should be copied in \u003cenabled_partitions/\u003e, they should be in quoted format like partition column of\n                 system.parts table.\n            --\u003e\n            \u003cengine\u003e\n            ENGINE=ReplicatedMergeTree('/clickhouse/tables/{cluster}/{shard}/hits2', '{replica}')\n            PARTITION BY toMonday(date)\n            ORDER BY (CounterID, EventDate)\n            \u003c/engine\u003e\n\n            \u003c!-- Sharding key used to insert data to destination cluster --\u003e\n            \u003csharding_key\u003ejumpConsistentHash(intHash64(UserID), 2)\u003c/sharding_key\u003e\n\n            \u003c!-- Optional expression that filter data while pull them from source servers --\u003e\n            \u003cwhere_condition\u003eCounterID != 0\u003c/where_condition\u003e\n\n            \u003c!-- This section specifies partitions that should be copied, other partition will be ignored.\n                 Partition names should have the same format as\n                 partition column of system.parts table (i.e. a quoted text).\n                 Since partition key of source and destination cluster could be different,\n                 these partition names specify destination partitions.\n\n                 NOTE: In spite of this section is optional (if it is not specified, all partitions will be copied),\n                 it is strictly recommended to specify them explicitly.\n                 If you already have some ready partitions on destination cluster they\n                 will be removed at the start of the copying since they will be interpeted\n                 as unfinished data from the previous copying!!!\n            --\u003e\n            \u003cenabled_partitions\u003e\n                \u003cpartition\u003e'2018-02-26'\u003c/partition\u003e\n                \u003cpartition\u003e'2018-03-05'\u003c/partition\u003e\n                ...\n            \u003c/enabled_partitions\u003e\n        \u003c/table_hits\u003e\n\n        \u003c!-- Next table to copy. It is not copied until previous table is copying. --\u003e\n        \u003ctable_visits\u003e\n        ...\n        \u003c/table_visits\u003e\n        ...\n    \u003c/tables\u003e\n\u003c/clickhouse\u003e\n```\n\n`clickhouse-copier` tracks the changes in `/task/path/description` and applies them on the fly. For instance, if you change the value of `max_workers`, the number of processes running tasks will also change.\n\n## Build from sources\n\nYou don't have to. Download the binaries from the [final release](releases/tag/final).\n\nBut if you want, use the following repository snapshot https://github.com/ClickHouse/ClickHouse/tree/1179a70c21eeca88410a012a73a49180cc5e5e2e and proceed with the normal ClickHouse build. The built `clickhouse` binary will contain the copier tool.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclickhouse%2Fcopier","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclickhouse%2Fcopier","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclickhouse%2Fcopier/lists"}