{"id":20102504,"url":"https://github.com/clickhouse/keeper-extend-cluster","last_synced_at":"2025-03-02T17:20:45.267Z","repository":{"id":239773175,"uuid":"799913550","full_name":"ClickHouse/keeper-extend-cluster","owner":"ClickHouse","description":"Experiment on how to upgrade single-node clickhouse-keeper to a cluster","archived":false,"fork":false,"pushed_at":"2024-11-26T09:39:40.000Z","size":16,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-01-13T04:29:26.861Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ClickHouse.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-13T10:51:18.000Z","updated_at":"2024-11-27T14:37:36.000Z","dependencies_parsed_at":"2024-05-14T15:31:38.157Z","dependency_job_id":"62e21ac2-e47d-4e48-8aab-bfc6ddfc05b4","html_url":"https://github.com/ClickHouse/keeper-extend-cluster","commit_stats":null,"previous_names":["felixoid/keeper-cluster"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fkeeper-extend-cluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fkeeper-extend-cluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fkeeper-extend-cluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fkeeper-extend-cluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ClickHouse","download_url":"https://codeload.github.com/ClickHouse/keeper-extend-cluster/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241542084,"owners_count":19979251,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T17:31:27.250Z","updated_at":"2025-03-02T17:20:45.237Z","avatar_url":"https://github.com/ClickHouse.png","language":"Makefile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sandbox to upgrade a single node keeper to a cluster\n\n## Preparation\n\nThe docker-compose starts the containers with the same user ID as the current user. To do it, `UID` and `GID` environment variables should be added to `.env`:\n\n```\n$ make prepare\n# Or, to clean and create it\n$ make reset\n```\n\n### Reset the progress, clean up everything\n\nIf at any stage you need to clean up the state, just run the following from the repository's root:\n\n```\n$ make reset\n```\n\n## Stage 1: single node keeper and its client\n\nThe default set of docker-compose services has two services: `zoo1` and `clickhouse`:\n\n```\n# Run clickhouse and zoo1 containers\n$ docker compose up -d\n[+] Running 3/3\n ✔ Network keeper-cluster_keeper-cluster  Created   0.1s\n ✔ Container keeper-cluster-zoo1-1        Started   0.5s\n ✔ Container keeper-cluster-clickhouse-1  Started   0.7s\n```\n\nAt this stage, the `clickhouse-server` nodes are connected to `zoo1` and has a ReplicatedMergeTree table `default.test_repliacation`\n\n```\n$ docker compose exec clickhouse1 clickhouse-client -q 'SELECT * FROM test_repliacation'\n1\n2\n$ docker compose exec clickhouse2 clickhouse-client -q 'SELECT * FROM test_repliacation'\n1\n2\n```\n\nAnd `zoo1` node has `clickhouse` \"directory\" in it's root:\n\n```\n$ docker compose exec zoo1 clickhouse-keeper-client -q 'ls \"/\"'\nclickhouse keeper\n```\n\n## Stage 2: adding a second node to the keeper cluster:\n\n\u003e Attention!  \n\u003e One should never add more nodes than existing currently in the cluster. In this case, they will decide to make a new cluster.\n\nNow, when everything works, let's add a second node:\n\n```\n$ docker compose --profile keeper-extend up -d\n[+] Running 3/3\n ✔ Container keeper-cluster-zoo2-1        Started   0.4s\n ✔ Container keeper-cluster-zoo1-1        Running   0.0s\n ✔ Container keeper-cluster-clickhouse-1  Running   0.0s\n```\n\n`zoo2` is unknown to `zoo1`. It can't join the cluster and does not work:\n\n```\n$ docker compose exec zoo2 clickhouse-keeper-client -q 'ls \"/\"'\nCoordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:9181\nPoco::Exception. Code: 1000, e.code() = 111, Connection refused (version 24.4.1.2088 (official build)), [::1]:9181\nPoco::Exception. Code: 1000, e.code() = 111, Connection refused (version 24.4.1.2088 (official build)), [::1]:9181\nPoco::Exception. Code: 1000, e.code() = 111, Connection refused (version 24.4.1.2088 (official build)), [::1]:9181\n\n```\n\nAnd we have the next lines in a log:\n\n```\n$ grep deny -C10 data/zoo1/clickhouse-keeper.log\n........\n2024.05.14 13:23:44.983985 [ 35 ] {} \u003cInformation\u003e RaftInstance: receive a incoming rpc connection\n2024.05.14 13:23:44.984019 [ 35 ] {} \u003cInformation\u003e RaftInstance: session 1 got connection from ::ffff:172.24.0.4:48888 (as a server)\n2024.05.14 13:23:44.984058 [ 35 ] {} \u003cTrace\u003e RaftInstance: asio rpc session created: 0x7eb9daab4018\n2024.05.14 13:23:44.984109 [ 36 ] {} \u003cDebug\u003e RaftInstance: Receive a pre_vote_request message from 2 with LastLogIndex=0, LastLogTerm 0, EntriesLength=0, CommitIndex=0 and Term=0\n2024.05.14 13:23:44.984136 [ 36 ] {} \u003cInformation\u003e RaftInstance: [PRE-VOTE REQ] my role leader, from peer 2, log term: req 0 / mine 3\nlast idx: req 0 / mine 67, term: req 0 / mine 3\nHB alive\n2024.05.14 13:23:44.984145 [ 36 ] {} \u003cInformation\u003e RaftInstance: pre-vote decision: XX (strong deny, non-existing node)\n2024.05.14 13:23:44.984154 [ 36 ] {} \u003cDebug\u003e RaftInstance: Response back a pre_vote_response message to 2 with Accepted=0, Term=0, NextIndex=18446744073709551615\n........\n```\n\n\nWe need to add the new server to a known one. To make it possible, the config parameter `clickhouse.keeper_server.enable_reconfiguration=true` should be set, see [keeper_single.xml](configs/keeper_single.xml) and [keeper_cluster.xml](configs/keeper_cluster.xml).\n\nNow, to add it, run the next command on the `zoo1` node:\n\n```\n$ docker compose exec zoo1 clickhouse-keeper-client -q 'reconfig ADD \"server.2=zoo2:9234\"'\nserver.2=zoo2:9234;participant;1\nserver.1=zoo1:9234;participant;1\n```\n\nThe next lines will be in the `zoo1` log:\n\n```\n$ grep 'Add server' -C10 data/zoo1/clickhouse-keeper.log\n2024.05.14 13:24:07.224522 [ 24 ] {} \u003cDebug\u003e RaftInstance: append at log_idx 74, timestamp 1715693047224499\n2024.05.14 13:24:07.242767 [ 46 ] {} \u003cDebug\u003e RaftInstance: commit upto 74, current idx 73\n2024.05.14 13:24:07.242838 [ 46 ] {} \u003cTrace\u003e RaftInstance: commit upto 74, current idx 74\n2024.05.14 13:24:07.242890 [ 46 ] {} \u003cDebug\u003e RaftInstance: DONE: commit upto 74, current idx 74\n2024.05.14 13:24:08.778083 [ 36 ] {} \u003cDebug\u003e RaftInstance: Receive a pre_vote_request message from 2 with LastLogIndex=0, LastLogTerm 0, EntriesLength=0, CommitIndex=0 and Term=0\n2024.05.14 13:24:08.778122 [ 36 ] {} \u003cInformation\u003e RaftInstance: [PRE-VOTE REQ] my role leader, from peer 2, log term: req 0 / mine 3\nlast idx: req 0 / mine 74, term: req 0 / mine 3\nHB alive\n2024.05.14 13:24:08.778128 [ 36 ] {} \u003cInformation\u003e RaftInstance: pre-vote decision: XX (strong deny, non-existing node)\n2024.05.14 13:24:08.778134 [ 36 ] {} \u003cDebug\u003e RaftInstance: Response back a pre_vote_response message to 2 with Accepted=0, Term=0, NextIndex=18446744073709551615\n2024.05.14 13:24:09.240087 [ 24 ] {} \u003cDebug\u003e KeeperDispatcher: Processing config update (Add server 2): pushed\n2024.05.14 13:24:09.240159 [ 48 ] {} \u003cDebug\u003e RaftInstance: Receive a add_server_request message from 0 with LastLogIndex=0, LastLogTerm 0, EntriesLength=1, CommitIndex=0 and Term=0\n2024.05.14 13:24:09.240336 [ 48 ] {} \u003cInformation\u003e RaftInstance: sent join request to peer 2, zoo2:9234\n2024.05.14 13:24:09.240347 [ 48 ] {} \u003cDebug\u003e RaftInstance: Response back a add_server_response message to 1 with Accepted=1, Term=3, NextIndex=75\n2024.05.14 13:24:09.240356 [ 48 ] {} \u003cDebug\u003e KeeperDispatcher: Processing config update (Add server 2): accepted\n2024.05.14 13:24:09.241586 [ 32 ] {} \u003cInformation\u003e RaftInstance: 0x7eb9d9676018 connected to zoo2:9234 (as a client)\n2024.05.14 13:24:09.258657 [ 33 ] {} \u003cDebug\u003e RaftInstance: type: 13, err 0\n2024.05.14 13:24:09.258703 [ 33 ] {} \u003cDebug\u003e RaftInstance: Receive an extended join_cluster_response message from peer 2 with Result=1, Term=0, NextIndex=1\n2024.05.14 13:24:09.258719 [ 33 ] {} \u003cInformation\u003e RaftInstance: new server (2) confirms it will join, start syncing logs to it\n2024.05.14 13:24:09.258732 [ 33 ] {} \u003cDebug\u003e RaftInstance: [SYNC LOG] peer 2 start idx 1, my log start idx 1\n2024.05.14 13:24:09.258745 [ 33 ] {} \u003cInformation\u003e RaftInstance: [SYNC LOG] LogSync is done for server 2 with log gap 73 (74 - 1, limit 99999), now put the server into cluster\n2024.05.14 13:24:09.266283 [ 46 ] {} \u003cDebug\u003e RaftInstance: commit upto 75, current idx 74\n2024.05.14 13:24:09.266348 [ 46 ] {} \u003cTrace\u003e RaftInstance: commit upto 75, current idx 75\n2024.05.14 13:24:09.266361 [ 46 ] {} \u003cInformation\u003e RaftInstance: config at index 75 is committed, prev config log idx 64\n2024.05.14 13:24:09.266377 [ 46 ] {} \u003cInformation\u003e RaftInstance: new config log idx 75, prev log idx 64, cur config log idx 64, prev log idx 63\n```\n\nAnd `zoo2` works now:\n\n```\n$ docker compose exec zoo2 clickhouse-keeper-client -q 'ls \"/\"'\nclickhouse keeper\n```\n\n## Stage 3: add a third node\n\nNow, we need to do the same with the `zoo3` node\n\n```\n$ docker compose --profile keeper-cluster up -d\n[+] Running 4/4\n ✔ Container keeper-cluster-zoo2-1        Running   0.0s\n ✔ Container keeper-cluster-zoo3-1        Started   0.4s\n ✔ Container keeper-cluster-zoo1-1        Running   0.0s\n ✔ Container keeper-cluster-clickhouse-1  Running   0.0s\n$ docker compose exec zoo1 clickhouse-keeper-client -q 'reconfig ADD \"server.3=zoo3:9234\"'\nserver.3=zoo3:9234;participant;1\nserver.1=zoo1:9234;participant;1\nserver.2=zoo2:9234;participant;1\n$ docker compose exec zoo3 clickhouse-keeper-client -q 'ls \"/\"'\nclickhouse keeper\n```\n\n## Check the cluster works after restart\n\nTo restart the cluster, just run the next command:\n\n```\n$ docker compose --profile keeper-cluster restart\n```\n\nAnd let's check the status of keeper nodes:\n\n```\n$ for id in {1..3}; do echo zoo${id}; docker compose exec zoo${id} clickhouse-keeper-client -q 'stat' | grep Mode; done\nzoo1\nMode: follower\nzoo2\nMode: follower\nzoo3\nMode: leader\n```\n\n## Remaining steps\n\nIn real world, there are few more things to do:\n\n- Update the [zookeeper.xml](configs/zookeeper.xml) configuration by adding new nodes there. The `clickhouse-server` must restart to apply zookeeper configuration.\n- Update the [keeper_single.xml](configs/keeper_single.xml) config by adding there two new nodes.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclickhouse%2Fkeeper-extend-cluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclickhouse%2Fkeeper-extend-cluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclickhouse%2Fkeeper-extend-cluster/lists"}