{"id":16921745,"url":"https://github.com/strrl/auto-failover-tikv-leader-evict","last_synced_at":"2025-03-20T23:28:01.785Z","repository":{"id":110577053,"uuid":"312500595","full_name":"STRRL/auto-failover-tikv-leader-evict","owner":"STRRL","description":"It's a ad-hoc solution.","archived":false,"fork":false,"pushed_at":"2021-03-29T08:10:06.000Z","size":53,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-19T08:46:03.029Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/STRRL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-13T07:15:19.000Z","updated_at":"2021-03-27T08:53:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"30382b92-1921-4a6a-b589-83b024b99b0a","html_url":"https://github.com/STRRL/auto-failover-tikv-leader-evict","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/STRRL%2Fauto-failover-tikv-leader-evict","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/STRRL%2Fauto-failover-tikv-leader-evict/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/STRRL%2Fauto-failover-tikv-leader-evict/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/STRRL%2Fauto-failover-tikv-leader-evict/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/STRRL","download_url":"https://codeload.github.com/STRRL/auto-failover-tikv-leader-evict/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244708539,"owners_count":20496904,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T19:53:02.786Z","updated_at":"2025-03-20T23:28:01.764Z","avatar_url":"https://github.com/STRRL.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# auto-failover-tikv-leader-evict\n\n## Overview\n\nThis project provides `evictor`. `evictor` pulls network latency metrics (provided by blackbox_exporter) from prometheus every 15 seconds(by default). If a tikv node trapped into network latency for a certain time, it will execute `pd-ctl scheduler add evict-leader-scheduler \u003cstoreId\u003e` for evicting leaders on this tikv.\n\nCaution: This tool is only compatible with TiDB v3.x, not play well with TiDB v4.x.\n\n## Prerequisites\n\n- Here must a metric named `probe_duration_seconds` exist in your prometheus. This metric is provided by `blackbox_exporter`, if you deploy tidb by `tidb-ansible`, it should be exists.\n- `evictor` should runs on a node which contains `pd-ctl` in its `$PATH`, as a daemon service.\n\n## Compile\n\n```shell\nmake\n```\n\n## Usage\n\nExample:  \n\n```shell\n./bin/evictor --prometheus=http://10.108.242.231:9090 --pd=10.99.183.247:2379 --debug\n```\n\nSystemd service example:\n\n```service\n[Unit]\nDescription=Auto failover tikv-leader-evict Service\n\n[Service]\nType=simple\nUser=tidb\nRestart=on-failure\nRestartSec=5s\nExecStart=/usr/local/bin/evictor --prometheus=http://10.96.206.21:9090 --pd=10.98.225.221:2379 --interval 10s --threshold 1s --pending-for-evict=60s --pending-for-recover=30s --debug\n\n[Install]\nWantedBy=multi-user.target\n```\n\n## Flags\n\n`--prometheus \u003cstring\u003e` address of prometheus; required;\n\n`--pd \u003cstring\u003e` address of pd; required;\n\n`--pd-version \u003cstring\u003e` version of TiDB cluster; default: `v3`; available values: `v3`, `v4`;\n\n`--max-evicted \u003cuint\u003e` max number of tikv which could be evicted leader by this tool; optional; default: 2\n\n`--interval \u003cduration\u003e` interval for refresh latency metrics; optional; default: 15s\n\n`--threshold \u003cduration\u003e` a link which hold a latency longer than threshold will be treated as bad link; optional; default: 1s\n\n`--bad-link-fuse-threshold`a node which node the threshold of bad link bigger than that will be treated as unhealthy; default 2\n\n`--pending-for-evict \u003cduration\u003e` an unhealthy tikv node will be evicted after this duration; optional; default: 1m\n\n`--pending-for-recover \u003cduration\u003e` an evicted tikv with stable latency will recover at least after this duration; optional; default: 30s\n\n`--debug` print debug logs; optional; default: false\n\n## Important Logs\n\nWhen a tikv store is evicted/recovered, it will print some logs like:\n\n```json\n{\"level\":\"info\",\"ts\":1605602637.155664,\"msg\":\"tikv node evicted\",\"store\":{\"id\":4,\"address\":\"basic-tikv-2.basic-tikv-peer.tidb-cluster.svc:20160\"}}\n...\n{\"level\":\"info\",\"ts\":1605602685.4265118,\"msg\":\"tikv node recovered\",\"store\":{\"id\":4,\"address\":\"basic-tikv-2.basic-tikv-peer.tidb-cluster.svc:20160\"}}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstrrl%2Fauto-failover-tikv-leader-evict","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstrrl%2Fauto-failover-tikv-leader-evict","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstrrl%2Fauto-failover-tikv-leader-evict/lists"}