https://github.com/strrl/auto-failover-tikv-leader-evict
It's a ad-hoc solution.
https://github.com/strrl/auto-failover-tikv-leader-evict
Last synced: over 1 year ago
JSON representation
It's a ad-hoc solution.
- Host: GitHub
- URL: https://github.com/strrl/auto-failover-tikv-leader-evict
- Owner: STRRL
- Created: 2020-11-13T07:15:19.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2021-03-29T08:10:06.000Z (over 5 years ago)
- Last Synced: 2025-03-19T08:46:03.029Z (over 1 year ago)
- Language: Go
- Size: 51.8 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# auto-failover-tikv-leader-evict
## Overview
This project provides `evictor`. `evictor` pulls network latency metrics (provided by blackbox_exporter) from prometheus every 15 seconds(by default). If a tikv node trapped into network latency for a certain time, it will execute `pd-ctl scheduler add evict-leader-scheduler ` for evicting leaders on this tikv.
Caution: This tool is only compatible with TiDB v3.x, not play well with TiDB v4.x.
## Prerequisites
- Here must a metric named `probe_duration_seconds` exist in your prometheus. This metric is provided by `blackbox_exporter`, if you deploy tidb by `tidb-ansible`, it should be exists.
- `evictor` should runs on a node which contains `pd-ctl` in its `$PATH`, as a daemon service.
## Compile
```shell
make
```
## Usage
Example:
```shell
./bin/evictor --prometheus=http://10.108.242.231:9090 --pd=10.99.183.247:2379 --debug
```
Systemd service example:
```service
[Unit]
Description=Auto failover tikv-leader-evict Service
[Service]
Type=simple
User=tidb
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/evictor --prometheus=http://10.96.206.21:9090 --pd=10.98.225.221:2379 --interval 10s --threshold 1s --pending-for-evict=60s --pending-for-recover=30s --debug
[Install]
WantedBy=multi-user.target
```
## Flags
`--prometheus ` address of prometheus; required;
`--pd ` address of pd; required;
`--pd-version ` version of TiDB cluster; default: `v3`; available values: `v3`, `v4`;
`--max-evicted ` max number of tikv which could be evicted leader by this tool; optional; default: 2
`--interval ` interval for refresh latency metrics; optional; default: 15s
`--threshold ` a link which hold a latency longer than threshold will be treated as bad link; optional; default: 1s
`--bad-link-fuse-threshold`a node which node the threshold of bad link bigger than that will be treated as unhealthy; default 2
`--pending-for-evict ` an unhealthy tikv node will be evicted after this duration; optional; default: 1m
`--pending-for-recover ` an evicted tikv with stable latency will recover at least after this duration; optional; default: 30s
`--debug` print debug logs; optional; default: false
## Important Logs
When a tikv store is evicted/recovered, it will print some logs like:
```json
{"level":"info","ts":1605602637.155664,"msg":"tikv node evicted","store":{"id":4,"address":"basic-tikv-2.basic-tikv-peer.tidb-cluster.svc:20160"}}
...
{"level":"info","ts":1605602685.4265118,"msg":"tikv node recovered","store":{"id":4,"address":"basic-tikv-2.basic-tikv-peer.tidb-cluster.svc:20160"}}
```