https://github.com/pingcap/tidiff
A toolset to improve efficiency
https://github.com/pingcap/tidiff
Last synced: 11 months ago
JSON representation
A toolset to improve efficiency
- Host: GitHub
- URL: https://github.com/pingcap/tidiff
- Owner: pingcap
- License: apache-2.0
- Created: 2019-04-26T09:29:29.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2023-05-30T14:58:58.000Z (almost 3 years ago)
- Last Synced: 2025-03-22T16:02:20.979Z (11 months ago)
- Language: Go
- Size: 856 KB
- Stars: 41
- Watchers: 94
- Forks: 10
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Changelog: history/recorder.go
- License: LICENSE
Awesome Lists containing this project
README
# tidiff
`tidiff` is a diff tool that sends the same SQL statement to TiDB and MySQL for processing, then compares the results and highlights the differences. You can use it in the following scenarios:
- Checking whether a SQL statement in TiDB is compatible with MySQL
- Generating the same extra workload for both MySQL and TiDB when it comes to the case that a large amount of data (e.g. 1 million rows of data) is needed to reproduce the BUG.
## Sample usages
The examples assume that you have already started the local MySQL and TiDB servers. The servers can be logged in with `root` username and an empty password.
```
~ tidiff -h
NAME:
tidiff - Execute SQL in TiDB and MySQL and returns the results
USAGE:
tidiff [global options] command [command options] [arguments...]
VERSION:
0.0.2
DESCRIPTION:
Used to compare the result different in MySQL and TiDB for the same SQL statement
COMMANDS:
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--mysql.host value MySQL host (default: "127.0.0.1")
--mysql.port value MySQL port (default: 3306)
--mysql.user value MySQL username (default: "root")
--mysql.password value MySQL password
--mysql.db value MySQL database
--mysql.options value MySQL DSN options (default: "charset=utf8mb4")
--tidb.host value TiDB host (default: "127.0.0.1")
--tidb.port value TiDB port (default: 4000)
--tidb.user value TiDB username (default: "root")
--tidb.password value TiDB password
--tidb.db value TiDB database
--tidb.options value TiDB DSN options (default: "charset=utf8mb4")
--log.diff value Log all query diff to file
--help, -h show help (default: false)
--version, -v print the version (default: false)
```
- Create a `demo` database for MySQL/TiDB
```
tidiff 'create database demo charset utf8mb4 collate utf8mb4_general_ci'
```
- Create a `demo.tt10000` table for MySQL/TiDB
```
tidiff 'create table demo.tt10000 (a bigint(20) not null auto_increment primary key, b varchar(20) not null, c bigint(20))'
```
or
```
tidiff --mysql.db demo --tidb.db demo 'create table tt10000 (a bigint(20) not null auto_increment primary key, b varchar(20) not null, c bigint(20))'
```
- Insert 10,000 rows of data into the `demo.tt10000` table using Golang template syntax
```
tidiff '! {{$count:=count 10000}} insert into demo.tt10000 values {{range $index := $count}} (NULL, "{{varchar 20}}", "{{int 100000 100000000}}"){{if head $index $count}},{{end}}{{end}}'
```
- Check the number of rows in the `demo.tt10000` table and get the sum for column `c`
```
tidiff 'select count(*) as count, sum(c) from demo.tt10000'
```
- Select 5 random rows from the `demo.tt10000` table
```
tidiff 'select * from demo.tt10000 order by rand() limit 5'
```
- Here is the screenshot of the operation above. The result differences are displayed in `diff` format.
You can use the command line mode as downstream pipeline, for example `randgen | xargs tididff`. The SQL statement should be quote with `instead of`.
## Interactive Mode
`tidiff` provides an interactive mode which records SQL statements execution history so as to run a SQL statement repeatedly.
- To enable the interactive mode, enter the `tidiff --mysql.db test --tidb.db test` without a SQL statement.
- If the interactive mode includes `--log.diff` parameters like `tidiff --log.diff /path/to/diff.log`, then all the results are saved into the log file. The difference is also highlighted when you view the results using the `cat` command.
- If a SQL statement that includes some Golang template fails, the error part is highlighted with `/* error message */` at the end of the statement.
- Shortcuts for interactive mode
- SQL Input Panel
- Use `Enter` to execute SQL statements. If a SQL statement begins with `!`, then Golang template is firstly used, where you can embed statements or expressions that generate random data. If a SQL statement begins with `!!`, the generated SQL statement appears on the output panel. The SQL statement rendered is not output in the output panel by default.
- Use `TAB` to switch between panels.
- Use `Up/Dn` to fast shift the focus to the `History` panel.
- MySQL/TiDB Output Panel
- Use `Up/Dn` to turn page up or down.
- Use `TAB` to switch between panels.
- Use `ESC` and return to the `SQL input` panel.
- History Panel
- Use `Up/Dn` to fast shift the focus to the `SQL input` panel.
- Select a history entry and use `Enter` to fill it in the `SQL input` panel for later editing and executing.
- Use `ESC` and return to the `SQL input` panel.
## Golang template
Here only introduces usages of `:=`, `range`, and `if`, and the functions provided in `tidiff`. All SQL statements must start with `!` so that `tidiff` can render them using the Golang template.
- `:=` Defines a template variable `$rand`. The value must be çan integer within the range of `10000- 100000`.
```
$ tidiff '! {{$rand:=int 10000 100000}} select {{$rand}} < 50000'
MySQL(127.0.0.1:3306)> select 78081 < 50000
+---------------+
| 78081 < 50000 |
+---------------+
| 0 |
+---------------+
1 row in set (0.001 sec)
TiDB(127.0.0.1:4000)> select 78081 < 50000
+---------------+
| 78081 < 50000 |
+---------------+
| 0 |
+---------------+
1 row in set (0.003 sec)
```
- Sample usages for `range` and `if`
```
$ tidiff '! {{$counter:=count 5}} select {{range $index:=$counter}} {{int 100 1000}} {{if head $index $counter}},{{end}}{{end}}'
MySQL(127.0.0.1:3306)> select 681 , 187 , 147 , 759 , 581
+-----------------------------+
| 681 | 187 | 147 | 759 | 581 |
+-----------------------------+
| 681 | 187 | 147 | 759 | 581 |
+-----------------------------+
1 row in set (0.002 sec)
TiDB(127.0.0.1:4000)> select 681 , 187 , 147 , 759 , 581
+-----------------------------+
| 681 | 187 | 147 | 759 | 581 |
+-----------------------------+
| 681 | 187 | 147 | 759 | 581 |
+-----------------------------+
1 row in set (0.005 sec)
```
Generate 5 random values through Golang template
a. First define a variable as a cycle counter `{{$counter:=count 5}}`
b. Then use `{{range $index:=$counter}}` to traverse `$counter`
c. Output a `,` if it’s not the last element of `$counter` until a valid SQL statement is constructed.
```
tidiff '! {{$counter:=count 5}}
select
{{range $index:=$counter}}
{{int 100 1000}}
{{if head $index $counter}},{{end}}
{{end}}'
```
- Built-in functions provided by `tidiff`
- `count n`: returns the counter with `n` elements
- `first $index $counter`: returns `true` if `$index` is the first element of `$counter`
- `last $index $counter`: returns `true` if `$index` is the last element of `$counter`
- `head $index $counter`: returns `true` if `$index` is not the last element of `$counter`
- `tail $index $counter`: returns `true` if `$index` is not the first element of `$counter`
- `int min max`: returns an integer within the range of `[min, max)`
- `char length`: returns a random string with a length of `length`
- `varchar length`: returns a random string with the length within the range of `[length/2, length)`
## Configuration
`tidiff` supports saving parameters to the configuration file as key-value pairs. The path of the configuration file is `~/.config/tidiff/config`. See the following sample configuration file:
```
mysql.host = 192.168.4.30
mysql.port = 3306
mysql.user = root
mysql.password = 12345678
mysql.db = test
mysql.options = charset=utf8mb4
tidb.host = 192.168.4.31
tidb.port = 4000
tidb.user = root
tidb.password = 1111
tidb.db = test
tidb.options = charset=utf8mb4
```