{"id":18408343,"url":"https://github.com/pingcap/tidiff","last_synced_at":"2025-04-07T08:33:04.645Z","repository":{"id":57483844,"uuid":"183600113","full_name":"pingcap/tidiff","owner":"pingcap","description":"A toolset to improve efficiency ","archived":false,"fork":false,"pushed_at":"2023-05-30T14:58:58.000Z","size":877,"stargazers_count":41,"open_issues_count":1,"forks_count":10,"subscribers_count":94,"default_branch":"master","last_synced_at":"2025-03-22T16:02:20.979Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pingcap.png","metadata":{"files":{"readme":"README.md","changelog":"history/recorder.go","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-26T09:29:29.000Z","updated_at":"2023-11-20T08:28:37.000Z","dependencies_parsed_at":"2024-06-20T08:19:43.499Z","dependency_job_id":"a8b5d26b-179e-4d64-b1e3-266c841e6579","html_url":"https://github.com/pingcap/tidiff","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pingcap%2Ftidiff","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pingcap%2Ftidiff/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pingcap%2Ftidiff/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pingcap%2Ftidiff/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pingcap","download_url":"https://codeload.github.com/pingcap/tidiff/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247620854,"owners_count":20968300,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T03:18:38.296Z","updated_at":"2025-04-07T08:32:59.634Z","avatar_url":"https://github.com/pingcap.png","language":"Go","readme":"# tidiff\n\n`tidiff` is a diff tool that sends the same SQL statement to TiDB and MySQL for processing, then compares the results and highlights the differences. You can use it in the following scenarios: \n\n- Checking whether a SQL statement in TiDB is compatible with MySQL\n\n- Generating the same extra workload for both MySQL and TiDB when it comes to the case that a large amount of data (e.g. 1 million rows of data) is needed to reproduce the BUG.\n\n## Sample usages\n\nThe examples assume that you have already started the local MySQL and TiDB servers. The servers can be logged in with `root` username and an empty password. \n\n```\n ~ tidiff -h\nNAME:\ntidiff - Execute SQL in TiDB and MySQL and returns the results\n\nUSAGE:\ntidiff [global options] command [command options] [arguments...]\n\nVERSION:\n0.0.2\n\nDESCRIPTION:\nUsed to compare the result different in MySQL and TiDB for the same SQL statement\n\nCOMMANDS:\n    help, h  Shows a list of commands or help for one command\n\nGLOBAL OPTIONS:\n--mysql.host value      MySQL host (default: \"127.0.0.1\")\n--mysql.port value      MySQL port (default: 3306)\n--mysql.user value      MySQL username (default: \"root\")\n--mysql.password value  MySQL password\n--mysql.db value        MySQL database\n--mysql.options value   MySQL DSN options (default: \"charset=utf8mb4\")\n--tidb.host value       TiDB host (default: \"127.0.0.1\")\n--tidb.port value       TiDB port (default: 4000)\n--tidb.user value       TiDB username (default: \"root\")\n--tidb.password value   TiDB password\n--tidb.db value         TiDB database\n--tidb.options value    TiDB DSN options (default: \"charset=utf8mb4\")\n--log.diff value        Log all query diff to file\n--help, -h              show help (default: false)\n--version, -v           print the version (default: false)\n```\n\n- Create a `demo` database for MySQL/TiDB\n\n    ```\n    tidiff 'create database demo charset utf8mb4 collate utf8mb4_general_ci'\n    ```\n\n- Create a `demo.tt10000` table for MySQL/TiDB\n\n    ```\n    tidiff 'create table demo.tt10000 (a bigint(20) not null auto_increment primary key, b varchar(20) not null, c bigint(20))'\n    ```\n\n    or\n\n    ```\n    tidiff --mysql.db demo --tidb.db demo 'create table tt10000 (a bigint(20) not null auto_increment primary key, b varchar(20) not null, c bigint(20))'\n    ```\n\n- Insert 10,000 rows of data into the `demo.tt10000` table using Golang template syntax\n\n    ```\n    tidiff '! {{$count:=count 10000}} insert into demo.tt10000 values {{range $index := $count}} (NULL, \"{{varchar 20}}\", \"{{int 100000 100000000}}\"){{if head $index $count}},{{end}}{{end}}'\n    ```\n\n- Check the number of rows in the `demo.tt10000` table and get the sum for column `c`\n\n    ```\n    tidiff 'select count(*) as count, sum(c) from demo.tt10000'\n    ```\n\n- Select 5 random rows from the `demo.tt10000` table\n\n    ```\n    tidiff 'select * from demo.tt10000 order by rand() limit 5'\n    ```\n\n- Here is the screenshot of the operation above. The result differences are displayed in `diff` format.\n\n    ![](media/tidiff-guide-demo1.png)\n\n    You can use the command line mode as downstream pipeline, for example `randgen | xargs tididff`. The SQL statement should be quote with `instead of`.\n\n## Interactive Mode\n\n`tidiff` provides an interactive mode which records SQL statements execution history so as to run a SQL statement repeatedly. \n\n- To enable the interactive mode, enter the `tidiff --mysql.db test --tidb.db test` without a SQL statement.\n\n- If the interactive mode includes `--log.diff` parameters like `tidiff --log.diff /path/to/diff.log`, then all the results are saved into the log file. The difference is also highlighted when you view the results using the `cat` command.  \n\n- If a SQL statement that includes some Golang template fails, the error part is highlighted with `/* error message */` at the end of the statement.\n\n    ![DEMO](media/tidiff-guide-ui.png)\n\n- Shortcuts for interactive mode\n\n  - SQL Input Panel\n\n    - Use `Enter` to execute SQL statements. If a SQL statement begins with `!`, then Golang template is firstly used, where you can embed statements or expressions that generate random data. If a SQL statement begins with `!!`, the generated SQL statement appears on the output panel. The SQL statement rendered is not output in the output panel by default. \n\n    - Use `TAB` to switch between panels. \n\n    - Use `Up/Dn` to fast shift the focus to the `History` panel.\n\n  - MySQL/TiDB Output Panel \n\n    - Use `Up/Dn` to turn page up or down.\n\n    - Use `TAB` to switch between panels.\n    \n    - Use `ESC` and return to the `SQL input` panel. \n\n  - History Panel \n\n    - Use `Up/Dn` to fast shift the focus to the `SQL input` panel. \n    \n    - Select a history entry and use `Enter` to fill it in the `SQL input` panel for later editing and executing.\n\n    - Use `ESC` and return to the `SQL input` panel.\n \n\n## Golang template\n\nHere only introduces usages of `:=`, `range`, and `if`, and the functions provided in `tidiff`. All SQL statements must start with `!` so that `tidiff` can render them using the Golang template. \n\n- `:=` Defines a template variable `$rand`. The value must be çan integer within the range of `10000- 100000`.\n\n    ```\n    $ tidiff '! {{$rand:=int 10000 100000}} select {{$rand}} \u003c 50000'\n    MySQL(127.0.0.1:3306)\u003e select 78081 \u003c 50000\n    +---------------+\n    | 78081 \u003c 50000 |\n    +---------------+\n    | 0             |\n    +---------------+\n    1 row in set (0.001 sec)\n\n    TiDB(127.0.0.1:4000)\u003e select 78081 \u003c 50000\n    +---------------+\n    | 78081 \u003c 50000 |\n    +---------------+\n    | 0             |\n    +---------------+\n    1 row in set (0.003 sec)\n    ```\n\n- Sample usages for `range` and `if` \n\n    ```\n    $ tidiff '! {{$counter:=count 5}} select {{range $index:=$counter}} {{int 100 1000}} {{if head $index $counter}},{{end}}{{end}}'\n    MySQL(127.0.0.1:3306)\u003e select  681 , 187 , 147 , 759 , 581\n    +-----------------------------+\n    | 681 | 187 | 147 | 759 | 581 |\n    +-----------------------------+\n    | 681 | 187 | 147 | 759 | 581 |\n    +-----------------------------+\n    1 row in set (0.002 sec)\n\n    TiDB(127.0.0.1:4000)\u003e select  681 , 187 , 147 , 759 , 581\n    +-----------------------------+\n    | 681 | 187 | 147 | 759 | 581 |\n    +-----------------------------+\n    | 681 | 187 | 147 | 759 | 581 |\n    +-----------------------------+\n    1 row in set (0.005 sec)\n    ```\n\n    Generate 5 random values through Golang template\n\n    a. First define a variable as a cycle counter `{{$counter:=count 5}}`\n    \n    b. Then use `{{range $index:=$counter}}` to traverse `$counter`\n    \n    c. Output a `,` if it’s not the last element of `$counter` until a valid SQL statement is constructed.   \n    \n    ```\n    tidiff '! {{$counter:=count 5}}\n    select\n        {{range $index:=$counter}}\n            {{int 100 1000}}\n            {{if head $index $counter}},{{end}}\n        {{end}}'\n    ```\n\n- Built-in functions provided by `tidiff`\n\n  - `count n`: returns the counter with `n` elements \n\n  - `first $index $counter`: returns `true` if `$index` is the first element of `$counter`\n  \n  - `last $index $counter`: returns `true` if `$index` is the last element of `$counter`\n\n  - `head $index $counter`: returns `true` if `$index` is not the last element of `$counter`\n\n  - `tail $index $counter`: returns `true` if `$index` is not the first element of `$counter`\n\n  - `int min max`: returns an integer within the range of `[min, max)`\n\n  - `char length`: returns a random string with a length of  `length` \n\n  - `varchar length`: returns a random string with the length within the range of `[length/2, length)`\n\n## Configuration\n\n`tidiff` supports saving parameters to the configuration file as key-value pairs. The path of the configuration file is  `~/.config/tidiff/config`. See the following sample configuration file: \n\n```\nmysql.host = 192.168.4.30\nmysql.port = 3306\nmysql.user = root\nmysql.password = 12345678\nmysql.db = test\nmysql.options = charset=utf8mb4\ntidb.host = 192.168.4.31\ntidb.port = 4000\ntidb.user = root\ntidb.password = 1111\ntidb.db = test\ntidb.options = charset=utf8mb4\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpingcap%2Ftidiff","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpingcap%2Ftidiff","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpingcap%2Ftidiff/lists"}