https://github.com/matrixorigin/mo-tester
This directory contains a test suite for the MatrixOne engine.
https://github.com/matrixorigin/mo-tester
Last synced: about 1 year ago
JSON representation
This directory contains a test suite for the MatrixOne engine.
- Host: GitHub
- URL: https://github.com/matrixorigin/mo-tester
- Owner: matrixorigin
- License: apache-2.0
- Created: 2022-05-18T06:53:22.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2025-04-09T08:18:51.000Z (about 1 year ago)
- Last Synced: 2025-04-24T00:12:17.321Z (about 1 year ago)
- Language: Java
- Size: 77.9 MB
- Stars: 4
- Watchers: 3
- Forks: 14
- Open Issues: 16
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# What's in MO-Tester?
MO-Tester is a java-based tester suite for MatrixOne. It has built a whole toolchain to run automatic SQL tests. It contains the test cases and results. Once launched, MO-Tester runs all SQL test cases with MatrixOne, and compares all output SQL results with expected results. All successful and failed cases will be logged into reports.
MO-Tester content locations:
* *Cases*:
* *Result*:
+ *Result* can also generated in the path of [/cases](https://github.com/matrixorigin/matrixone/tree/main/test/cases/), for example, [/cases/auto_increment](https://github.com/matrixorigin/matrixone/tree/main/test/cases/auto_increment).
* *Report*: once finished running, a `mo-tester/report` will be generated in the local directory.
The Cases and Results are 1-1 correspondence, and they are actually `git submodules` from MatrixOne repository. Adding new cases and results should be in MatrixOne repo:
# How to use MO-Tester?
## 1. Prepare the testing environment
* Make sure you have installed jdk8.
* Launch MatrixOne or other database instance. Please refer to more information about [how to install and launch MatrixOne](https://github.com/matrixorigin/matrixorigin.io/blob/main/docs/MatrixOne/Get-Started/install-standalone-matrixone.md).
* Clone *mo-tester* repository.
```
git clone https://github.com/matrixorigin/mo-tester.git
```
* Clone *matrixOne* repository.
```
git clone https://github.com/matrixorigin/matrixone.git
```
## 2. Configure `mo-tester`
* In `mo.yml` file, configure the server address, default database name, username, and password, etc. MO-tester is based on java, so these parameters are required for the JDBC(JDBC，Java Database Connectivity) driver. Below is a default example for a local standalone version MatrixOne.
```
#jdbc
jdbc:
driver: "com.mysql.cj.jdbc.Driver"
server:
- addr: "127.0.0.1:6001"
database:
default: "test"
paremeter:
characterSetResults: "utf8"
continueBatchOnError: "false"
useServerPrepStmts: "true"
alwaysSendSetIsolation: "false"
useLocalSessionState: "true"
zeroDateTimeBehavior: "CONVERT_TO_NULL"
failoverReadOnly: "false"
serverTimezone: "Asia/Shanghai"
socketTimeout: 30000
#users
user:
name: "dump"
passwrod: "111"
```
* In `kafka.yml` file, configure the kafka server address, only for the cases that need to produce or consume messages from kafka server.
## 3. Run mo-tester
* With the simple below command, all the SQL test cases will automatically run and generate reports and error messages to *report/report.txt* and *report/error.txt*.
```
> ./run.sh -p {path_name}/matrixone/test/cases
```
If you'd like to adjust the test range, you can just change the `path` parameter of `run.yml`. And you can also specify some parameters when executing the command `./run.sh`, parameters are as followings:
| Parameters |Description|
|------------|---|
| -p |set the path of test cases needed to be executed by mo-tester, the default value is configured by the `path` in `run.yaml`|
| -m |set the method that mo-tester will run with, the default value is configured by the `method` in `run.yaml`|
| -t |set the times that mo-tester will execute cases for, must be numeric, default is 1|
| -r |set The success rate that test cases should reach, the default value is configured by the `rate` in `run.yaml`|
| -i |set the including list, and only script files in the path whose name contains one of the lists will be executed, if more than one, separated by `,`, if not specified, refers to all cases included|
| -e |set the excluding list, and script files in the path whose name contains one of the lists will not be executed, if more than one, separated by `,`, if not specified, refers to none of the cases excluded|
| -g |means SQL commands which is marked with [bvt:issue] flag will not be executed,this flag starts with [-- @bvt:issue#{issueNO.}],and ends with [-- @bvt:issue],eg:
-- @bvt:issue#3236
select date_add("1997-12-31 23:59:59",INTERVAL "-10000:1" HOUR_MINUTE);
select date_add("1997-12-31 23:59:59",INTERVAL "-100 1" YEAR_MONTH);
-- @bvt:issue
Those two sql commands are associated with issue#3236, and they will not be executed in bvt test, until the flag is removed when issue#3236 is fixed.
|
| -n |means the metadata of the resultset will be ignored when comparing the result|
| -c |only check whether the case file matches the related result file|
| -s |set the resource path that mo-tester use to store resources, and can be refered to in test file|
**Examples**:
```
./run.sh -p {path_name}/matrixone/test/cases -m run -t script -r 100 -i select,subquery -e substring -g
```
If you want to automatically generate SQL results for the new SQL cases, you can just change the `method` parameter of *run.yml* file to `genrs`, or you can just change the command `-m run` to `-m genrs`, then running the `./run.sh` scripts will directly record test results in the same path of the new SQL case file. For more information on example, see Example 4.
!!! note
Every time running `run.sh` will overwrite the report of the *error.txt* file, *report.txt* file, and *success.txt* file in the *mo-tester* repository.
## 4. Set tags in case scripts
Sometimes, to achieve some specific purposes, such as pausing or creating a new connection, you can add some special tags to the script file. The mo tester provides the following tags for use:
| Tags | Description |
|---------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| -- @skip:issue#{IssueNo.} | If set, the whole script file will be skipped, and not be executed any more for issue{IssueNo.} |
| -- @bvt:issue#{IssueNo.}
-- @bvt:issue | The sql statements between those two tags will be not executed for issue{IssueNo.} |
| -- @sleep:{time} | The mo-tester will wait for {time} s |
| -- @session:id=2&user=root&password=111
-- @session | The mo-tester will create a new connetion to execute sql statements between those two tags.
Default value of id is 1, max is 10.
Defualt value of user and password is configured in `mo.yml`. |
| -- @sortkey: | If the result is sorted, need set this tag for the sql statement. e.g.
-- @sortkey:0,1: means sort keys are first column and second colum. |
| -- @delimiter {C} | Set new delimeter to String C, C can any string expect that starts with [/,#,--] |
| -- @system {C} | Set System Command that will be executed by the runner system |
| -- @wait:{D}:[commit or wait] | means this command will be blocked until the connection[id={D}] commit or rollback |
| -- @ignore:{num},...{num} | means the designated columns which index are in {num}s will not be check. |
| -- @kafka:produce:{topic}}
JSON ARRAY
-- @bvt:produce | means the mo-tester will send all the items in this json array to the designated topic of kafka server.
The kafka server is configured in `kafka.yml` |
## 5. Check the report
* Once the test is finished, *mo-tester* generates *error.txt* file, *report.txt* file and *success.txt* file reports.
* An example of *report.txt* file looks like this:
```
[SUMMARY] COST : 98s, TOTAL :12702, SUCCESS : 11851, FAILED :13, IGNORED :838, ABNORAML :0, SUCCESS RATE : 99%
[{path_name}/matrixone/test/cases/auto_increment/auto_increment_columns.sql] COST : 2.159s, TOTAL :185, SUCCESS :163, FAILED :0, IGNORED :22, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/benchmark/tpch/01_DDL/01_create_table.sql] COST : 0.226s, TOTAL :11, SUCCESS :11, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/benchmark/tpch/02_LOAD/02_insert_customer.sql] COST : 0.357s, TOTAL :16, SUCCESS :16, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
```
|Report Keywords|Description|
|---|---|
|TOTAL|the total number of executed test cases (SQL)|
|SUCCESS|The total number of successfully executed test cases(SQL)|
|FAILED|the total number of failed executed test case(SQL)|
|IGNORED| the total number of ignored executed test cases (SQL), especially with the `--bvt:issue` tag test cases (SQL)|
|ABNORAML|the total number of abnormal executed test cases (SQL), such as the execution of MatrixOne can't determine the actual result is a system exception or *.result* file parsing error, etc|
|SUCCESS RATE|success rate: SUCCESS/(TOTAL - IGNORED)|
* An example of *error.txt* file looks like this:
```
[ERROR]
[SCRIPT FILE]: cases/transaction/atomicity.sql
[ROW NUMBER]: 14
[SQL STATEMENT]: select * from test_11 ;
[EXPECT RESULT]:
c d
1 1
2 2
[ACTUAL RESULT]:
c d
1 1
```
## 6. Test Examples
### Example 1
**Example Description**: Run all test cases in the */cases* path of the *matrixone* repository.
**Steps**:
1. Get the latest *matrixone* code.
```
cd matrixone
git pull https://github.com/matrixorigin/matrixone.git
```
2. To run all the test cases of the *matrixone* repository, you need switch into the *mo-tester* repository first, see the following commands:
```
cd mo-tester
./run.sh -p {path_name}/matrixone/test/cases
```
3. Check the result reports in the *error.txt* file, *report.txt* file, and *success.txt* file in the *mo-tester/report/* path.
### Example 2
**Example Description**: Run the test cases in the */cases/transaction/* path of the *matrixone* repository.
**Steps**:
1. Get the latest *matrixone* code.
```
cd matrixone
git pull https://github.com/matrixorigin/matrixone.git
```
2. To run the test cases in the *cases/transaction/* path of the *matrixone* repository, you need switch into the *mo-tester* repository first, see the following commands:
```
cd mo-tester
./run.sh -p {path_name}/matrixone/test/cases/transaction/
```
3. Check the result reports in the *error.txt* file, *report.txt* file, and *success.txt* file in the *mo-tester/report/* path. The example of the expected *report.txt* looks like this:
```
[SUMMARY] COST : 5s, TOTAL :1362, SUCCESS : 1354, FAILED :0, IGNORED :8, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/atomicity.sql] COST : 0.575s, TOTAL :66, SUCCESS :66, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/autocommit.test] COST : 0.175s, TOTAL :50, SUCCESS :50, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/autocommit_1.sql] COST : 1.141s, TOTAL :296, SUCCESS :288, FAILED :0, IGNORED :8, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/autocommit_atomicity.sql] COST : 0.52s, TOTAL :75, SUCCESS :75, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/autocommit_isolation.sql] COST : 1.607s, TOTAL :215, SUCCESS :215, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/autocommit_isolation_1.sql] COST : 1.438s, TOTAL :241, SUCCESS :241, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/isolation.sql] COST : 1.632s, TOTAL :202, SUCCESS :202, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/isolation_1.sql] COST : 1.512s, TOTAL :217, SUCCESS :217, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
```
### Example 3
**Example Description**: Run the single test case *cases/transaction/atomicity.sql*.
**Steps**:
1. Get the latest *matrixone* code.
```
cd matrixone
git pull https://github.com/matrixorigin/matrixone.git
```
2. To run the test cases *cases/transaction/atomicity.sql*, you need switch into the *mo-tester* repository first, see the following commands:
```
cd mo-tester
./run.sh -p {path_name}/matrixone/test/cases/transaction/atomicity.sql
```
3. Check the result reports in the *error.txt* file, *report.txt* file, and *success.txt* file in the *mo-tester/report/* path. The example of the expected *report.txt* looks like this:
```
[SUMMARY] COST : 0s, TOTAL :66, SUCCESS : 66, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/atomicity.sql] COST : 0.56s, TOTAL :66, SUCCESS :66, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
```
###
Example 4
**Example Description**:
- Create a new folder named *local_test* and place it in *{path_name}/matrixone/test/cases*
- Add a test file named *new_test.sql* to *{path_name}/matrixone/test/cases/local_test/*
- Only run the single test case *new_test.sql**
**Steps**
1. Get the latest *matrixone* code.
```
cd matrixone
git pull https://github.com/matrixorigin/matrixone.git
```
2. Generate test results:
- Method 1: To generate the test result, you need switch into the *mo-tester* repository first, then, run the following command.
```
cd mo-tester
./run.sh -p {path_name}/matrixone/test/cases/local_test/new_test.sql -m genrs -g
```
- Method 2: Open the *run.yml* file in the *mo-tester* repository, change the *method* parameter from the default `run` to `genrs`, and run the following command to generate the test result.
```
cd mo-tester
./run.sh -p {path_name}/matrixone/test/cases/local_test/new_test.sql
```
3. Check the result file in the *test/cases、result/* path of the *matrixone* repository.
4. Check the result reports in the *error.txt* file, *report.txt* file, and *success.txt* file in the *mo-tester/report/* path. The example of the expected *report.txt* looks like this:
```
[SUMMARY] COST : 0s, TOTAL :66, SUCCESS : 66, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/atomicity.sql] COST : 0.56s, TOTAL :66, SUCCESS :66, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
```
## Reference
For more information on the annotations of MO-Tester, see [MO-Tester Annotations](mo-tester-reference.md).