{"id":27643303,"url":"https://github.com/matrixorigin/mo-tester","last_synced_at":"2025-04-24T00:12:24.768Z","repository":{"id":36959172,"uuid":"493544360","full_name":"matrixorigin/mo-tester","owner":"matrixorigin","description":"This directory contains a test suite for the MatrixOne engine.","archived":false,"fork":false,"pushed_at":"2025-04-09T08:18:51.000Z","size":81736,"stargazers_count":4,"open_issues_count":16,"forks_count":14,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-24T00:12:17.321Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/matrixorigin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-18T06:53:22.000Z","updated_at":"2025-04-09T08:18:55.000Z","dependencies_parsed_at":"2024-05-28T13:27:38.910Z","dependency_job_id":null,"html_url":"https://github.com/matrixorigin/mo-tester","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matrixorigin%2Fmo-tester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matrixorigin%2Fmo-tester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matrixorigin%2Fmo-tester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matrixorigin%2Fmo-tester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/matrixorigin","download_url":"https://codeload.github.com/matrixorigin/mo-tester/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250535106,"owners_count":21446508,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-24T00:12:24.199Z","updated_at":"2025-04-24T00:12:24.757Z","avatar_url":"https://github.com/matrixorigin.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# What's in MO-Tester?\n\nMO-Tester is a java-based tester suite for MatrixOne. It has built a whole toolchain to run automatic SQL tests. It contains the test cases and results. Once launched, MO-Tester runs all SQL test cases with MatrixOne, and compares all output SQL results with expected results. All successful and failed cases will be logged into reports.\n\nMO-Tester content locations:\n\n* *Cases*: \u003chttps://github.com/matrixorigin/matrixone/tree/main/test/cases\u003e\n\n* *Result*: \u003chttps://github.com/matrixorigin/mo-tester/tree/main/result\u003e\n\n + *Result* can also generated in the path of [/cases](https://github.com/matrixorigin/matrixone/tree/main/test/cases/), for example, [/cases/auto_increment](https://github.com/matrixorigin/matrixone/tree/main/test/cases/auto_increment).\n\n* *Report*: once finished running, a `mo-tester/report` will be generated in the local directory.\n\nThe Cases and Results are 1-1 correspondence, and they are actually `git submodules` from MatrixOne repository. Adding new cases and results should be in MatrixOne repo: \u003chttps://github.com/matrixorigin/matrixone/tree/main/test\u003e\n\n# How to use MO-Tester?\n\n## 1. Prepare the testing environment\n\n* Make sure you have installed jdk8.\n\n* Launch MatrixOne or other database instance. Please refer to more information about [how to install and launch MatrixOne](https://github.com/matrixorigin/matrixorigin.io/blob/main/docs/MatrixOne/Get-Started/install-standalone-matrixone.md).\n\n* Clone *mo-tester* repository.\n\n ```\n git clone https://github.com/matrixorigin/mo-tester.git\n ```\n\n* Clone *matrixOne* repository.\n\n ```\n git clone https://github.com/matrixorigin/matrixone.git\n ```\n\n## 2. Configure `mo-tester`\n\n* In `mo.yml` file, configure the server address, default database name, username, and password, etc. MO-tester is based on java, so these parameters are required for the JDBC(JDBC，Java Database Connectivity) driver. Below is a default example for a local standalone version MatrixOne.\n\n ```\n #jdbc\n jdbc:\n driver: \"com.mysql.cj.jdbc.Driver\"\n server:\n - addr: \"127.0.0.1:6001\"\n database:\n default: \"test\"\n paremeter:\n characterSetResults: \"utf8\"\n continueBatchOnError: \"false\"\n useServerPrepStmts: \"true\"\n alwaysSendSetIsolation: \"false\"\n useLocalSessionState: \"true\"\n zeroDateTimeBehavior: \"CONVERT_TO_NULL\"\n failoverReadOnly: \"false\"\n serverTimezone: \"Asia/Shanghai\"\n socketTimeout: 30000\n #users\n user:\n name: \"dump\"\n passwrod: \"111\"\n ```\n* In `kafka.yml` file, configure the kafka server address, only for the cases that need to produce or consume messages from kafka server.\n## 3. Run mo-tester\n\n* With the simple below command, all the SQL test cases will automatically run and generate reports and error messages to *report/report.txt* and *report/error.txt*.\n\n```\n\u003e ./run.sh -p {path_name}/matrixone/test/cases\n```\n\nIf you'd like to adjust the test range, you can just change the `path` parameter of `run.yml`. And you can also specify some parameters when executing the command `./run.sh`, parameters are as followings:\n\n| Parameters |Description|\n|------------|---|\n| -p |set the path of test cases needed to be executed by mo-tester, the default value is configured by the `path` in `run.yaml`|\n| -m |set the method that mo-tester will run with, the default value is configured by the `method` in `run.yaml`|\n| -t |set the times that mo-tester will execute cases for, must be numeric, default is 1|\n| -r |set The success rate that test cases should reach, the default value is configured by the `rate` in `run.yaml`|\n| -i |set the including list, and only script files in the path whose name contains one of the lists will be executed, if more than one, separated by `,`, if not specified, refers to all cases included|\n| -e |set the excluding list, and script files in the path whose name contains one of the lists will not be executed, if more than one, separated by `,`, if not specified, refers to none of the cases excluded|\n| -g |means SQL commands which is marked with [bvt:issue] flag will not be executed,this flag starts with [-- @bvt:issue#{issueNO.}],and ends with [-- @bvt:issue],eg:\u003cbr\u003e-- @bvt:issue#3236\u003cbr/\u003e\u003cbr\u003eselect date_add(\"1997-12-31 23:59:59\",INTERVAL \"-10000:1\" HOUR_MINUTE);\u003cbr/\u003e\u003cbr\u003eselect date_add(\"1997-12-31 23:59:59\",INTERVAL \"-100 1\" YEAR_MONTH);\u003cbr/\u003e\u003cbr\u003e-- @bvt:issue\u003cbr/\u003e\u003cbr\u003eThose two sql commands are associated with issue#3236, and they will not be executed in bvt test, until the flag is removed when issue#3236 is fixed.\u003cbr/\u003e|\n| -n |means the metadata of the resultset will be ignored when comparing the result|\n| -c |only check whether the case file matches the related result file|\n| -s |set the resource path that mo-tester use to store resources, and can be refered to in test file|\n\n**Examples**:\n\n```\n./run.sh -p {path_name}/matrixone/test/cases -m run -t script -r 100 -i select,subquery -e substring -g\n```\n\nIf you want to automatically generate SQL results for the new SQL cases, you can just change the `method` parameter of *run.yml* file to `genrs`, or you can just change the command `-m run` to `-m genrs`, then running the `./run.sh` scripts will directly record test results in the same path of the new SQL case file. For more information on example, see \u003ca href=\"#new_test_scenario\"\u003eExample 4\u003c/a\u003e.\n\n!!! note\nEvery time running `run.sh` will overwrite the report of the *error.txt* file, *report.txt* file, and *success.txt* file in the *mo-tester* repository.\n\n\n## 4. Set tags in case scripts\nSometimes, to achieve some specific purposes, such as pausing or creating a new connection, you can add some special tags to the script file. The mo tester provides the following tags for use:\n\n| Tags | Description |\n|---------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| -- @skip:issue#{IssueNo.} | If set, the whole script file will be skipped, and not be executed any more for issue{IssueNo.} |\n| -- @bvt:issue#{IssueNo.}\u003cbr/\u003e-- @bvt:issue | The sql statements between those two tags will be not executed for issue{IssueNo.} |\n| -- @sleep:{time} | The mo-tester will wait for {time} s |\n| -- @session:id=2\u0026user=root\u0026password=111\u003cbr/\u003e -- @session | The mo-tester will create a new connetion to execute sql statements between those two tags.\u003cbr/\u003eDefault value of id is 1, max is 10.\u003cbr/\u003eDefualt value of user and password is configured in `mo.yml`. |\n| -- @sortkey: | If the result is sorted, need set this tag for the sql statement. e.g.\u003cbr/\u003e -- @sortkey:0,1: means sort keys are first column and second colum. |\n| -- @delimiter {C} | Set new delimeter to String C, C can any string expect that starts with [/,#,--] |\n| -- @system {C} | Set System Command that will be executed by the runner system |\n| -- @wait:{D}:[commit or wait] | means this command will be blocked until the connection[id={D}] commit or rollback |\n| -- @ignore:{num},...{num} | means the designated columns which index are in {num}s will not be check. |\n| -- @kafka:produce:{topic}}\u003cbr/\u003eJSON ARRAY\u003cbr/\u003e-- @bvt:produce | means the mo-tester will send all the items in this json array to the designated topic of kafka server.\u003cbr/\u003eThe kafka server is configured in `kafka.yml` |\n\n \n\n\n## 5. Check the report\n\n* Once the test is finished, *mo-tester* generates *error.txt* file, *report.txt* file and *success.txt* file reports.\n\n* An example of *report.txt* file looks like this:\n\n```\n[SUMMARY] COST : 98s, TOTAL :12702, SUCCESS : 11851, FAILED :13, IGNORED :838, ABNORAML :0, SUCCESS RATE : 99%\n[{path_name}/matrixone/test/cases/auto_increment/auto_increment_columns.sql] COST : 2.159s, TOTAL :185, SUCCESS :163, FAILED :0, IGNORED :22, ABNORAML :0, SUCCESS RATE : 100%\n[{path_name}/matrixone/test/cases/benchmark/tpch/01_DDL/01_create_table.sql] COST : 0.226s, TOTAL :11, SUCCESS :11, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%\n[{path_name}/matrixone/test/cases/benchmark/tpch/02_LOAD/02_insert_customer.sql] COST : 0.357s, TOTAL :16, SUCCESS :16, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%\n```\n\n|Report Keywords|Description|\n|---|---|\n|TOTAL|the total number of executed test cases (SQL)|\n|SUCCESS|The total number of successfully executed test cases(SQL)|\n|FAILED|the total number of failed executed test case(SQL)|\n|IGNORED| the total number of ignored executed test cases (SQL), especially with the `--bvt:issue` tag test cases (SQL)|\n|ABNORAML|the total number of abnormal executed test cases (SQL), such as the execution of MatrixOne can't determine the actual result is a system exception or *.result* file parsing error, etc|\n|SUCCESS RATE|success rate: SUCCESS/(TOTAL - IGNORED)|\n\n* An example of *error.txt* file looks like this:\n\n```\n[ERROR]\n[SCRIPT FILE]: cases/transaction/atomicity.sql\n[ROW NUMBER]: 14\n[SQL STATEMENT]: select * from test_11 ;\n[EXPECT RESULT]:\nc\td\n1\t1\n2 2\n[ACTUAL RESULT]:\nc\td\n1\t1\n```\n\n## 6. Test Examples\n\n### Example 1\n\n**Example Description**: Run all test cases in the */cases* path of the *matrixone* repository.\n\n**Steps**:\n\n1. Get the latest *matrixone* code.\n\n ```\n cd matrixone\n git pull https://github.com/matrixorigin/matrixone.git\n ```\n\n2. To run all the test cases of the *matrixone* repository, you need switch into the *mo-tester* repository first, see the following commands:\n\n ```\n cd mo-tester\n ./run.sh -p {path_name}/matrixone/test/cases\n ```\n\n3. Check the result reports in the *error.txt* file, *report.txt* file, and *success.txt* file in the *mo-tester/report/* path.\n\n### Example 2\n\n**Example Description**: Run the test cases in the */cases/transaction/* path of the *matrixone* repository.\n\n**Steps**:\n\n1. Get the latest *matrixone* code.\n\n ```\n cd matrixone\n git pull https://github.com/matrixorigin/matrixone.git\n ```\n\n2. To run the test cases in the *cases/transaction/* path of the *matrixone* repository, you need switch into the *mo-tester* repository first, see the following commands:\n\n ```\n cd mo-tester\n ./run.sh -p {path_name}/matrixone/test/cases/transaction/\n ```\n\n3. Check the result reports in the *error.txt* file, *report.txt* file, and *success.txt* file in the *mo-tester/report/* path. The example of the expected *report.txt* looks like this:\n\n ```\n [SUMMARY] COST : 5s, TOTAL :1362, SUCCESS : 1354, FAILED :0, IGNORED :8, ABNORAML :0, SUCCESS RATE : 100%\n [{path_name}/matrixone/test/cases/transaction/atomicity.sql] COST : 0.575s, TOTAL :66, SUCCESS :66, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%\n [{path_name}/matrixone/test/cases/transaction/autocommit.test] COST : 0.175s, TOTAL :50, SUCCESS :50, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%\n [{path_name}/matrixone/test/cases/transaction/autocommit_1.sql] COST : 1.141s, TOTAL :296, SUCCESS :288, FAILED :0, IGNORED :8, ABNORAML :0, SUCCESS RATE : 100%\n [{path_name}/matrixone/test/cases/transaction/autocommit_atomicity.sql] COST : 0.52s, TOTAL :75, SUCCESS :75, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%\n [{path_name}/matrixone/test/cases/transaction/autocommit_isolation.sql] COST : 1.607s, TOTAL :215, SUCCESS :215, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%\n [{path_name}/matrixone/test/cases/transaction/autocommit_isolation_1.sql] COST : 1.438s, TOTAL :241, SUCCESS :241, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%\n [{path_name}/matrixone/test/cases/transaction/isolation.sql] COST : 1.632s, TOTAL :202, SUCCESS :202, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%\n [{path_name}/matrixone/test/cases/transaction/isolation_1.sql] COST : 1.512s, TOTAL :217, SUCCESS :217, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%\n ```\n\n### Example 3\n\n**Example Description**: Run the single test case *cases/transaction/atomicity.sql*.\n\n**Steps**:\n\n1. Get the latest *matrixone* code.\n\n ```\n cd matrixone\n git pull https://github.com/matrixorigin/matrixone.git\n ```\n\n2. To run the test cases *cases/transaction/atomicity.sql*, you need switch into the *mo-tester* repository first, see the following commands:\n\n ```\n cd mo-tester\n ./run.sh -p {path_name}/matrixone/test/cases/transaction/atomicity.sql\n ```\n\n3. Check the result reports in the *error.txt* file, *report.txt* file, and *success.txt* file in the *mo-tester/report/* path. The example of the expected *report.txt* looks like this:\n\n ```\n [SUMMARY] COST : 0s, TOTAL :66, SUCCESS : 66, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%\n [{path_name}/matrixone/test/cases/transaction/atomicity.sql] COST : 0.56s, TOTAL :66, SUCCESS :66, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%\n ```\n\n### \u003ch3\u003e\u003ca name=\"new_test_scenario\"\u003eExample 4\u003c/a\u003e\u003c/h3\u003e\n\n**Example Description**:\n\n- Create a new folder named *local_test* and place it in *{path_name}/matrixone/test/cases*\n- Add a test file named *new_test.sql* to *{path_name}/matrixone/test/cases/local_test/*\n- Only run the single test case *new_test.sql**\n\n**Steps**\n\n1. Get the latest *matrixone* code.\n\n ```\n cd matrixone\n git pull https://github.com/matrixorigin/matrixone.git\n ```\n\n2. Generate test results:\n\n - Method 1: To generate the test result, you need switch into the *mo-tester* repository first, then, run the following command.\n\n ```\n cd mo-tester\n ./run.sh -p {path_name}/matrixone/test/cases/local_test/new_test.sql -m genrs -g\n ```\n\n - Method 2: Open the *run.yml* file in the *mo-tester* repository, change the *method* parameter from the default `run` to `genrs`, and run the following command to generate the test result.\n\n ```\n cd mo-tester\n ./run.sh -p {path_name}/matrixone/test/cases/local_test/new_test.sql\n ```\n\n3. Check the result file in the *test/cases、result/* path of the *matrixone* repository.\n\n4. Check the result reports in the *error.txt* file, *report.txt* file, and *success.txt* file in the *mo-tester/report/* path. The example of the expected *report.txt* looks like this:\n\n ```\n [SUMMARY] COST : 0s, TOTAL :66, SUCCESS : 66, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%\n [{path_name}/matrixone/test/cases/transaction/atomicity.sql] COST : 0.56s, TOTAL :66, SUCCESS :66, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%\n ```\n\n## Reference\n\nFor more information on the annotations of MO-Tester, see [MO-Tester Annotations](mo-tester-reference.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatrixorigin%2Fmo-tester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatrixorigin%2Fmo-tester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatrixorigin%2Fmo-tester/lists"}