https://github.com/rumbledb/rumble-test-suite
W3C test suite implemenbtation for RumbleDB
https://github.com/rumbledb/rumble-test-suite
Last synced: 12 months ago
JSON representation
W3C test suite implemenbtation for RumbleDB
- Host: GitHub
- URL: https://github.com/rumbledb/rumble-test-suite
- Owner: RumbleDB
- Created: 2025-03-12T13:13:58.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2025-03-26T16:53:01.000Z (over 1 year ago)
- Last Synced: 2025-03-26T17:43:38.732Z (over 1 year ago)
- Language: Java
- Size: 29 MB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Introduction
This project aims to run the entire W3C QT3 test suite for XQuery on RumbleDB. There are two ways of running this testsuite:
1. Run against the JSONiq parser in RumbleDB. This is the default option and converts some XQuery specific things to JSONiq++ for it to work.
2. Run against the XQuery parser in RumbleDB. This skips conversions and specifies RumbleDB to use its XQuery parser.
# Usage
There is a pipeline setup that allows you to run the testsuite against any branch in the gitlab rumble repo. To run the tests against your branch follow these steps:
1. In the gitlab webinterface on the left, go to build -> pipelines and choose "new pipeline" in the top right
2. Add a variable with the *input variable key* ``TESTED_BRANCH`` and the *input variable value* of your branch name that you want to test. Default is ``master``
3. Optionally add a variable ``TESTED_PARSER`` with the value "xquery" if you want to test the XQuery parser. By default it tests the JSONiq parser.
4. Click "run pipeline"
This will run the pipeline with all the tests. To view your results, go to build -> pipelines and click on the status of the newest pipeline. Now you will see a tab called tests at the top which you can click on to see all the results of all the different parts of the testsuite.
You can also access and download the xml test reports directly by selecting the artifacts of the collect-artifacts job.
# Reading the report
If you run the pipeline all test results are shown in the tests tab on gitlab. Clicking on a testcase gives you more information. The same information can also be accessed in the xml reports directly. Most info like assertion, original testcase etc are logged in ```` using the following pattern:
```[[key|value]]```
The most important keys and the meaning of their values are:
- ``originalTest`` contains the original text of the testcase
- ``originalAssertion`` contains the original assertion of the testcase (before possible modification)
- ``convertedString`` contains the converted assertion or testcase text
- ``query`` a query that is evaluated by rumbleDB (can possibly be multiple per testcase incase there are multiple assertions)
- ``category`` can be
- ``PASS`` testcase passed
- ``FAIL`` testcase ran but output didnt fullfill assertion
- ``ERROR`` testcase threw error that is not a part of the skip reasons
- ``SKIP`` testcase was skipped due to a skip reason
The current skip reasons are:
- Parser error XPST0003 (we assume that an unimplemented feature was encountered)
- Method or type constructor not implemented error XPST0017
- Type not implemented error XPST0051
- Unsupported errorcode in assertion (based on hardcoded list in Constants.java)
- Testcase or Testset is on list of Testcases/Testsets to skip (hardcoded list in Constants.java)
# View Analytics
There are some analytics that are run after the tests are evaluated. You can view the resulting plots (and the complete JSON files) by browsing the artifacts of the *plot* job.
Currently implemented analytics:
- **Overview.png** Overview how many tests pass, fail, error, skip in general and per testset
- **failure_types.png** Shows the most frequent failures
- **error_types.png** Shows the most frequent errors
- **failure_types.png** Shows the most frequent reasons for skips
# Local installation
If you want to work on this repo, you can install it locally with the following steps. Note that you need to have maven installed so if that isn't the case, make sure to follow those steps in the installation guide for rumble.
1. ``git clone https://gitlab.inf.ethz.ch/gfourny/rumble-test-suite.git``
2. ``cd rumble-test-suite``
3. ``git clone http://gitlab.inf.ethz.ch/gfourny/rumble.git``
4. ``cd rumble``
5. ``mvn clean compile assembly:single -quiet``
6. ``cd ..``
Then you can run a test folder like this (example for *prod*)
```
mvn -Dtest=Prod1Test test
```
Or all the tests at once with
```
mvn test
```
The output reports will be generated in rumble-test-suite/target/surefire-reports/*
(Tested with Python 3.12)
Finally, analytics can be generated and plotted by running
```
find target/surefire-reports -type f ! -name '*.xml' -delete
mvn compile
mvn exec:java
pip install - r analytics/requirements.txt
python analytics/plot.py
```
# Authors
- Stevan Mihajlovic
- Marco Schöb
# Notes
This work was started as part of a master's thesis by Stevan Mihajlovic. To view the original state of that final work, check out the ``stevan-master`` branch.