https://github.com/sofa-framework/regression
https://github.com/sofa-framework/regression
project sofa-framework
Last synced: 1 day ago
JSON representation
- Host: GitHub
- URL: https://github.com/sofa-framework/regression
- Owner: sofa-framework
- Created: 2017-03-16T10:05:37.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2026-05-07T23:16:38.000Z (about 2 months ago)
- Last Synced: 2026-05-08T01:24:51.874Z (about 2 months ago)
- Topics: project, sofa-framework
- Language: Python
- Homepage:
- Size: 39.9 MB
- Stars: 1
- Watchers: 21
- Forks: 9
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE.LGPL.txt
Awesome Lists containing this project
README
SOFA Regression Documentation
==================
## This repository contains: ##
- The tests to be perform on the scene for non regression: */Regression_test/*
- All the reference file of the tested scenes: */References*
## Building Regression_test program ##
- This repository should be included inside the main SOFA repository as /applications/projects/Regression folder.
- Then while configuring SOFA cmake project, check the option: **APPLICATION_REGRESSION_TEST** and be sure to also have **SOFA_BUILD_TESTS**.
# 1 - Non Regression Test general mechanism #
## 1.a - Non regression-tests files ##
The **Regression_test** program will search for **\*.regression-tests** files in the target directory (SOFA directory).
This file contains a list of scene to be tested.
The first information that is not a comment in this file should be the path to the reference folder containing all data needed for regression testing. This path should be relative to this folder path.
The folder should be structured as the folder containing this file and each reference of each scene should have a relative path to the reference folder equivalent to the relative path of the corresponding scene in this folder.
--> e.g. for the scene in `very/specific/path/to/MyScene.scn`, a reference with the same name as the scene should be found in `referenceFolder/very/specific/path/to`
--> A special string can be use while specifying this path: `$REGRESSION_DIR`. If used at the beginning of the path, then it will be used as an absolute path with `$REGRESSION_DIR` replaced by the content of the environment variable with the same name.
Use this, when the reference are not in the repository containing the examples but directly in the repo Regression
Each following line of the list file must contain:
1. A local path to the scene
2. The number of simulation steps to run
3. A numerical epsilon for comparison
4. Set this value to 1 if mechanicalObject inside a mapped Node need to be tested. Otherwise 0.
5. Set this value to 1 if only last iteration need to be dumped and tested. Otherwise 0.
See for example: SOFA_DIR/examples/RegressionStateScenes.regression-tests
```
### Demo scenes ###
# References folder
../path/to/the/references (OR $REGRESSION_DIR/path/to/the/references)
# Tested Scenes
Demos/caduceus.scn 100 1e-3 0 1
Demos/TriangleSurfaceCutting.scn 100 1e-4 1 1
Demos/liver.scn 100 1e-4 1 1
```
See https://github.com/sofa-framework/regression/blob/ca388cf402244e5196d2b46da69ed2f5a92fbdb1/Regression_test/RegressionSceneList.cpp#L73 for the parsing code.
The class ```RegressionSceneList``` is used to parse folders and look for those files.
Right now we have:
- **State tests:** RegressionStateScenes.regression-tests
- **Topology tests:** RegressionTopologyScenes.regression-tests
See section 4 to add a new type of test.
## 1.b - References
Reference files are stored in this repository under the **References/** folder, with the same folder hiearchy as the target scenes.
*For example, for Demo scenes inside SOFA, references will be stored inside **References/Demos/** *
**The reference files are generated when running the test for the first time on a scene and must be manually added to this repository. See section 3.a.**
If improvements of SOFA break some regression tests, the reference files of those tests need to be replaced by the new ground truth.
To do that, the reference files must be locally deleted so they can be regenerated by running the tests. Then the new referenced files need to be pushed onto this repository to update the CI.
## 1.c - Non regression tests types
For the moment non regression tests class are:
- **StateRegression_test**: At each step, the state (position/velocity) of every independent dofs is compared to values in reference files.
- **TopologyRegression_test**: At each step, the number of topology element of every topology container is compared to values in reference files.
# 2 - How to run the regression tests
## 2.a - Full solution: By setting up environment
The program **Regression_test** can not take arguments because of its internal mechanism. Thus to run it correctly, several Environment variables need to be set :
- REGRESSION_SCENES_DIR : path to the root folder containing the scenes (usually SOFA_SRC_DIR/examples)
- REGRESSION_REFERENCES_DIR : path to the root folder containing the references (this repository/References). The reference file normally provides a path to the reference directory, but when it doesn't exist, this will be the fallback.
- REGRESSION_DIR : path to the root directory of this project sources. This is only required when the references used by your project are in the Regression repo (this is the case for SOFA). If so, the path specified in your reference file should start with `$REGRESSION_DIR`
- SOFA_ROOT : path to sofa build directory.
- SOFA_PLUGIN_PATH : path to the plugin library to be loaded by "required plugin" in the scenes (usually SOFA_ROOT/lib)
Then just launch the **Regression_test** programm and it will parse the REGRESSION_SCENES_DIR folder to search for .regression-tests files.
## 2.b - Quick solution: By using python script
To avoid setting up Env. Variables, a python script **regression_test.py**, is available in this repository. You need to execute it by passing the Environment variables and paths as arguments of the script:
```
#arguments:
# 0 - script name
# 1 - regression_test binary path
# 2 - regression scenes dir
# 3 - regression references dir
# 4 - sofa build dir
```
### Example of use on linux: ###
```python regression_test.py /home/epernod/projects/regression_test_sofa/build/bin/Regression_test ~/projects/sofa-src ~/projects/regression_test_sofa/src/references ~/projects/sofa-build/```
### Example of use on windows: ###
```python ./sofa-src/applications/projects/Regression/regression_test.py /c/projects/sofa-build/bin/Release/Regression_test.exe /c/projects/sofa-src/ /c/projects/sofa-src/applications/projects/Regression/references/ /c/projects/sofa-build/```
# 3 - How to add new tests
## 3.a - Add a new SOFA scene to be tested
If the scene is already in SOFA repository:
1. Just add its path inside the corresponding file: *Examples/RegressionStateScenes.regression-tests* for state test or *Examples/RegressionTopologyScenes.regression-tests* for topology test.
2. Create the reference file by runing the **Regression_test** program.
3. **Important Push first the reference files** into the sofa-framework/regression repository so the CI has the new refs for your SOFA PR.
4. Push the updated .regression-tests file into your PR in sofa-framework/sofa repository
## 3.b - Add scene to be tested in a plugin
If the scene is inside a plugin:
Create first a new *.regression-tests* file inside your plugin and do the steps mentionned in 3.a.
# 4 - Add a new non-regression test type
## 4.a - Tests internal mechanism
For each line of the *regression-tests* file, the class ```RegressionSceneList``` store all the parameters, as well as the scene path and its reference path, inside the a structure ```RegressionSceneData``` .
The result of this process is thus a vector of ```RegressionSceneData``` structures: ```std::vector m_scenes;```
Then, for each ```RegressionSceneData``` a gtest is created and the method ```runTest``` from the Regression_test class will be called to really perform the test (or create the reference).
## 4.b - Add a new type of test
To add a new type of test you will need to add:
- Inside the **RegressionSceneList** file:
- Update the ```RegressionSceneData``` structure to store, per scene, the data needed for the test.
- Inside the **Regression_test** file:
- Add a new **xxRegressionSceneList** to specify the targeted name file. For example ```static struct StateRegressionSceneList : public RegressionSceneList``` is looking for "RegressionStateScenes.regression-tests" files.
- Add a new class **xxRegression_test** inheriting from **BaseRegression_test**. And implement the method ```void runTestImpl(RegressionSceneData data, sofa::simulation::Node::SPtr root, bool createReference = false);``` with the wanted test.
This method is called by ```BaseRegression_test::runTest``` which will check for the scene.
See for example ```class StateRegression_test : public BaseRegression_test``` where runTestImpl check At each step, the state (position/velocity) of every independent dofs is compared to values in reference files and if the mean difference per node exceed threshold this will be reported as an error.
- Finally in Regression_test.cpp you need to add the code to create a gtest for each RegressionSceneData of your list with your xxRegression_test::runTest method.
### See for example the code of StateRegression_test: ###
```
/// Create one instance of StateRegression_test per scene in stateRegressionSceneList.m_scenes list
/// Note: if N differents TEST_P(StateRegression_test, test_N) are created this will create M x N gtest. M being the number of values in the list.
INSTANTIATE_TEST_CASE_P(Regression_test,
StateRegression_test,
::testing::ValuesIn( stateRegressionSceneList.m_scenes ),
StateRegression_test::getTestName);
// Run state regression test on the listed scenes
TEST_P(StateRegression_test, sceneTest)
{
runTest(GetParam());
}
```