https://github.com/sandialabs/vvtest
A flexible test harness designed for scientific application development.
https://github.com/sandialabs/vvtest
hpc python scr-2277 snl-performance-workflow testing-tools
Last synced: 12 months ago
JSON representation
A flexible test harness designed for scientific application development.
- Host: GitHub
- URL: https://github.com/sandialabs/vvtest
- Owner: sandialabs
- License: other
- Created: 2022-06-07T18:41:57.000Z (about 4 years ago)
- Default Branch: master
- Last Pushed: 2024-06-30T00:33:22.000Z (about 2 years ago)
- Last Synced: 2025-05-02T04:35:10.228Z (about 1 year ago)
- Topics: hpc, python, scr-2277, snl-performance-workflow, testing-tools
- Language: Python
- Homepage:
- Size: 4.21 MB
- Stars: 3
- Watchers: 3
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# vvtest
**vvtest** is a test harness specialized for running regression, validation,
and verification tests on high performance computing (HPC) platforms.
At a high level, **vvtest** just selects and runs scripts, called tests.
It is the test specification and management capabilities together with the
plugin and control features that enable the execution of complex tests on
workstations up to large HPC platforms.
Notable features include:
* Pure Python, compatible with versions 2.7 and 3.x
* Built-in CPU & GPU resource management and load balancing
* Built-in support for HPC batch queuing systems, such as SLURM and LSF
* Test parameterization, allowing a single script file to produce multiple
test instances with different parameter values
* Inter-test dependencies - tests can depend on other tests. Dependencies
are run first and the results are available to the dependent test.
* Test selection and execution control, such as by keyword, parameter,
platform, and a general option string
* Tests can be any script language, but there is built-in support for Python
and (to a lesser extent) Bash
* Configuration and plugins allow control over testing behavior
* Tests are executed in a separate directory from the test scripts
("out of source" execution)
* Test result output formats include CDash, GitLab markdown, JUnit, and HTML.
## Documentation
* Run `vvtest -h` to show the options.
* The RELEASE_NOTES file is kept up to date.
* Additional documentation is available on the wiki.
## Download and Install
Since the only dependency is Python, just clone the
**vvtest** repository and run the `vvtest` script. You can optionally run the
`install_vvtest` script to install into a prefix of your choice.
## An Example
Consider a file called `mytest.vvt` with contents
```
# This is my cool test.
#VVT: parameterize: np = 1 2
import vvtest_util as vvt
import shared_stuff as stuff
print ( 'Starting test {0} np={1}'.format( vvt.NAME, vvt.np ) )
stuff.run_simulation( myoption='foo', numcpu=vvt.np )
stuff.check_results()
```
This test could be run as follows:
```
$ vvtest mytest.vvt
==================================================
Test list:
completed: 0
notrun: 2
total: 2
Platform ceelan, num procs = 16, max procs = 16
Start time: Fri Jun 5 18:07:34 2020
Starting: TestResults.ceelan/mytest.np=2
Starting: TestResults.ceelan/mytest.np=1
Finished: pass 0:01 06/05 18:07:34 TestResults.ceelan/mytest.np=2
Finished: pass 0:01 06/05 18:07:34 TestResults.ceelan/mytest.np=1
Progress: 2/2 = %100.0, time = 1s
==================================================
Summary:
completed: 2
2 pass
total: 2
Finish date: Fri Jun 5 18:07:35 2020 (elapsed time 2s)
Test directory: TestResults.ceelan
```
Note that the test specification file, `mytest.vvt`, resulted in two test
instances, `mytest.np=1` and `mytest.np=2`, which were run and both passed.
A test can import the `vvtest_util.py` file, which is specific to each
test and contains information about the test instance and runtime parameters.
The test is run in a subdirectory to keep the output and artifacts separate.
```
$ ls -F TestResults.ceelan/mytest.np=2
execute.log mytest.vvt@ vvtest_util.py vvtest_util.pyc vvtest_util.sh
```
The output from the test script (both stdout and stderr) is contained in
the file `execute.log`.
## History
**vvtest** evolved from Mike Wong's test infrastructure in the late 1990s,
through a python rewrite in the mid 2000s, to a refactoring in 2016 to
make it a project independent utility.
## Copyright
Copyright 2018 National Technology & Engineering Solutions of Sandia, LLC
(NTESS). Under the terms of Contract DE-NA0003525 with NTESS, the U.S.
Government retains certain rights in this software.
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.