Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/jrmadsen/pyctest
Python bindings of select portions of CMake/CTest package -- enabling generation of CTest test files from Python without a CMake build system
https://github.com/jrmadsen/pyctest
cmake cmake-modules cmake-resources cmake-scripts cmake-toolchain cmake-toolchains cpack ctest ctest-extension cxx pybind11 python test-automation test-driven-development test-framework test-runner testing testing-framework testing-tools
Last synced: 2 months ago
JSON representation
Python bindings of select portions of CMake/CTest package -- enabling generation of CTest test files from Python without a CMake build system
- Host: GitHub
- URL: https://github.com/jrmadsen/pyctest
- Owner: jrmadsen
- License: other
- Created: 2018-06-07T12:52:30.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2022-06-29T04:23:05.000Z (over 2 years ago)
- Last Synced: 2024-11-05T13:49:17.909Z (3 months ago)
- Topics: cmake, cmake-modules, cmake-resources, cmake-scripts, cmake-toolchain, cmake-toolchains, cpack, ctest, ctest-extension, cxx, pybind11, python, test-automation, test-driven-development, test-framework, test-runner, testing, testing-framework, testing-tools
- Language: C++
- Size: 1.36 MB
- Stars: 3
- Watchers: 2
- Forks: 1
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGES.md
- License: LICENSE
Awesome Lists containing this project
README
# PyCTest
| Travis | AppVeyor |
| ------ | -------- |
| [](https://travis-ci.org/jrmadsen/pyctest) | [](https://ci.appveyor.com/project/jrmadsen/pyctest/branch/master) |
[PyCTest Documentation (readthedocs)](https://pyctest.readthedocs.io/en/latest/)
================================================================================
## GitHub
- `git clone https://github.com/jrmadsen/pyctest.git`
## PyPi
- `pip install pyctest`
## Anaconda Cloud
- `conda install -c jrmadsen pyctest`
| Name | Version | Platforms | Downloads |
| --- | --- | --- | --- |
| [](https://anaconda.org/jrmadsen/pyctest) | [](https://anaconda.org/jrmadsen/pyctest) | [](https://anaconda.org/jrmadsen/pyctest) | [](https://anaconda.org/jrmadsen/pyctest) |
## conda-forge
- `conda install -c conda-forge pyctest`
| Name | Version | Platforms | Downloads |
| --- | --- | --- | --- |
| [](https://anaconda.org/conda-forge/pyctest) | [](https://anaconda.org/conda-forge/pyctest) | [](https://anaconda.org/conda-forge/pyctest) | [](https://anaconda.org/conda-forge/pyctest) |
Benefits
========
- Integration into continuous integration systems (e.g. Travis, AppVeyor, CircleCI, etc.) and pushing to CDash dashboard will combine all the results in one place
- The warnings and errors are enumerated in CDash (no more parsing stdout logs for errors)
- Easily create platform-independent testing
- No need to migrate build system to CMake -- just specify `pyctest.BUILD_COMMAND`
General Setup
=============
- Add PyCTest to conda environment: `conda install pyctest`
- Write a driver Python script, e.g. `pyctest-runner.py`, in top of the source code tree
## Example for Python project
The following is an example for a Python code with a compiled C extension that uses `nosetests` for unit-testing:
```python
#!/usr/bin/env python
import os, sys, platform
import pyctest.pyctest as pyctest
import pyctest.helpers as helpers
parser = helpers.ArgumentParser("ProjectName", source_dir=os.getcwd(),binary_dir=os.getcwd(), vcs_type="git")
args = parser.parse_args()
pyctest.BUILD_NAME = "{}".format(args.build)
pyctest.BUILD_COMMAND = "python setup.py build_ext --inplace"
test = pyctest.test()
test.SetName("unittest")
# insert the command to run the tests for project
test.SetCommand(["nosetests"])
pyctest.run()
```
## Example for autotools project
```python
#!/usr/bin/env python
import os, sys, platform
import multiprocessing as mp
import pyctest.pyctest as pyctest
import pyctest.helpers as helpers
parser = helpers.ArgumentParser("ProjectName", source_dir=os.getcwd(), binary_dir=os.getcwd(),
vcs_type="git")
parser.add_argument("-n", "--build", type=str, required=True, help="Build name for identification")
args = parser.parse_args()
# CONFIGURE_COMMAND can only run one command so if autogen is required, just execute it here
cmd = pyctest.command(["./autogen.sh"])
cmd.SetWorkingDirectory(pyctest.SOURCE_DIRECTORY)
cmd.SetErrorQuiet(False)
cmd.Execute()
pyctest.BUILD_NAME = "{}".format(args.build)
pyctest.CONFIGURE_COMMAND = "./configure"
pyctest.BUILD_COMMAND = "make -j{}".format(mp.cpu_count())
test = pyctest.test()
test.SetName("unittest")
# insert the command to run the tests for project
test.SetCommand(["./run-testing.sh"])
pyctest.run()
```
## Example for CMake project
```python
#!/usr/bin/env python
import os
import sys
import platform
import multiprocessing as mp
import pyctest.pyctest as pyctest
import pyctest.helpers as helpers
binary_dir = os.path.join(os.getcwd(), "build-ProjectName")
parser = helpers.ArgumentParser("ProjectName", os.getcwd(), binary_dir)
parser.add_argument("-n", "--build", type=str, required=True, help="Build name for identification")
args = parser.parse_args()
pyctest.BUILD_NAME = "{}".format(args.build)
pyctest.UPDATE_COMMAND = "git"
pyctest.CONFIGURE_COMMAND = "cmake {}".format(pyctest.SOURCE_DIRECTORY)
pyctest.BUILD_COMMAND = "cmake --build {} --target all -- -j{}".format(pyctest.BINARY_DIRECTORY, mp.cpu_count())
test = pyctest.test()
test.SetName("unittest")
# insert the command to run the tests for project
test.SetCommand(["./run-testing.sh"])
pyctest.run()
```
### Python Modules
- `import pyctest` -- global package
- `import pyctest.pyctest` -- CTest module
- `import pyctest.pycmake` -- CMake module
- `import pyctest.helpers` -- Helpers module
- includes command line arguments (`argparse`) for PyCTest
### Direct Access to CMake/CTest/CPack Executables
- `python -m pyctest.cmake ` == `cmake `
- `python -m pyctest.ctest ` == `ctest `
- `python -m pyctest.cpack ` == `cpack `
Following Python code:
```python
from pyctest.ctest import CTest
from pyctest.cmake import CMake
from pyctest.cpack import CPack
CMake({"CMAKE_BUILD_TYPE":"Release"}, os.getcwd(), "-G", "Ninja")
CTest("--build-and-test", os.getcwd(), "-VV")
CPack("-G", "TGZ")
```
is equivalent to the following shell commands:
```shell
$ cmake -DCMAKE_BUILD_TYPE=Release ${PWD} -G Ninja
$ ctest --build-and-test ${PWD} -VV
$ cpack -G TGZ
```
## Standard Configuration Variables
- `pyctest.PROJECT_NAME`
- `pyctest.SOURCE_DIRECTORY`
- `pyctest.BINARY_DIRECTORY`
- `pyctest.SITE`
- `pyctest.BUILD_NAME`
- `pyctest.TRIGGER`
- `pyctest.CHECKOUT_COMMAND`
- `pyctest.BUILD_COMMAND`
- `pyctest.MODEL`
- `pyctest.CUSTOM_COVERAGE_EXCLUDE`
- `pyctest.CUSTOM_MAXIMUM_NUMBER_OF_ERRORS`
- `pyctest.CUSTOM_MAXIMUM_NUMBER_OF_WARNINGS`
- `pyctest.CUSTOM_MAXIMUM_PASSED_TEST_OUTPUT_SIZE`
### Setting Arbitrary Variables
```python
pyctest.set("CTEST_TOKEN_FILE", "${CMAKE_CURRENT_LIST_DIR}/.ctest-token")
```
### Generating a Test
```python
test = pyctest.test()
test.SetName("nosetests")
test.SetCommand(["nosetests", "test", "--cover-xml", "--cover-xml-file=coverage.xml"])
# set directory to run test
test.SetProperty("WORKING_DIRECTORY", pyctest.BINARY_DIRECTORY)
test.SetProperty("RUN_SERIAL", "ON")
test.SetProperty("ENVIRONMENT", "OMP_NUM_THREADS=1")
```
### Examples
- [Basic example](https://github.com/jrmadsen/pyctest/blob/master/examples/Basic/README.md)
- [Advanced example](https://github.com/jrmadsen/pyctest/blob/master/examples/TomoPy/README.md)
- includes submission to CDash dashboard
### CDash Integration Example
Results from running the TomoPy example can be found at the [TomoPy CDash Testing Dashboard @ NERSC](https://cdash.nersc.gov/index.php?project=TomoPy)
- Python code with C extensions without CMake build system
- The build logs from "python setup.py install" are registered in the "Build" section
- The `nosetests test` command + other are wrapped into CTests
## Testing Example
PyCTest can be used to simple execute tests and submit to a dashboard without any configuration, build, etc. steps
```python
#!/usr/bin/env python
import os
import sys
import shutil
import argparse
import platform
import traceback
import pyctest.pyctest as pyctest
import pyctest.pycmake as pycmake
import pyctest.helpers as helpers
if __name__ == "__main__":
directory = os.path.join(os.getcwd(), "pycm-test")
# these are required
pyctest.PROJECT_NAME = "PyCTest"
pyctest.SOURCE_DIRECTORY = directory
pyctest.BINARY_DIRECTORY = directory
args = helpers.ArgumentParser(pyctest.PROJECT_NAME,
pyctest.SOURCE_DIRECTORY,
pyctest.BINARY_DIRECTORY).parse_args()
# set explicitly
pyctest.MODEL = "Continuous"
pyctest.SITE = platform.node()
# create a Test object
test = pyctest.test()
test.SetName("list_directory")
test.SetCommand(["ls", directory])
test.SetProperty("WORKING_DIRECTORY", os.getcwd())
# create a second test
# previous test is already stored by PyCTest
test = pyctest.test()
test.SetName("hostname")
test.SetCommand(["hostname"])
test.SetProperty("TIMEOUT", "10")
# generate the CTestConfig.cmake and CTestCustom.cmake
pyctest.generate_config(directory)
# generate the CTestTestfile.cmake file
pyctest.generate_test_file(directory)
# run CTest -- e.g. ctest -VV ${PWD}/pycm-test
pyctest.run(pyctest.ARGUMENTS, directory)
```
```bash
CTest arguments (default): '-V -DSTAGES=Start;Update;Configure;Build;Test;Coverage;MemCheck -S Stages.cmake -j1'
Writing CTest test file: "/Users/jrmadsen/devel/c++/pyctest-master/pycm-test/CTestTestfile.cmake"...
Generating test "list_directory"...
Generating test "hostname"...
-- STAGES = Start;Update;Configure;Build;Test;Coverage;MemCheck
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Running CTEST_START stage...
Run dashboard with model Continuous
Source directory: /Users/jrmadsen/devel/c++/pyctest-master/pycm-test
Build directory: /Users/jrmadsen/devel/c++/pyctest-master/pycm-test
Track: Continuous
Reading ctest configuration file: /Users/jrmadsen/devel/c++/pyctest-master/pycm-test/CTestConfig.cmake
Site: JRM-macOS-DOE.local
Build name: [Darwin macOS 10.13.6 x86_64] [Python 3.6.7]
Use Continuous tag: 20181129-2118
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_UPDATE stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_CONFIGURE stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_BUILD stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Running CTEST_TEST stage...
Test project /Users/jrmadsen/devel/c++/pyctest-master/pycm-test
Start 1: list_directory
1/2 Test #1: list_directory ................... Passed 0.00 sec
Start 2: hostname
2/2 Test #2: hostname ......................... Passed 0.00 sec
100% tests passed, 0 tests failed out of 2
Total Test time (real) = 0.01 sec
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_COVERAGE stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_MEMCHECK stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_SUBMIT stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Finished Continuous Stages (Start;Update;Configure;Build;Test;Coverage;MemCheck)
```
## Command Line Interface
```shell
PyCTest args: ['--help']
CTest args: []
CMake args: []
usage: pyctest-runner.py [-h] [-F] [-S] [-A] [-j ] [-m ] [-b ] [-i ] [-o ] [-M ] [-H ] [-P ] [-N ]
[-C [ ...]] [-L [ ...]] [-T [ [ ...]]] [-fc ] [-cc ] [-cxx ]
[--pyctest-token ] [--pyctest-token-file ] [--pyctest-vcs-type ] [--pyctest-build-type ]
[--pyctest-trigger ] [--pyctest-use-launchers ] [--pyctest-ctest-args [ [ ...]]] [--pyctest-cdash-version ]
[--pyctest-submit-retry-count ] [--pyctest-submit-retry-delay ] [--pyctest-curl-options ]
[--pyctest-drop-location ] [--pyctest-drop-method ] [--pyctest-drop-site ] [--pyctest-drop-site-password ]
[--pyctest-drop-site-user ] [--pyctest-nightly-start-time ] [--pyctest-update-version-only]
PyCTest argparse. Arguments after first '--' are passed directly to CTest, arguments after second '--' are passed directly to CMake
optional arguments:
-h, --help show this help message and exit
-F, --pyctest-clean-first Remove standard PyCTest files and binary directory (if not source directory)
-S, --pyctest-submit Enable submission to dashboard
-A, --pyctest-append Append to last invocation of CTest
-j , --pyctest-jobs number of concurrent jobs
-m , --pyctest-mode Run specific stage. Choices:
[Start,Update,Configure,Build,Test,Coverage,MemCheck,Submit,Stages]
-b , --pyctest-build-name Build name for identification
-i , --pyctest-source-dir Source directory
-o , --pyctest-binary-dir Binary/build/working directory
-M , --pyctest-model CTest submission model (track). Choices: [Nightly,Continuous,Experimental]
-H , --pyctest-site CTest submission site
-P , --pyctest-python-exe Python executable to use. This can be absolue, relative, or CMake path
-N , --pyctest-project-name Name of project using PyCTest
-C [ ...], --pyctest-cleanup [ ...] Remove standard PyCTest files and binary directory (if not source directory) and
exit
-L [ ...], --pyctest-subproject-labels [ ...] Add labels for subproject
-T [ [ ...]], --pyctest-stages [ [ ...]] Run multiple stages. Choices:
[Start,Update,Configure,Build,Test,Coverage,MemCheck,Submit]
-fc , --pyctest-fortran-compiler Specify Fortan compiler (if needed)
-cc , --pyctest-c-compiler Specify the C compiler (if needed)
-cxx , --pyctest-cxx-compiler Specify C++ compiler (if needed)
--pyctest-token CTest site token for submission
--pyctest-token-file Path to file for CTest site token for submission
--pyctest-vcs-type Set to enable revision ID discovery during the Update stage. Choices:
[bzr,cvs,git,hg,p4,svn]
--pyctest-build-type Specify CMAKE_BUILD_TYPE (if using CMake). Choices:
[Release,RelWithDebInfo,Debug,MinSizeRel]
--pyctest-trigger DEPRECATED
--pyctest-use-launchers Enable launchers
--pyctest-ctest-args [ [ ...]] CTest arguments
--pyctest-cdash-version Set CTest variable: 'CTEST_CDASH_VERSION'
--pyctest-submit-retry-count Set CTest variable: 'CTEST_SUBMIT_RETRY_COUNT'
--pyctest-submit-retry-delay Set CTest variable: 'CTEST_SUBMIT_RETRY_DELAY'
--pyctest-curl-options Set CTest variable: 'CTEST_CURL_OPTIONS'
--pyctest-drop-location Set CTest variable: 'CTEST_DROP_LOCATION'
--pyctest-drop-method Set CTest variable: 'CTEST_DROP_METHOD'
--pyctest-drop-site Set CTest variable: 'CTEST_DROP_SITE'
--pyctest-drop-site-password Set CTest variable: 'CTEST_DROP_SITE_PASSWORD'
--pyctest-drop-site-user Set CTest variable: 'CTEST_DROP_SITE_USER'
--pyctest-nightly-start-time Set CTest variable: 'CTEST_NIGHTLY_START_TIME'
--pyctest-update-version-only Specify that you want the version control update command to only discover the
current version that is checked out, and not to update to a different version
```