https://github.com/fredericbonnet/cmake-test-explorer
CMake Test Explorer for Visual Studio Code
https://github.com/fredericbonnet/cmake-test-explorer
c cmake cpp ctest test-automation test-explorer testing vscode vscode-extension
Last synced: 9 months ago
JSON representation
CMake Test Explorer for Visual Studio Code
- Host: GitHub
- URL: https://github.com/fredericbonnet/cmake-test-explorer
- Owner: fredericbonnet
- License: bsd-3-clause
- Created: 2019-08-16T09:09:31.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2024-05-04T20:06:48.000Z (almost 2 years ago)
- Last Synced: 2024-05-04T21:21:14.592Z (almost 2 years ago)
- Topics: c, cmake, cpp, ctest, test-automation, test-explorer, testing, vscode, vscode-extension
- Language: TypeScript
- Homepage:
- Size: 329 KB
- Stars: 7
- Watchers: 3
- Forks: 10
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# CMake Test Explorer for Visual Studio Code
Run your [CMake] tests using Visual Studio Code's [native Testing UI][vscode-testing].
## Features
- Autodects CMake build directories in workspace
- Reasonable defaults for zero-config
- Integrates with CMake Tools
- Runs tests in parallel
- Supports custom test properties for advanced features like hierarchical suites
and test locations
## Configuration
| Property | Description | Default value |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| `cmakeExplorer.buildConfig` | Name of the CMake build configuration. Can be set to any standard or custom configuration name (e.g. `Debug`, `Release`, `RelWithDebInfo`, `MinSizeRel` ). Case-insensitive. | `${buildType}` (see [Variable substitution](#variable-substitution)) |
| `cmakeExplorer.cmakeIntegration` | Integrate with the [CMake Tools][cmake-tools] extension for additional variables. See [Variable substitution](#variable-substitution) for more info. | `true` |
| `cmakeExplorer.debugConfig` | Custom debug configuration to use. See [Debugging](#debugging) for more info. | Empty |
| `cmakeExplorer.parallelJobs` | Maximum number of parallel test jobs to run (zero=autodetect, 1 or negative=disable). See [Parallel test jobs](#parallel-test-jobs) for more info. | 0 |
| `cmakeExplorer.extraCtestLoadArgs` | Extra command-line arguments passed to CTest at load time. For example, `-R foo` will only load the tests containing the string `foo`. | Empty |
| `cmakeExplorer.extraCtestRunArgs` | Extra command-line arguments passed to CTest at run time. For example, `-V` will enable verbose output from tests. | Empty |
| `cmakeExplorer.extraCtestEnvVars` | Extra environment variables passed to CTest at run time. | Empty |
| `cmakeExplorer.suiteDelimiter` | Delimiter used to split CMake test names into suite/test hierarchy. For example, if you name your tests `suite1/subsuite1/test1`, `suite1/subsuite1/test2`, `suite2/subsuite3/test4`, etc. you may set this to `/` in order to group your suites into a tree. If empty, the tests are not grouped. | Empty |
| `cmakeExplorer.testFileVar` | CTest environment variable defined for a test, giving the path of the source file containing the test. See [Source files](#source-files) for more info. | Empty |
| `cmakeExplorer.testLineVar` | CTest environment variable defined for a test, giving the line number within the file where the test definition starts (if known). See [Source files](#source-files) for more info. | Empty |
| `cmakeExplorer.errorPattern` | Regular expression used to match error lines in test outputs. See [Error pattern](#error-pattern) for more info. | GCC-style pattern |
## Variable substitution
Some options support the replacement of special values in their string value by
using a `${variable}` syntax.
The following built-in variables are expanded:
| Variable | Expansion |
| -------------------- | ---------------------------------------------- |
| `${workspaceFolder}` | The full path to the workspace root directory. |
Environments variables are prefixed with `env:`. For example `${env:HOME}` will
be substituted with the home path on Unix systems.
| Variable | Expansion |
| ------------------ | ----------------------------------------------------------------- |
| `${env:}` | The value of the environment variable `VARNAME` at session start. |
_(Note: On Windows, variable names are case insensitive but must be uppercase
for `env:` substitition to work properly)_
Additionally, if the [CMake Tools][cmake-tools] extension is active in the
current workspace and `cmakeExplorer.cmakeIntegration` is enabled, then the
following variables can be used:
| Variable | Expansion | Default value |
| ------------------- | --------------------------------------------------------------------------- | ------------- |
| `${buildType}` | The current CMake build type. For example: `Debug`, `Release`, `MinSizeRel` | `Debug` |
| `${buildDirectory}` | The full path to the current CMake build directory. | Empty |
If you want the extension to infer the right configuration automatically from
CMake Tools, simply keep these default settings:
| Property | Value |
| --------------------------- | -------------- |
| `cmakeExplorer.buildConfig` | `${buildType}` |
If these variables are missing from the current settings (for example, if the
CMake Tools extension is missing or disabled) then they are substituted with an
empty string, falling back to default behavior.
Note that any change to the CMake Tools configuration, either from the settings
or the status bar, requires a manual test reload from the Testing UI.
## Source files
The Testing UI has a feature to link tests with their source files. CMake
provides the [`set_tests_properties()`][cmake-set_tests_properties] command to
associate tests with various metadata, however it only support a predefined
[list of properties][cmake-test-properties], and none of them seems suitable for
this purpose. To support this feature anyway, the extension expects that the
file path and line number be passed as test environment variables using the
[`ENVIRONMENT`][cmake-environment] property, like so:
```
add_test(
NAME
COMMAND [ ...]
)
set_tests_properties( PROPERTIES
ENVIRONMENT "TEST_FILE=;TEST_LINE="
)
```
Here we are using `TEST_FILE` and `TEST_LINE` environment variables but you are
free to choose other variable names. You can then edit the
`cmakeExplorer.testFileVar` and `cmakeExplorer.testLineVar` settings
accordingly, and you should see an extra '**Go to Test**' entry in the
contextual menu.
Note that the `cmakeExplorer.testFileVar` setting must be set for these features
to work, however if the `cmakeExplorer.testLineVar` setting is missing or the
variable is not set for a given test then the '**Go to Test**' will simply open
the file.
## Error pattern
The `cmakeExplorer.errorPattern` setting can be used to capture error messages
on the test output. If the `testExplorer.useNativeTesting` setting is enabled,
the captured messages will be displayed in the file editor next to the line
where the error occurred.
This setting expects a regular expression string with the following named
capturing groups:
- `file`
- `line`
- `severity` (optional)
- `message`
See the [MDN documentation on regular expression groups][re-groups] for more
information on the syntax. For example, here is the default value that captures
GCC-style error messages:
```
^(?[^<].*?):(?\\d+):\\d*:?\\s+(?(?:fatal\\s+)?(?:warning|error)):\\s+(?.*)$
```
This pattern will match the following error message:
```
/path/to/my/file.c:123: error: unexpected value
```
The captured groups will be:
| Name | Value |
| ---------- | -------------------- |
| `file` | `/path/to/my/file.c` |
| `line` | 123 |
| `severity` | `error` |
| `message` | `unexpected value` |
The advantage of using named groups is that their order does not matter. Let's
consider the following error message:
```
ERROR: assertion failed on line 123 (file=/path/to/my/file.c)
```
To capture this message you could use the following syntax:
```
^(?ERROR):\s+(?.*)\s+on line (?\d+) \(file=(?.+)\)$
```
## Debugging
The extension comes pre-configured with sensible defaults for debugging tests:
```json
{
"name": "CTest",
"type": "cppdbg",
"request": "launch",
"windows": {
"type": "cppvsdbg"
},
"linux": {
"type": "cppdbg",
"MIMode": "gdb"
},
"osx": {
"type": "cppdbg",
"MIMode": "lldb"
}
}
```
You can also use a custom configuration defined in the standard `launch.json`.
To do so, edit the `cmakeExplorer.debugConfig` setting with the name of the
debug configuration to use.
Debugging a test will overwrite the following debug configuration fields with
values from the CTest metadata:
| Field | Value |
| --------- | -------------------------------- |
| `name` | `CTest ${test name}` |
| `program` | CTest `COMMAND` option |
| `args` | CTest arguments |
| `cwd` | CTest `WORKING_DIRECTORY` option |
For example, if you want the debugger to stop at the entry point of your tests,
add the following config in your `launch.json` then set
`cmakeExplorer.debugConfig` to "`myCustomDebugConfig`" :
```json
{
"name": "myCustomDebugConfig",
"type": "cppdbg",
"request": "launch",
"stopAtEntry": true,
"windows": {
"type": "cppvsdbg"
}
}
```
## Parallel test jobs
The extension can run test jobs in parallel. The maximum number of jobs to run
is the first non-zero value in the following order:
- The `cmakeExplorer.parallelJobs` setting (see [Configuration](#configuration))
- The `cmake.ctest.parallelJobs` then `cmake.parallelJobs` settings if the
[CMake Tools][cmake-tools] extension is installed
- The number of processors on the machine
A negative value will disable parallel execution.
The extension uses the native CTest parallel execution support (i.e. the
`-j|--parallel` command-line option).
## Troubleshooting
First, make sure that CTest works from the command line. Some issues come from
the CTest configuration and not the extension itself. See issues #10 and #14 for
examples of such cases.
### The Testing UI displays an error
Clicking on the error message in the Testing panel should open the log panel
with the output of the CTest command used by the extension to load the test
list.
- `SyntaxError: Unexpected token T in JSON at position 0`
The extension requires CTest option `--show-only=json-v1` to load the test
list. This option was introduced with CMake version 3.14. Make sure to use a
version that supports this flag. See issue #2.
- `Error: CMake cache file /path/to/project/${buildDirectory}/CMakeCache.txt does not exist`
The `cmakeExplorer.cmakeIntegration` flag is enabled by default. This adds
support for extra variables in other settings (See [Variable
substitution](#variable-substitution) for more info). If the extension is not
installed or active then these variables are not substituted. You can activate
the extension's log panel in the settings for more details.
### The Testing UI shows no error but the test list is empty
The extension autodetects CMake build directories by finding all
`CMakeCache.txt` in the workspace. Ensure you have at least one and run `ctest
-C Debug` from there
[vscode-testing]: https://code.visualstudio.com/docs/debugtest/testing
[cmake-tools]: https://marketplace.visualstudio.com/items?itemName=ms-vscode.cmake-tools
[cmake]: https://cmake.org
[cmake-set_tests_properties]: https://cmake.org/cmake/help/latest/command/set_tests_properties.html
[cmake-test-properties]: https://cmake.org/cmake/help/latest/manual/cmake-properties.7.html#test-properties
[cmake-environment]: https://cmake.org/cmake/help/latest/prop_test/ENVIRONMENT.html
[re-groups]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions/Groups_and_Backreferences