Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/trivago/cucable-plugin
Maven plugin that simplifies running Cucumber scenarios in parallel.
https://github.com/trivago/cucable-plugin
cucumber-jvm cucumber-scenarios gherkin-parser maven-failsafe maven-plugin parallelization test-automation test-runner
Last synced: about 1 month ago
JSON representation
Maven plugin that simplifies running Cucumber scenarios in parallel.
- Host: GitHub
- URL: https://github.com/trivago/cucable-plugin
- Owner: trivago
- License: apache-2.0
- Created: 2017-06-09T13:29:55.000Z (over 7 years ago)
- Default Branch: main
- Last Pushed: 2024-05-08T16:08:38.000Z (7 months ago)
- Last Synced: 2024-08-04T01:14:46.828Z (4 months ago)
- Topics: cucumber-jvm, cucumber-scenarios, gherkin-parser, maven-failsafe, maven-plugin, parallelization, test-automation, test-runner
- Language: Java
- Homepage:
- Size: 1.43 MB
- Stars: 115
- Watchers: 24
- Forks: 47
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
- awesome-bdd - Cucable Maven Plugin - Cucable is a Maven plugin for Cucumber scenarios that simplifies fine-grained and efficient parallel test runs. (Tools / Java)
README
[](http://www.apache.org/licenses/LICENSE-2.0)
[](https://repo1.maven.org/maven2/com/trivago/rta/cucable-plugin/)
_Run Cucumber Scenarios in Parallel with Maven_
- [Cucable Maven Plugin](#cucable-maven-plugin)
- [Cucumber 4](#cucumber-4)
- [Cucumber 5 and higher](#cucumber-5-and-higher)
- [JUnit 5](#junit-5)
- [Repository Structure](#repository-structure)
- [Changelog](#changelog)
- [Maven dependency](#maven-dependency)
- [How it works](#how-it-works)
- [Runner template placeholders](#runner-template-placeholders)
- [[CUCABLE:RUNNER]](#cucablerunner)
- [[CUCABLE:FEATURE]](#cucablefeature)
- [Custom template placeholders - [CUCABLE:CUSTOM:xxx]](#custom-template-placeholders---cucablecustomxxx)
- [One runner per generated scenario](#one-runner-per-generated-scenario)
- [One runner per group of generated scenarios](#one-runner-per-group-of-generated-scenarios)
- [No runners](#no-runners)
- [Typical workflow](#typical-workflow)
- [1. Generation of runners and features](#1-generation-of-runners-and-features)
- [Parameters](#parameters)
- [sourceRunnerTemplateFile](#sourcerunnertemplatefile)
- [Using a java file as a runner template](#using-a-java-file-as-a-runner-template)
- [Using a text file as a runner template](#using-a-text-file-as-a-runner-template)
- [sourceFeatures](#sourcefeatures)
- [Combining different feature sources](#combining-different-feature-sources)
- [generatedFeatureDirectory](#generatedfeaturedirectory)
- [generatedRunnerDirectory](#generatedrunnerdirectory)
- [numberOfTestRuns](#numberoftestruns)
- [includeScenarioTags](#includescenariotags)
- [parallelizationMode](#parallelizationmode)
- [logLevel](#loglevel)
- [desiredNumberOfRunners](#desirednumberofrunners)
- [desiredNumberOfFeaturesPerRunner](#desirednumberoffeaturesperrunner)
- [scenarioNames](#scenarionames)
- [Generating runners and features inside target directory](#generating-runners-and-features-inside-target-directory)
- [Complete Example](#complete-example)
- [Source feature file](#source-feature-file)
- [Runner template file](#runner-template-file)
- [Generated Scenarios](#generated-scenarios)
- [Generated runners](#generated-runners)
- [2. Running the generated tests with Maven failsafe](#2-running-the-generated-tests-with-maven-failsafe)
- [3. Aggregation of a single test report after all test runs](#3-aggregation-of-a-single-test-report-after-all-test-runs)
- [Example project](#example-project)
- [Appendix](#appendix)
- [Building](#building)
- [License](#license)
- [Credits](#credits)
# Cucable Maven Plugin
Cucable is a Maven plugin for [Cucumber](https://cucumber.io) scenarios that simplifies fine-grained and efficient parallel test runs.
This plugin does the following:
- Generate single [Cucumber](https://cucumber.io) features containing one single scenario each (scenario outlines are also split up into separate scenarios)
- Generating [Cucumber](https://cucumber.io) runners
- for every generated "single scenario" feature file or
- for multiple generated "single scenario" feature files
Those generated runners and features can then be used with [Maven Failsafe](http://maven.apache.org/surefire/maven-failsafe-plugin/) in order to parallelize test runs.
This also works for **non-english** feature files!
## Cucumber 4
Even though Cucumber 4 supports basic parallel runs, Cucable has more options that may be beneficial for your use case:
* It supports running single scenarios, complete features or sequences of single scenarios in parallel
* It supports splitting scenarios and attaching them to a fixed number of runners
* It supports splitting scenarios and attaching batches of them to a dynamic number of runners
* You don't need any test framework changes because Cucable runs before the framework invocations
* You have full control over your runners because of template variables and custom placeholders
## Cucumber 5 and higher
* Cucumber starting with version 5 (using testng or junit 5) can natively run features and scenarios in parallel. Cucable __can__ be used but does not __have__ to be.
## JUnit 5
When using the JUnit 5 platform, Cucable can still help parallelize scenarios more fine-grained and with more options than the standard JUnit and Cucumber properties.
## Repository Structure
* [plugin-code](plugin-code) contains the full plugin source code.
* [example-project](example-project) contains an example Maven project to see the plugin in action.
## Changelog
All changes are documented in the [full changelog](CHANGELOG.md).
## Maven dependency
```xml
com.trivago.rta
cucable-plugin
(check version on top of the page)
```
# How it works
* Cucable will cut up feature file into the smallest possible runnable scenarios
* Each generated feature file includes a single scenario
* After this, the runner classes for those generated features are generated based on a provided template file, either
* one runner per generated "single scenario" feature file or
* one runner per group of "single scenario" feature files or
* no runners at all (not needed if your tests are run as unit tests with JUnit 5 or TestNG)
## Runner template placeholders
__Note:__ If you don't need runner classes to be generated, you can skip this section.
### [CUCABLE:RUNNER]
The `[CUCABLE:RUNNER]` template placeholder is automatically replaced with the class name of the generated runner class.
If the generated runner runs only one "single scenario" feature, its name will be the same as the generated feature (e.g. `Runner_MyFeature_scenario001_run001_IT`).
In case the runner runs multiple "single scenario" features, its name will be auto-generated (e.g. `CucableMultiRunner_1da810a2_c4c6_4edb_b078_d81329593950_IT`).
### [CUCABLE:FEATURE]
The `[CUCABLE:FEATURE]` can be placed in the `feature` option of the `@CucumberOptions` block in your template:
@CucumberOptions(
features = {"target/parallel/features/[CUCABLE:FEATURE].feature"}
)
Cucable will automatically detect the string containing the `[CUCABLE:FEATURE]` placeholder and use this to generate one line for each feature this runner should trigger.
### Custom template placeholders - [CUCABLE:CUSTOM:xxx]
In some cases, you may need to set custom values that should be written to your template files.
In this case, just add a block to your POM file:
```
Any value
bar
```
These custom placeholders can be used anywhere in your template:
import cucumber.api.CucumberOptions;
@CucumberOptions(
features = {"target/parallel/features/[CUCABLE:FEATURE].feature"},
plugin = {"json:target/cucumber-report/[CUCABLE:CUSTOM:foo].json"}
)
public class [CUCABLE:RUNNER] {
// [CUCABLE:CUSTOM:somename]
}
In this case the result would be
import cucumber.api.CucumberOptions;
@CucumberOptions(
features = {"target/parallel/features/[CUCABLE:FEATURE].feature"},
plugin = {"json:target/cucumber-report/bar.json"}
)
public class [CUCABLE:RUNNER] {
// Any value
}
**Note:** The custom placeholder names are case-sensitive!
## One runner per generated scenario
This is the default mode of Cucable. Having multiple runners that run one "single scenario" feature each is best for parallelization with [Maven Failsafe](http://maven.apache.org/surefire/maven-failsafe-plugin/).
## One runner per group of generated scenarios
If you use the `desiredNumberOfRunners` or `desiredNumberOfFeaturesPerRunner` option, Cucable will automatically switch to the multi-feature runner mode.
This means that it will only generate the specified number of runners (or features per runner) and distribute the generated features evenly among the runners. This is helpful if a group of scenarios should be executed during each forked run of your test framework.
**Note:** If a runner runs only one feature, it automatically has the same name as the feature. Otherwise it will have a unique auto-generated name.
## No runners
If you set `desiredNumberOfRunners` to `0`, this means that Cucable will not generate runner classes at all. In this case, you do not need to set these properties:
* `sourceRunnerTemplateFile` (as you do not need a template file)
* `generatedRunnerDirectory` (as no runners will be generated)
# Typical workflow
1. Generation of runners and features
2. Running the generated tests with [Maven Failsafe](http://maven.apache.org/surefire/maven-failsafe-plugin/)
3. Aggregation of a single test report after all test runs
The following sections break down the above steps.
## 1. Generation of runners and features
```xml
com.trivago.rta
cucable-plugin
${cucable-plugin.version}
generate-test-resources
generate-test-resources
parallel
src/test/resources/parallel/cucable.template
src/test/resources/features
src/test/resources/parallel/features
src/test/java/parallel/runners
1
@includeMe and @includeMeAsWell
compact
2
```
### Parameters
#### sourceRunnerTemplateFile
__Note:__ This is only needed, when you want to generate runner classes!
The specified file will be used to generate runner classes for the generated feature file that can be run using [Maven Failsafe](http://maven.apache.org/surefire/maven-failsafe-plugin/).
This can be either a text file or a Java class. The difference can be seen below:
##### Using a java file as a runner template
If you use a java file (e.g. _src/test/java/some/template/CucableJavaTemplate.java_), the **[CUCABLE:FEATURE]** placeholder as well as the **class name** will be substituted for the generated feature file name(s).
The **[CUCABLE:RUNNER]** placeholder will be replaced by the runner class name.
Additionally, the **package declaration** will be stripped.
_Example:_
package some.template;
import cucumber.api.CucumberOptions;
@CucumberOptions(
features = {"target/parallel/features/[CUCABLE:FEATURE].feature"},
plugin = {"json:target/cucumber-report/[CUCABLE:RUNNER].json"}
)
public class CucableJavaTemplate {
}
will turn into
import cucumber.api.CucumberOptions;
@CucumberOptions(
features = {"target/parallel/features/MyFeature_scenario001_run001_IT.feature"},
plugin = {"json:target/cucumber-report/MyFeature_scenario001_run001_IT.json"}
)
public class MyFeature_scenario001_run001_IT {
}
// Generated by Cucable from src/test/java/some/template/CucableJavaTemplate.java
In case of a fixed number of runners that contain multiple scenarios (via `desiredNumberOfRunners` property), the runner name will be auto-generated:
import cucumber.api.junit.Cucumber;
import cucumber.api.CucumberOptions;
import org.junit.runner.RunWith;
@RunWith(Cucumber.class)
@CucumberOptions(
features = {"target/parallel/features/Bookmarks_scenario001_run001_IT.feature",
"target/parallel/features/Bookmarks_scenario003_run001_IT.feature"},
plugin = {
"json:target/cucumber-report/CucableMultiRunner_1da810a2_c4c6_4edb_b078_d81329593950_IT.json",
"com.trivago.trupi.plugin.TrupiCucumberPlugin"
},
glue = {"com.trivago.trupi.glue"}
)
public class CucableMultiRunner_1da810a2_c4c6_4edb_b078_d81329593950_IT {
}
// Generated by Cucable from src/test/java/some/template/CucableJavaTemplate.java
##### Using a text file as a runner template
If you use a text file (e.g. _src/test/resources/cucable.template_), all **[CUCABLE:FEATURE]** placeholder will be substituted for the generated feature file name(s). The **[CUCABLE:RUNNER]** placeholder will be replaced by the runner class name.
#### sourceFeatures
This property specifies the location of the features that Cucable should process. It must point to one or more of the following:
* the root path of your __existing__ [Cucumber](https://cucumber.io) _.feature_ files, e.g.
`src/test/resources/features`
* the path to a specific __existing__ [Cucumber](https://cucumber.io) _.feature_ file, e.g.
`src/test/resources/features/MyFeature.feature`
* the path to a specific __existing__ [Cucumber](https://cucumber.io) _.feature_ file with optional line numbers of specific scenarios e.g.
`src/test/resources/features/MyFeature.feature:12:19`
* the path to a [Cucumber](https://cucumber.io) text file containing the path to a feature including line number(s) per line (as written by the [Cucumber rerun reporter plugin](https://cucumber.io/docs/cucumber/reporting/?sbsearch=rerun#built-in-reporter-plugins), e.g.
```
@src/test/resources/rerun.txt
```
__Note:__ The path to a text file has to start with an `@` character!
The file contents can look like this:
```
file:///pathToProject/resources/features/feature1.feature:12
file:///pathToProject/resources/features/feature4.feature:6
```
* the root path of your __existing__ [Cucumber](https://cucumber.io) text files (also starting with an `@`), e.g.
```
@src/test/resources
```
#### Combining different feature sources
Starting from Cucumber 1.10.0, you can use any combination of sources by separating them with commas, e.g.
```xml
src/test/resources/features, @src/test/rerun.txt
```
This example would use all `.feature` files in `src/test/resources/features` __plus__ all features that are defined in `src/test/rerun.txt`!
__Note:__ You need to be careful when selecting multiple sources as the same feature could be generated multiple times if it appears in more than one source.
#### generatedFeatureDirectory
The path where the __generated__ [Cucumber](https://cucumber.io) .feature files should be located (e.g. _src/test/resources/parallel_).
Additionally, a file called `generated-features.properties` will be generated there that shows all generated features along with their reference to the respective source feature.
**Note:** This directory should be located under a valid resource folder to be included as a test source by Maven.
If you want to use a directory inside Maven's target folder, [check this example](#generating-runners-and-features-inside-target-directory).
**Caution:** This directory will be wiped prior to the feature file generation!
#### generatedRunnerDirectory
__Note:__ This is only needed, when you want to generate runner classes!
The path where the __generated__ runner classes should be located (e.g. _src/test/java/parallel/runners_).
**Note:** This directory should be located under a valid source folder to be included as a test source by Maven.
If you want to use a directory inside Maven's target folder, [check this example](#generating-runners-and-features-inside-target-directory).
**Caution:** This directory will be wiped prior to the runner file generation!
#### numberOfTestRuns
Optional number of test runs. This can be used if specific scenarios should be run multiple times.
If this options is not set, its default value is __1__.
For each test run, the whole set of features and runners is generated like this:
- MyFeature_scenario001_run001_IT.feature
- MyFeature_scenario001_run002_IT.feature
- MyFeature_scenario001_run003_IT.feature
- etc.
**Note:** Characters other than letters from A to Z, numbers and underscores will be stripped out of the feature file name.
#### includeScenarioTags
`includeScenarioTags` can be used to provide a [Cucumber tag expression](https://docs.cucumber.io/cucumber/api/#tag-expressions) in order to specify which tags should be included or excluded from scenario generation:
__Example:__ include scenarios that are tagged with _@scenario1_:
```xml
@scenario1
```
__Example:__ include scenarios that are tagged with _@scenario1_ __or__ _@scenario2_:
```xml
@scenario1 or @scenario2
```
__Example:__ include scenarios that are tagged with _@scenario1_ __and__ _@scenario2_:
```xml
@scenario1 and @scenario2
```
__Example:__ include scenarios that are __not__ tagged with _@scenario1_:
```xml
not @scenario1
```
__Example:__ include scenarios that are __not__ tagged with _@scenario1_ but tagged with _scenario2_ __or__ _scenario3_:
```xml
not @scenario1 and (@scenario2 or scenario3)
```
#### parallelizationMode
By default, Cucable uses the `parallelizationMode = scenarios` meaning that feature files are split into individual scenarios that each have a dedicated runner.
Sometimes it may be desirable, to parallelize complete features. When setting the `parallelizationMode = features`, only complete features containing all of their source scenarios are generated so __each runner runs a complete feature__.
```xml
features
```
__Note:__ For this mode to work, `` must specify a directory. Also, `includeScenarioTags` cannot be used.
#### logLevel
By default, Cucable logs all information including
* its own name and version
* all passed property values
* a list of processed feature paths
This can be configured by passing the `logLevel` property:
```
default|compact|minimal|off
```
* _default_ will log all the mentioned information
* _compact_ will only log the plugin name, version, properties and one line of summary
* _minimal_ will only log a summary line
* _off_ will prevent any logging
#### desiredNumberOfRunners
If you set this options, all generated features will be distributed to a fixed set of runner classes. This means that one runner can potentially run multiple features in sequence.
If this option is not set, its default value is `-1` which basically means "Generate a dedicated runner for every generated feature".
__Note:__ This cannot be used together with `desiredNumberOfFeaturesPerRunner`!
#### desiredNumberOfFeaturesPerRunner
If you set this option, all generated features will be distributed to a dynamic set of runner classes so that every runner contains a fixed number of generated features. This means that one runner can potentially run multiple features in sequence.
If this option is not set, its default value is `0` which basically means "Generate a dedicated runner for every generated feature".
__Note:__ This cannot be used together with `desiredNumberOfRunners`!
#### scenarioNames
A comma separated list of strings matching a scenario name, either completely or partially. Please see `--name` in Cucumber command-line options (`java cucumber.api.cli.Main --help` or `mvn test -Dcucumber.options="--help`). If you set this option, only scenarios matching the specified names will be loaded into the generated runners. The number of runner files will default to the number of scenario names and each runner file will contain the scenarios matching 1 name. Please note that this will override `desiredNumberOfRunners`.
For example, if the following scenario names are specified:
```xml
name1,
name2
```
2 runner files will be generated. The first file will contain all the scenarios matching `name1` and the second file will contain all the scenarios matching `name2`.
__Note:__ This *cannot* be used together with `desiredNumberOfFeaturesPerRunner` or `parallelizationMode = features`!
### Generating runners and features inside target directory
It may be desirable for you to generate the Cucable features and runners in Maven's `target` directory.
The advantage of this is that this directory is wiped by the `mvn clean` command and older generated files do not reside in your `src` directory.
In order to achieve this, you can specify subdirectories under target (`${project.build.directory}`) for Cucable, e.g. `${project.build.directory}/parallel/runners` and `${project.build.directory}/parallel/features`
After this step, use the *build-helper-maven-plugin* in your POM file in order to consider the generated runner classes test sources:
```xml
com.trivago.rta
cucable-plugin
${cucable.plugin.version}
generate-test-resources
generate-test-resources
parallel
path_to_template_file
path_to_features
${project.build.directory}/parallel/features
${project.build.directory}/parallel/runners
org.codehaus.mojo
build-helper-maven-plugin
${build.helper.plugin.version}
add-test-source
generate-test-sources
add-test-source
${project.build.directory}/parallel/runners
```
### Complete Example
Below, you can see a full example of what Cucable does.
#### Source feature file
This is our source feature file. It contains a scenario and a scenario outline with two examples.
*MyFeature.feature*
```
Feature: This is the feature name
Scenario: First scenario
Given I am on the start page
And I click the login button
Then I see an error message
Scenario Outline: Second scenario with an amount of
Given I am on the start page
And I add items
And I navigate to the shopping basket
Then I see items
Examples:
| amount |
| 12 |
| 85 |
```
#### Runner template file
This is the runner template file (in this example we use a text file) that is used to generate single scenario runners.
* The placeholder **[CUCABLE:FEATURE]** and its enclosing string will be replaced with the generated feature names by Cucable.
* The placeholder **[CUCABLE:RUNNER]** will be replaced with the generated runner class name by Cucable.
import cucumber.api.CucumberOptions;
@CucumberOptions(
features = {"target/parallel/features/[CUCABLE:FEATURE].feature"},
plugin = {"json:target/cucumber-report/[CUCABLE:RUNNER].json"}
)
public class [CUCABLE:RUNNER] {
}
**Note:** The specified _plugin_ generates Cucumber JSON files which are needed for custom aggregated test reports.
#### Generated Scenarios
For each scenario, a single feature file is created:
*MyFeature_scenario001_run001_IT.feature*
```
Feature: This is the feature name
Scenario: First scenario
Given I am on the start page
And I click the login button
Then I see an error message
```
Note that for the scenario outlines, each example is converted _to its own scenario and feature file_:
*MyFeature_scenario002_run001_IT.feature*
Feature: This is the feature name
Scenario: Second scenario with an amount of 12
Given I am on the start page
And I add 12 items
And I navigate to the shopping basket
Then I see 12 items
*MyFeature_scenario003_run001_IT.feature*
Feature: This is the feature name
Scenario: Second scenario with an amount of 85
Given I am on the start page
And I add 85 items
And I navigate to the shopping basket
Then I see 85 items
#### Generated runners
The generated runners point to each one of the generated feature files (unless you use the `desiredNumberOfRunners` or `desiredNumberOfFeaturesPerRunner` option).
This is an example for one of the generated runners - note how the placeholders are now replaced with the name of the feature to run:
*Runner_MyFeature_scenario001_run001_IT.java*
import cucumber.api.CucumberOptions;
@CucumberOptions(
features = {"target/parallel/features/MyFeature_scenario001_run001_IT.feature"},
plugin = {"json:target/cucumber-report/MyFeature_scenario001_run001_IT.json"}
)
public class MyFeature_scenario001_run001_IT {
}
## 2. Running the generated tests with Maven failsafe
This will skip the unit tests (if any) and run the generated runner classes with [Maven Failsafe](http://maven.apache.org/surefire/maven-failsafe-plugin/).
Since all generated runner classes from the step before end with ___IT__, they are automatically considered integration tests and run with [Maven Failsafe](http://maven.apache.org/surefire/maven-failsafe-plugin/).
**Note:** If all tests should be run regardless of their result, it is important to set ```true``` for [Maven Failsafe](http://maven.apache.org/surefire/maven-failsafe-plugin/) - otherwise the plugin execution will stop on failing tests.
However, if this is specified, the build will not fail in case of failing tests!
To circumvent that, it is possible to specify a custom [rule](https://maven.apache.org/enforcer/enforcer-api/writing-a-custom-rule.html) for [Maven enforcer](https://maven.apache.org/enforcer/maven-enforcer-plugin/) that passes or fails the build depending on custom conditions.
```xml
org.apache.maven.plugins
maven-surefire-plugin
true
org.apache.maven.plugins
maven-failsafe-plugin
Run parallel tests
integration-test
integration-test
true
${maven.fork.count}
false
-Dfile.encoding=UTF-8
true
```
## 3. Aggregation of a single test report after all test runs
We use the [Cluecumber](https://github.com/trivago/cluecumber-report-plugin) plugin to aggregate all generated __.json__ report files into one overall test report.
```xml
com.trivago.rta
cluecumber-report-plugin
${cluecumber.report.version}
report
post-integration-test
reporting
${project.build.directory}/cucumber-report
${project.build.directory}/test-report
```
# Example project
You can test the complete flow and POM configuration by checking out the [Cucable example project](example-project).
# Appendix
## Building
Cucable requires Java >= 8 and Maven >= 3.3.9.
It is available in [Maven central](https://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22com.trivago.rta%22%20AND%20a%3A%22cucable-plugin%22).
## License
Copyright 2017 [trivago N.V.](https://www.trivago.de/impressum)
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
## Credits
This plugin was inspired by the [Cucumber Slices Maven Plugin](https://github.com/DisneyStudios/cucumber-slices-maven-plugin).