https://intuit.github.io/karate
Test Automation Made Simple
https://intuit.github.io/karate
api-testing assertions automated-testing automation bdd contract-testing cucumber developer-tools load-testing microservices mock-server test-automation testing testing-framework testing-tools webdriver
Last synced: 9 months ago
JSON representation
Test Automation Made Simple
- Host: GitHub
- URL: https://intuit.github.io/karate
- Owner: karatelabs
- License: mit
- Created: 2017-02-07T15:59:17.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2024-04-22T20:38:28.000Z (about 2 years ago)
- Last Synced: 2024-05-01T11:38:42.469Z (about 2 years ago)
- Topics: api-testing, assertions, automated-testing, automation, bdd, contract-testing, cucumber, developer-tools, load-testing, microservices, mock-server, test-automation, testing, testing-framework, testing-tools, webdriver
- Language: Java
- Homepage: https://karatelabs.github.io/karate
- Size: 24.9 MB
- Stars: 7,850
- Watchers: 225
- Forks: 1,909
- Open Issues: 61
-
Metadata Files:
- Readme: README.md
- Contributing: .github/CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE
- Code of conduct: .github/CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
README
⏩
Get Started
Get started using VS Code, IntelliJ, Maven, Gradle, NPM,
GitHub Codespaces, Docker or the command-line
💡
Examples
Examples and demos of integrations with other frameworks
📺
Kick Start Videos
Beginner-friendly step-by-step guides to starting from scratch
# Karate
## Test Automation Made `Simple.`
Karate is the only open-source tool to combine API test-automation, [mocks](karate-netty), [performance-testing](karate-gatling) and even [UI automation](karate-core) into a **single**, *unified* framework. The syntax is language-neutral, and easy for even non-programmers. Assertions and HTML reports are built-in, and you can run tests in parallel for speed.
There's also a cross-platform [stand-alone executable](karate-netty#standalone-jar) for teams not comfortable with Java. You don't have to compile code. Just write tests in a **simple**, *readable* syntax - carefully designed for HTTP, JSON, GraphQL and XML. And you can mix API and [UI test-automation](karate-core) within the same test script.
A [Java API](#java-api) also exists for those who prefer to programmatically integrate Karate's rich automation and data-assertion capabilities.
## Hello World
### For API Testing
> If you are familiar with Cucumber / Gherkin, the [*big difference*](#cucumber-vs-karate) here is that you **don't** need to write extra "glue" code or Java "step definitions" !
It is worth pointing out that JSON is a 'first class citizen' of the syntax such that you can express payload and expected data without having to use double-quotes and without having to enclose JSON field names in quotes. There is no need to 'escape' characters like you would have had to in Java or other programming languages.
And you don't need to create additional Java classes for any of the payloads that you need to work with.
# Index
Start
Maven
| Gradle
| Quickstart
| Standalone Executable
| Naming Conventions
| Script Structure
Run
JUnit 5
| Command Line
| IDE Support
| Tags / Grouping
| Parallel Execution
| Java API
| jbang
Report
Configuration
| Environment Switching
| Reports
| JUnit HTML Report
| Dry Run
| Report Verbosity
| Logging
| Log Masking
Types
JSON
| XML
| JavaScript Functions
| Reading Files
| Type / String Conversion
| Floats and Integers
| Embedded Expressions
| JsonPath
| XPath
| Karate Expressions
Variables
def
| text
| table
| yaml
| csv
| string
| json
| xml
| xmlstring
| bytes
| copy
Actions
assert
| print
| replace
| get
| set
| remove
| configure
| call
| callonce
| eval
| listen
| doc
| read()
| compareImage
| karate JS API
HTTP
url
| path
| request
| method
| status
| soap action
| retry until
Request
param
| header
| cookie
| form field
| multipart file
| multipart field
| multipart entity
| params
| headers
| cookies
| form fields
| multipart files
| multipart fields
Response
response
| responseBytes
| responseStatus
| responseHeaders
| responseCookies
| responseTime
| responseType
| requestTimeStamp
Assert
match ==
| match !=
| match contains
| match contains only
| match contains any
| match contains deep
| match contains only deep
| match !contains
| match each
| match each contains deep
| match header
| Fuzzy Matching
| Schema Validation
| contains short-cuts
Re-Use
Calling Other *.feature Files
| Data Driven Features
| Calling JavaScript Functions
| Calling Java Code
| Commonly Needed Utilities
| Data Driven Scenarios
Advanced
Polling
| Conditional Logic
| Before / After Hooks
| JSON Transforms
| Loops
| HTTP Basic Auth
| Header Manipulation
| GraphQL
| Websockets / Async
| call vs read()
More
Test Doubles
| Performance Testing
| UI Testing
| Desktop Automation
| VS Code / Debug
| Karate vs REST-assured
| Karate vs Cucumber
| Examples and Demos
# Features
* Java knowledge is not required and even non-programmers can write tests
* Scripts are plain-text, require no compilation step or IDE, and teams can collaborate using Git / standard SCM
* Based on the popular Cucumber / Gherkin standard - with [IDE support](https://github.com/karatelabs/karate/wiki/IDE-Support) and syntax-coloring options
* Elegant [DSL](https://en.wikipedia.org/wiki/Domain-specific_language) syntax 'natively' supports JSON and XML - including [JsonPath](#set) and [XPath](#xpath-functions) expressions
* Eliminate the need for 'Java Beans' or 'helper code' to represent payloads and HTTP end-points, and [dramatically reduce the lines of code](https://twitter.com/KarateDSL/status/873035687817117696) needed for a test
* Ideal for testing the highly dynamic responses from [GraphQL](http://graphql.org) API-s because of Karate's built-in [text-manipulation](#text) and [JsonPath](https://github.com/json-path/JsonPath#path-examples) capabilities
* Tests are super-readable - as scenario data can be expressed in-line, in human-friendly [JSON](#json), [XML](#xml), Cucumber [Scenario](#the-cucumber-way) Outline [tables](#table), or a [payload builder](#set-multiple) approach [unique to Karate](https://gist.github.com/ptrthomas/d6beb17e92a43220d254af942e3ed3d9)
* Express expected results as readable, well-formed JSON or XML, and [assert in a single step](#match) that the entire response payload (no matter how complex or deeply nested) - is as expected
* Comprehensive [assertion capabilities](#fuzzy-matching) - and failures clearly report which data element (and path) is not as expected, for easy troubleshooting of even large payloads
* [Fully featured debugger](https://github.com/karatelabs/karate/wiki/IDE-Support#vs-code-karate-plugin) that can step *backwards* and even [re-play a step while editing it](https://twitter.com/KarateDSL/status/1167533484560142336) - a *huge* time-saver
* Simpler and more [powerful alternative](https://twitter.com/KarateDSL/status/878984854012022784) to JSON-schema for [validating payload structure](#schema-validation) and format - that even supports [cross-field](#referring-to-the-json-root) / domain validation logic
* Scripts can [call other scripts](#calling-other-feature-files) - which means that you can easily re-use and maintain authentication and 'set up' flows efficiently, across multiple tests
* Embedded JavaScript engine that allows you to build a library of [re-usable functions](#calling-javascript-functions) that suit your specific environment or organization
* Re-use of payload-data and user-defined functions across tests is [so easy](#reading-files) - that it becomes a natural habit for the test-developer
* Built-in support for [switching configuration](#switching-the-environment) across different environments (e.g. dev, QA, pre-prod)
* Support for [data-driven tests](#data-driven-tests) and being able to [tag or group](#tags) tests is built-in, no need to rely on an external framework
* Native support for reading [YAML](#yaml) and even [CSV](#csv-files) files - and you can use them for data-driven tests
* Standard Java / Maven project structure, and [seamless integration](#command-line) into CI / CD pipelines - and support for [JUnit 5](#junit-5)
* Option to use as a light-weight [stand-alone executable](https://github.com/karatelabs/karate/tree/master/karate-netty#standalone-jar) - convenient for teams not comfortable with Java
* Multi-threaded [parallel execution](#parallel-execution), which is a huge time-saver, especially for integration and end-to-end tests
* Built-in [test-reports](#test-reports) compatible with Cucumber so that you have the option of using third-party (open-source) maven-plugins for even [better-looking reports](karate-demo#example-report)
* Reports include HTTP request and response [logs *in-line*](#test-reports), which makes [troubleshooting](https://twitter.com/KarateDSL/status/899671441221623809) and [debugging](https://twitter.com/KarateDSL/status/935029435140489216) easier
* Easily invoke JDK classes, Java libraries, or re-use custom Java code if needed, for [ultimate extensibility](#calling-java)
* Simple plug-in system for [authentication](#http-basic-authentication-example) and HTTP [header management](#configure-headers) that will handle any complex, real-world scenario
* [Cross-browser Web UI automation](karate-core) so that you can test *all* layers of your application with the same framework
* [experimental] [Android and iOS mobile support](https://github.com/karatelabs/karate/issues/743) via [Appium](http://appium.io)
* Visual Validation via the built-in [image comparison](#compare-image) capabilities
* Cross platform [Desktop Automation](karate-robot) that can be [mixed into Web Automation flows](https://twitter.com/ptrthomas/status/1215534821234995200) if needed
* Option to invoke via a [Java API](#java-api), which means that you can easily [mix Karate into Java projects or legacy UI-automation suites](https://stackoverflow.com/q/47795762/143475)
* [Save significant effort](https://twitter.com/ptrthomas/status/986463717465391104) by re-using Karate test-suites as [Gatling performance tests](karate-gatling) that *deeply* assert that server responses are accurate under load
* Gatling integration can hook into [*any* custom Java code](https://github.com/karatelabs/karate/tree/master/karate-gatling#custom) - which means that you can perf-test even non-HTTP protocols such as [gRPC](https://github.com/karatelabs/karate-examples/blob/main/grpc/README.md)
* [API mocks](karate-netty) or test-doubles that even [maintain CRUD 'state'](https://hackernoon.com/api-consumer-contract-tests-and-test-doubles-with-karate-72c30ea25c18) across multiple calls - enabling TDD for micro-services and [Consumer Driven Contracts](https://martinfowler.com/articles/consumerDrivenContracts.html)
* [Async](#async) support that allows you to seamlessly integrate the handling of custom events or listening to message-queues
* Built-in [HTML templating](https://twitter.com/KarateDSL/status/1338892932691070976) so that you can extend your test-reports into readable specifications
* Comprehensive support for different flavors of HTTP calls:
* [SOAP](#soap-action) / XML requests
* HTTPS / [SSL](#configure) - without needing certificates, key-stores or trust-stores
* HTTP [proxy server](#configure) support
* URL-encoded [HTML-form](#form-field) data
* [Multi-part](#multipart-field) file-upload - including `multipart/mixed` and `multipart/related`
* Browser-like [cookie](#cookie) handling
* Full control over HTTP [headers](#header), [path](#path) and query [parameters](#param)
* [Re-try](#retry-until) until condition
* [Websocket](http://www.websocket.org) [support](#async)
## Real World Examples
A set of real-life examples can be found here: [Karate Demos](karate-demo)
## Comparison with REST-assured
For teams familiar with or currently using [REST-assured](http://rest-assured.io), this detailed comparison of [Karate vs REST-assured](http://tinyurl.com/karatera) - can help you evaluate Karate. Do note that if you prefer a pure Java API - Karate has [that covered](#java-api), and with far more capabilities.
## References
* [API Testing with Karate](https://youtu.be/WT4gg7Jutzg) - video + demos by [Peter Thomas](https://twitter.com/ptrthomas) (creator / lead dev of Karate)
* [Introducing Karate: The ULTIMATE Test Automation Tool](https://youtu.be/_kMxvd37auc?si=WOEkvEUH6eWeHZUC) - video by [James Willett](https://www.linkedin.com/in/willettjames/)
* [Karate in the ThoughtWorks Tech Radar](https://twitter.com/KarateDSL/status/1120985060843249664) and featured [a second time](https://twitter.com/KarateDSL/status/1262719979104817152)
* [マイクロサービスにおけるテスト自動化 with Karate](https://speakerdeck.com/takanorig/microservices-testing-automation-with-karate) - (*Microservices Test Automation with Karate*) presentation by [Takanori Suzuki](https://twitter.com/takanorig)
* [Writing API Tests with Karate](https://www.softwaretester.blog/writing-api-tests-with-karate) - book by [Benjamin Bischoff](https://www.softwaretester.blog/about), Packt Publishing, 2023
* [Karate Webinar](https://www.youtube.com/watch?v=cXDIYpT6zck&t=4333s) - Simplificando automação de API com Karate Framework by [Luana Assis](https://www.linkedin.com/in/luanapassis/) from [Base2 Tecnologia](https://www.base2.com.br/)
Karate also has a dedicated "tag", and a very active and supportive community at [Stack Overflow](https://stackoverflow.com/questions/tagged/karate) - where you can get support and ask questions.
You can find a lot more references, tutorials and blog-posts at [karatelabs.io](https://karatelabs.io).
# Getting Started
If you are a Java developer - Karate requires at least [Java](http://www.oracle.com/technetwork/java/javase/downloads/index.html) 17 and then either [Maven](http://maven.apache.org), [Gradle](https://gradle.org), or a Java IDE that embeds either to be installed. Note that Karate works fine on OpenJDK.
If you are new to programming or test-automation, the [official IntelliJ plugin](https://plugins.jetbrains.com/plugin/19232-karate) is recommended.
If you *don't* want to use Java, the [Karate extension for Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=karatelabs.karate) is recommended, and JavaScript, .NET, Ruby and Python programmers will feel right at home.
Both the official Visual Studio Code and IntelliJ plugins support step-through debugging of Karate tests.
## Maven
All you need is available in the [`karate-core`](https://search.maven.org/artifact/com.intuit.karate/karate-core) artifact. You can run tests with this [directly](#parallel-execution), but teams can choose the JUnit variant (shown below) that pulls in JUnit 5 and [slightly improves the in-IDE experience](https://stackoverflow.com/a/65578167/143475).
```xml
io.karatelabs
karate-junit5
1.5.1
test
```
## Gradle
Alternatively for [Gradle](https://gradle.org):
```yml
testImplementation 'io.karatelabs:karate-junit5:1.5.1'
```
Also refer to the wiki for using [Karate with Gradle](https://github.com/karatelabs/karate/wiki/Gradle).
### Karate Core "Fat JAR"
If you mix Karate into a Maven or Gradle project with many other dependendies, you may run into problems because of dependency conflicts. For example a lot of Java projects directly (or indirectly) depend on Netty or Thymeleaf or ANTLR, etc.
If you face issues such as "class not found", just pull in the `karate-core` dependency, and use the `all` [classifier](https://www.baeldung.com/maven-artifact-classifiers) in your `pom.xml` (or `build.gradle`).
For example when using Maven:
```xml
io.karatelabs
karate-core
${karate.version}
all
test
```
Note that for very complicated projects you can consider using a [Maven profile](https://maven.apache.org/guides/introduction/introduction-to-profiles.html) so that testing-related dependencies don't collide with your development-time dependencies. Of course it is an option to have Karate tests in a separate stand-alone maven project and folder, while still being in the same Git repository.
### Quickstart
It may be easier for you to use the Karate Maven archetype to create a skeleton project with one command. You can then skip the next few sections, as the `pom.xml`, recommended directory structure, sample test and [JUnit 5](#junit-5) runners - will be created for you.
> If you are behind a corporate proxy, or especially if your local Maven installation has been configured to point to a repository within your local network, the command below may not work. One workaround is to temporarily disable or rename your Maven [`settings.xml`](https://maven.apache.org/settings.html) file, and try again.
You can replace the values of `com.mycompany` and `myproject` as per your needs.
```
mvn archetype:generate \
-DarchetypeGroupId=io.karatelabs \
-DarchetypeArtifactId=karate-archetype \
-DarchetypeVersion=1.5.1 \
-DgroupId=com.mycompany \
-DartifactId=myproject
```
This will create a folder called `myproject` (or whatever you set the name to).
Note, that when running the command in Windows (PowerShell) you must use quotes around each property value to prevent parsing errors.
## IDE Support
Refer to the wiki - [IDE Support](https://github.com/karatelabs/karate/wiki/IDE-Support).
## Folder Structure
A Karate test script has the file extension `.feature` which is the standard followed by Cucumber. You are free to organize your files using regular Java package conventions.
The Maven tradition is to have non-Java source files in a separate `src/test/resources` folder structure - but we recommend that you keep them side-by-side with your `*.java` files. When you have a large and complex project, you will end up with a few data files (e.g. `*.js`, `*.json`, `*.txt`) as well and it is much more convenient to see the `*.java` and `*.feature` files and all related artifacts in the same place.
This can be easily achieved with the following tweak to your maven `` section.
```xml
src/test/java
**/*.java
...
```
This is very common in the world of Maven users and keep in mind that these are tests and not production code.
Alternatively, if using Gradle then add the following `sourceSets` definition
```yml
sourceSets {
test {
resources {
srcDir file('src/test/java')
exclude '**/*.java'
}
}
}
```
With the above in place, you don't have to keep switching between your `src/test/java` and `src/test/resources` folders, you can have all your test-code and artifacts under `src/test/java` and everything will work as expected.
Once you get used to this, you may even start wondering why projects need a `src/test/resources` folder at all !
## Naming Conventions
Since these are tests and not production Java code, you don't need to be bound by the `com.mycompany.foo.bar` convention and the un-necessary explosion of sub-folders that ensues. We suggest that you have a folder hierarchy only one or two levels deep - where the folder names clearly identify which 'resource', 'entity' or API is the web-service under test.
For example:
```
src/test/java
|
+-- karate-config.js
+-- logback-test.xml
+-- some-reusable.feature
+-- some-classpath-function.js
+-- some-classpath-payload.json
|
\-- animals
|
+-- AnimalsTest.java
|
+-- cats
| |
| +-- cats-post.feature
| +-- cats-get.feature
| +-- cat.json
| \-- CatsRunner.java
|
\-- dogs
|
+-- dog-crud.feature
+-- dog.json
+-- some-helper-function.js
\-- DogsRunner.java
```
Assuming you use JUnit, there are some good reasons for the recommended (best practice) naming convention and choice of file-placement shown above:
* Not using the `*Test.java` convention for the JUnit classes (e.g. `CatsRunner.java`) in the `cats` and `dogs` folder ensures that these tests will **not** be picked up when invoking `mvn test` (for the whole project) from the [command line](#command-line). But you can still invoke these tests from the IDE, which is convenient when in development mode.
* `AnimalsTest.java` (the only file that follows the `*Test.java` naming convention) acts as the 'test suite' for the entire project. By default, Karate will load all `*.feature` files from sub-directories as well. But since `some-reusable.feature` is _above_ `AnimalsTest.java` in the folder hierarchy, it will **not** be picked-up. Which is exactly what we want, because `some-reusable.feature` is designed to be [called](#calling-other-feature-files) only from one of the other test scripts (perhaps with some parameters being passed). You can also use [tags](#tags) to skip files.
* `some-classpath-function.js` and `some-classpath-payload.json` are in the 'root' of the Java ['classpath'](#classpath) which means they can be easily [read](#reading-files) (and re-used) from any test-script by using the `classpath:` prefix, for e.g: `read('classpath:some-classpath-function.js')`. Relative paths will also work.
For details on what actually goes into a script or `*.feature` file, refer to the [syntax guide](#syntax-guide).
## JUnit 5
Karate supports [JUnit 5](https://junit.org/junit5) and the advantage is that you can have multiple methods in a test-class. Only 1 `import` is needed, and instead of a class-level annotation, you use a nice [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) and [fluent-api](https://en.wikipedia.org/wiki/Fluent_interface) to express which tests and tags you want to use.
Note that the Java class does not need to be `public` and even the test methods do not need to be `public` - so tests end up being very concise.
> Karate will traverse sub-directories and look for `*.feature` files. For example if you have the JUnit class in the `com.mycompany` package, `*.feature` files in `com.mycompany.foo` and `com.mycompany.bar` will also be run. This is one reason why you may want to prefer a 'flat' directory structure as [explained above](#naming-conventions).
Here is an [example](karate-junit5/src/test/java/karate/SampleTest.java):
```java
package karate;
import com.intuit.karate.junit5.Karate;
class SampleTest {
@Karate.Test
Karate testSample() {
return Karate.run("sample").relativeTo(getClass());
}
@Karate.Test
Karate testTags() {
return Karate.run("tags").tags("@second").relativeTo(getClass());
}
@Karate.Test
Karate testSystemProperty() {
return Karate.run("classpath:karate/tags.feature")
.tags("@second")
.karateEnv("e2e")
.systemProperty("foo", "bar");
}
}
```
Note that more "builder" methods are available from the [`Runner.Builder`](#parallel-execution) class such as `reportDir()` etc.
You should be able to right-click and run a single method using your IDE - which should be sufficient when you are in development mode. But to be able to run JUnit 5 tests from the command-line, you need to ensure that the latest version of the [maven-surefire-plugin](https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html) is present in your project `pom.xml` (within the `/` section):
```xml
org.apache.maven.plugins
maven-surefire-plugin
2.22.2
```
To run a single test method, for example the `testTags()` in the example above, you can do this:
```
mvn test -Dtest=SampleTest#testTags
```
Also look at how to run tests via the [command-line](#command-line) and the [parallel runner](#parallel-execution).
### JUnit HTML report
When you use the JUnit runner - after the execution of each feature, an HTML report is output to the `target/karate-reports` folder and the full path will be printed to the console (see [video](https://twitter.com/KarateDSL/status/935029435140489216)).
```
html report: (paste into browser to view)
-----------------------------------------
file:///projects/myproject/target/karate-reports/mypackage.myfeature.html
```
You can easily select (double-click), copy and paste this `file:` URL into your browser address bar. This report is useful for troubleshooting and debugging a test because all requests and responses are shown in-line with the steps, along with error messages and the output of [`print`](#print) statements. Just re-fresh your browser window if you re-run the test.
### Dry Run
This will give you the usual HTML report showing what features will be run, including all steps shown (including comments) so that it can be reviewed. Of course the actual time-durations, and logs will be missing, and *everything* will pass.
The “dry run” report is useful to review the tag "coverage" of what will be run. For example you can get a nice feature “coverage” report, provided you have a rich set of [tags](#tags). e.g. `@smoke @module=one @module=two` etc.
The [`Runner.Builder`](#parallel-execution) API has a `dryRun()` method to switch this on. Note that this mode can be also triggered via the command-line by adding `-D` or `--dryrun` to the [`karate.options`](#karateoptions).
## Command Line
If you are using Karate via the [VS Code Plugin](https://marketplace.visualstudio.com/items?itemName=karatelabs.karate) or the stand-alone JAR, refer to the [CLI usage guide](https://github.com/karatelabs/karate/wiki/Get-Started:-Other-Runtime-Options).
### Command Line - Maven
Normally in dev mode, you will use your IDE to run a `*.feature` file directly or via the companion 'runner' JUnit Java class. When you have a 'runner' class in place, it would be possible to run it from the command-line as well.
Note that the `mvn test` command only runs test classes that follow the `*Test.java` [naming convention](#naming-conventions) by default. But you can choose a single test to run like this:
```
mvn test -Dtest=CatsRunner
```
### `karate.options`
When your Java test "runner" is linked to multiple feature files, which will be the case when you use the recommended [parallel runner](#parallel-execution), you can narrow down your scope to a single feature, scenario or directory via the command-line, useful in dev-mode. Note how even [tags](#tags) to exclude (or include) can be specified:
> Note that any `Feature` or `Scenario` with the [special `@ignore` tag](#special-tags) will be skipped by default.
```
mvn test "-Dkarate.options=--tags ~@skipme classpath:demo/cats/cats.feature" -Dtest=DemoTestParallel
```
Multiple feature files (or paths) can be specified, de-limited by the space character. They should be at the end of the `karate.options`. To run only a single scenario, append the line number on which the scenario is defined, de-limited by `:`.
```
mvn test "-Dkarate.options=PathToFeatureFiles/order.feature:12" -Dtest=DemoTestParallel
```
Since paths are expected at the end of the command-line options - if you want to only over-ride tags, use the `=` sign to make argument values clear. For example:
```
mvn test -Dkarate.options='-t=@dev -t=@src' -Dtest=ExamplesTest
```
### Command Line - Gradle
For Gradle, you must extend the test task to allow the `karate.options` to be passed to the runtime (otherwise they get consumed by Gradle itself). To do that, add the following:
```yml
test {
// pull karate options into the runtime
systemProperty "karate.options", System.properties.getProperty("karate.options")
// pull karate env into the runtime
systemProperty "karate.env", System.properties.getProperty("karate.env")
// ensure tests are always run
outputs.upToDateWhen { false }
}
```
And then the above command in Gradle would look like:
```
./gradlew test --tests *CatsRunner
```
or
```
./gradlew test -Dtest.single=CatsRunner
```
### Test Suites
> The recommended way to define and run test-suites and reporting in Karate is to use the [parallel runner](#parallel-execution), described in the next section. The approach in this section is more suited for troubleshooting in dev-mode, using your IDE.
One way to define 'test-suites' in Karate is to have a JUnit class at a level 'above' (in terms of folder hierarchy) all the `*.feature` files in your project. So if you take the previous [folder structure example](#naming-conventions), you can do this on the command-line:
```
mvn test "-Dkarate.options=--tags ~@skipme" -Dtest=AnimalsTest
```
Here, `AnimalsTest` is the name of the Java class we designated to run the multiple `*.feature` files that make up your test-suite. There is a neat way to [tag your tests](#tags) and the above example demonstrates how to run all tests _except_ the ones tagged `@skipme`.
Note that the special, built-in tag `@ignore` will *always* be skipped by default, and you don't need to specify `~@ignore` anywhere.
You can 'lock down' the fact that you only want to execute the single JUnit class that functions as a test-suite - by using the following [maven-surefire-plugin configuration](http://maven.apache.org/surefire/maven-surefire-plugin/examples/inclusion-exclusion.html):
```xml
org.apache.maven.plugins
maven-surefire-plugin
${maven.surefire.version}
animals/AnimalsTest.java
--tags @smoke
```
Note how the [`karate.options`](#karateoptions) can be specified using the `` configuration.
For Gradle, you simply specify the test which is to be `include`-d:
```yml
test {
include 'animals/AnimalsTest.java'
// pull karate options into the runtime
systemProperty "karate.options", System.properties.getProperty("karate.options")
// pull karate env into the runtime
systemProperty "karate.env", System.properties.getProperty("karate.env")
// ensure tests are always run
outputs.upToDateWhen { false }
}
```
The big drawback of the approach above is that you cannot run tests in parallel. The recommended approach for Karate reporting in a Continuous Integration set-up is described in the next section which can generate the [JUnit XML](https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Plugin) format that most CI tools can consume. The [Cucumber JSON format](https://cucumber.io/docs/cucumber/reporting) can be also emitted, which gives you plenty of options for generating pretty reports using third-party maven plugins.
And most importantly - you can run tests in parallel without having to depend on third-party hacks that introduce code-generation and config 'bloat' into your `pom.xml` or `build.gradle`.
## Parallel Execution
> Please note that some [user analytics is tracked](karate-core/src/test/resources/analytics.md) only when you view the built-in Karate HTML report.
Karate can run tests in parallel, and dramatically cut down execution time. This is a 'core' feature and does not depend on JUnit, Maven or Gradle.
> For those running Karate in non-Java projects via the command-line, note that you can set the number of threads via `--threads` or `-T` as explained [here](https://karatelabs.github.io/karate/karate-netty/#parallel-execution).
* You can easily "choose" features and tags to run and compose test-suites in a very flexible manner.
* You can use the returned `Results` object to check if any scenarios failed, and to even summarize the errors
* [JUnit XML](https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Plugin) reports can be generated in the "`reportDir`" path you specify, and you can easily configure your CI to look for these files after a build (for e.g. in `**/*.xml` or `**/karate-reports/*.xml`). Note that you have to call the `outputJunitXml(true)` method on the `Runner` "builder".
* [Cucumber JSON reports](https://cucumber.io/docs/cucumber/reporting) can be generated, except that the extension will be `.json` instead of `.xml`. Note that you have to call the `outputCucumberJson(true)` method on the `Runner` "builder".
* HTML reports can be disabled by calling [`outputHtmlReport(false)`](https://github.com/karatelabs/karate/blob/v1.2.1.RC1/karate-core/src/main/java/com/karatelabs/karate/Runner.java#L515).
* The `Runner.path()` "builder" method in `karate-core` is how you refer to the package you want to execute, and all feature files within sub-directories will be picked up
* `Runner.path()` takes multiple string parameters, so you can refer to multiple packages or even individual `*.feature` files and easily "compose" a test-suite
* e.g. `Runner.path("classpath:animals", "classpath:some/other/package.feature")`
* To [choose tags](#tags), call the `tags()` API, note that by default, any `*.feature` file tagged with the special (built-in) tag: `@ignore` will be skipped. You can also specify tags on the [command-line](#test-suites). The `tags()` method also takes multiple arguments, for e.g.
* this is an "AND" operation: `tags("@customer", "@smoke")`
* and this is an "OR" operation: `tags("@customer,@smoke")`
* There is an optional `reportDir()` method if you want to customize the directory to which the [HTML, XML and JSON](#parallel-execution) files will be output, it defaults to `target/karate-reports`
* If you want to dynamically and programmatically determine the tags and features to be included - the API also accepts `List` as the `path()` and `tags()` methods arguments
* `parallel()` *has* to be the last method called, and you pass the number of parallel threads needed. It returns a `Results` object that has all the information you need - such as the number of passed or failed tests.
### JUnit 5 Parallel Execution
The example below assumes that [JUnit 5](https://junit.org/junit5) is available on the classpath, and uses the `@Test` annotation and the `assertEquals()` method.
> But if you really want, you could use the `Runner` and `Results` API directly in *any* Java class, and even a "main" method.
Use the [`karate-template`](https://github.com/karatelabs/karate-template) project if you want to get an [example](https://github.com/karatelabs/karate-template/blob/main/src/test/java/examples/ExamplesTest.java) as part of a working, "skeleton" project.
```java
import com.intuit.karate.Results;
import com.intuit.karate.Runner;
import static org.junit.jupiter.api.Assertions.*;
import org.junit.jupiter.api.Test;
class TestParallel {
@Test
void testParallel() {
Results results = Runner.path("classpath:animals").tags("~@skipme").parallel(5);
assertEquals(0, results.getFailCount(), results.getErrorMessages());
}
}
```
### Parallel Stats
For convenience, some stats are logged to the console when execution completes, which should look something like this:
```
======================================================
elapsed: 2.35 | threads: 5 | thread time: 4.98
features: 54 | ignored: 25 | efficiency: 0.42
scenarios: 145 | passed: 145 | failed: 0
======================================================
```
The parallel runner will always run `Feature`-s in parallel. Karate will also run `Scenario`-s in parallel by default. So if you have a `Feature` with multiple `Scenario`-s in it - they will execute in parallel, and even each `Examples` row in a `Scenario Outline` will do so !
A `karate-timeline.html` file will also be saved to the report output directory mentioned above (`target/karate-reports` by default) - which is useful for visually verifying or troubleshooting the effectiveness of the test-run ([see video](https://twitter.com/KarateDSL/status/1049321708241317888)).
### `@parallel=false`
In rare cases you may want to suppress the default of `Scenario`-s executing in parallel and the special [`tag`](#tags) `@parallel=false` can be used. If you place it above the [`Feature`](#script-structure) keyword, it will apply to all `Scenario`-s. And if you just want one or two `Scenario`-s to NOT run in parallel, you can place this tag above only *those* `Scenario`-s. See [example](karate-demo/src/test/java/demo/encoding/encoding.feature).
Note that forcing `Scenario`-s to run in a particular sequence [is an anti-pattern](https://stackoverflow.com/a/46080568/143475), and should be avoided as far as possible.
## Test Reports
As mentioned above, most CI tools would be able to process the JUnit XML output of the [parallel runner](#parallel-execution) and determine the status of the build as well as generate reports.
The [Karate Demo](karate-demo) has a working example of the recommended parallel-runner set up. It also [details how](karate-demo#example-report) a third-party library can be easily used to generate some very nice-looking reports, from the JSON output of the parallel runner.
For example, here below is an actual report generated by the [cucumber-reporting](https://github.com/damianszczepanik/cucumber-reporting) open-source library.
Another example for a popular Maven reporting plugin that is compatible with Karate JSON is [Cluecumber](https://github.com/trivago/cluecumber-report-plugin).
The demo also features [code-coverage using Jacoco](karate-demo#code-coverage-using-jacoco), and some tips for even non-Java back-ends. Some third-party report-server solutions integrate with Karate such as [ReportPortal.io](https://github.com/reportportal/agent-java-karate).
## Logging
> This is optional, and Karate will work without the logging config in place, but the default console logging may be too verbose for your needs.
Karate uses [LOGBack](http://logback.qos.ch) which looks for a file called `logback-test.xml` on the '[classpath](#classpath)'.
> In rare cases, e.g. if you are using Karate to create a Java application, [LOGBack will look for `logback.xml`](http://logback.qos.ch/manual/configuration.html)
Here is a sample `logback-test.xml` for you to get started.
```xml
%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n
target/karate.log
%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n
```
You can change the `com.intuit.karate` logger level to `INFO` to reduce the amount of logging. When the level is `DEBUG` the entire request and response payloads are logged. If you use the above config, logs will be captured in `target/karate.log`.
If you want to keep the level as `DEBUG` ([for HTML reports](#test-reports)) but suppress logging to the console, you can comment out the `STDOUT` "root" `appender-ref`:
```xml
```
Or another option is to use a [`ThresholdFilter`](http://logback.qos.ch/manual/filters.html#thresholdFilter), so you still see critical logs on the console:
```xml
WARN
%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n
```
If you want to exclude the logs from your CI/CD pipeline but keep them in the execution of your users in their locals you can configure your logback using [Janino](http://logback.qos.ch/manual/configuration.html#conditional). In such cases it might be desirable to have your tests using `karate.logger.debug('your additional info')` instead of the `print` keyword so you can keep logs in your pipeline in INFO.
For suppressing sensitive information such as secrets and passwords from the log and reports, see [Log Masking](#log-masking) and [Report Verbosity](#report-verbosity).
# Configuration
> You can skip this section and jump straight to the [Syntax Guide](#syntax-guide) if you are in a hurry to get started with Karate. Things will work even if the `karate-config.js` file is not present.
## Classpath
The 'classpath' is a Java concept and is where some configuration files such as the one for [logging](#logging) are expected to be by default. If you use the Maven `` tweak [described earlier](#folder-structure) (recommended), the 'root' of the classpath will be in the `src/test/java` folder, or else would be `src/test/resources`.
## `karate-config.js`
The only 'rule' is that on start-up Karate expects a file called `karate-config.js` to exist on the 'classpath' and contain a [JavaScript function](#javascript-functions). The function is expected to return a JSON object and all keys and values in that JSON object will be made available as script variables.
And that's all there is to Karate configuration ! You can easily get the value of the [current 'environment' or 'profile'](#switching-the-environment), and then set up 'global' variables using some simple JavaScript. Here is an example:
```javascript
function fn() {
var env = karate.env; // get java system property 'karate.env'
karate.log('karate.env system property was:', env);
if (!env) {
env = 'dev'; // a custom 'intelligent' default
}
var config = { // base config JSON
appId: 'my.app.id',
appSecret: 'my.secret',
someUrlBase: 'https://some-host.com/v1/auth/',
anotherUrlBase: 'https://another-host.com/v1/'
};
if (env == 'stage') {
// over-ride only those that need to be
config.someUrlBase = 'https://stage-host/v1/auth';
} else if (env == 'e2e') {
config.someUrlBase = 'https://e2e-host/v1/auth';
}
// don't waste time waiting for a connection or if servers don't respond within 5 seconds
karate.configure('connectTimeout', 5000);
karate.configure('readTimeout', 5000);
return config;
}
```
> Here above, you see the [`karate.log()`](#karate-log), [`karate.env`](#karate-env) and [`karate.configure()`](#karate-configure) "helpers" being used. Note that the `karate-config.js` is re-processed for *every* `Scenario` and in rare cases, you may want to initialize (e.g. auth tokens) only once for all of your tests. This can be achieved using [`karate.callSingle()`](#karatecallsingle).
A common requirement is to pass dynamic parameter values via the command line, and you can use the [`karate.properties['some.name']`](#karate-properties) syntax for getting a system property passed via JVM options in the form `-Dsome.name=foo`. Refer to the section on [dynamic port numbers](#dynamic-port-numbers) for an example.
> You can even retrieve operating-system environment variables via [Java interop](#calling-java) as follows: `var systemPath = java.lang.System.getenv('PATH');`
This decision to use JavaScript for config is influenced by years of experience with the set-up of complicated test-suites and fighting with [Maven profiles](http://maven.apache.org/guides/introduction/introduction-to-profiles.html), [Maven resource-filtering](https://maven.apache.org/plugins/maven-resources-plugin/examples/filter.html) and the XML-soup that somehow gets summoned by the [Maven AntRun plugin](http://maven.apache.org/plugins/maven-antrun-plugin/usage.html).
Karate's approach frees you from Maven, is far more expressive, allows you to eyeball all environments in one place, and is still a plain-text file. If you want, you could even create [nested chunks of JSON that 'name-space' your config variables](https://stackoverflow.com/a/49693808/143475).
> One way to appreciate Karate's approach is to think over what it takes to add a new environment-dependent variable (e.g. a password) into a test. In typical frameworks it could mean changing multiple properties files, maven profiles and placeholders, and maybe even threading the value via a dependency-injection framework - before you can even access the value within your test.
This approach is indeed slightly more complicated than traditional `*.properties` files - but you _need_ this complexity. Keep in mind that these are tests (not production code) and this config is going to be maintained more by the dev or QE team instead of the 'ops' or operations team.
And there is no more worrying about Maven profiles and whether the 'right' `*.properties` file has been copied to the proper place.
## Switching the Environment
There is only one thing you need to do to switch the environment - which is to set a Java system property.
> By default, the value of `karate.env` when you access it within [`karate-config.js`](#karate-configjs) - would be `null`.
The recipe for doing this when running Maven from the command line is:
```
mvn test -DargLine="-Dkarate.env=e2e"
```
Or in Gradle:
```
./gradlew test -Dkarate.env=e2e
```
You can refer to the documentation of the
[Maven Surefire Plugin](http://maven.apache.org/plugins-archives/maven-surefire-plugin-2.12.4/examples/system-properties.html) for alternate ways of achieving this, but the `argLine` approach is the simplest and should be more than sufficient for your Continuous Integration or test-automation needs.
Here's a reminder that running any [single JUnit test via Maven](https://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html) can be done by:
```
mvn test -Dtest=CatsRunner
```
Where `CatsRunner` is the JUnit class name (in any package) you wish to run.
Karate is flexible, you can easily over-write config variables within the Java or JUnit "runner" - which is very convenient when in dev-mode or rapid-prototyping.
```java
System.setProperty("karate.env", "pre-prod");
```
But the recommended way is to use the `karateEnv(name, value)` or `systemProperty(name, value)` API on the [parallel-runner](#parallel-execution).
For advanced users, note that [tags](#tags) and the `karate.env` environment-switch can be "linked" using the special [environment tags](#environment-tags).
## Environment Specific Config
When your project gets complex, you can have separate `karate-config-.js` files that will be processed for that specific value of [`karate.env`](#switching-the-environment). This is especially useful when you want to maintain passwords, secrets or even URL-s specific for your local dev environment.
> Make sure you configure your source code management system (e.g. Git) to ignore `karate-config-*.js` if needed.
> There should always be `karate-config.js` in the "root" folder, even if you don't have any "common" config. In such cases, the function can do nothing or return an empty JSON. [Learn more](https://github.com/karatelabs/karate/issues/1588).
Here are the rules Karate uses on bootstrap (before every `Scenario` or `Examples` row in a `Scenario Outline`):
* if the system-property `karate.config.dir` was set, Karate will look in this folder for `karate-config.js` - and if found, will process it
* else if `karate-config.js` was not found in the above location (or `karate.config.dir` was not set), [`classpath:karate-config.js`](#configuration) would be processed (this is the default / common case)
* if the `karate.env` system property was set
* if `karate.config.dir` was set, Karate will also look for `file:/karate-config-.js`
* else (if the `karate.config.dir` was *not* set), Karate will look for `classpath:karate-config-.js`
* if the over-ride `karate-config-.js` exists, it will be processed, and the configuration (JSON entries) returned by this function will over-ride any set by `karate-config.js`
Refer to the [karate demo](karate-demo) for an [example](karate-demo/src/test/java/karate-config-contract.js).
### `karate-base.js`
Advanced users who build frameworks on top of Karate have the option to supply a `karate-base.js` file that Karate will look for on the [`classpath:`](#classpath). This is useful when you ship a JAR file containing re-usable features and JavaScript / Java code and want to 'default' a few variables that teams can 'inherit' from. So an additional rule in the above flow of 'rules' (before the *first* step) is as follows:
* if `classpath:karate-base.js` exists - Karate will process this as a [configuration source](#configuration) before anything else
# Syntax Guide
## Script Structure
Karate scripts are technically in '[Gherkin](https://docs.cucumber.io/gherkin/reference/)' format - but all you need to grok as someone who needs to test web-services are the three sections: `Feature`, `Background` and `Scenario`. There can be multiple Scenario-s in a `*.feature` file, and at least one should be present. The `Background` is optional.
> Variables set using [`def`](#def) in the `Background` will be re-set before *every* `Scenario`. If you are looking for a way to do something only **once** per `Feature`, take a look at [`callonce`](#callonce). On the other hand, if you are expecting a variable in the `Background` to be modified by one `Scenario` so that later ones can see the updated value - that is *not* how you should think of them, and you should combine your 'flow' into one scenario. Keep in mind that you should be able to comment-out a `Scenario` or skip some via [`tags`](#tags) without impacting any others. Note that the [parallel runner](#parallel-execution) will run `Scenario`-s in parallel, which means they can run in *any* order. If you are looking for ways to do something only *once* per feature or across *all* your tests, see [Hooks](#hooks).
Lines that start with a `#` are comments.
```cucumber
Feature: brief description of what is being tested
more lines of description if needed.
Background:
# this section is optional !
# steps here are executed before each Scenario in this file
# variables defined here will be 'global' to all scenarios
# and will be re-initialized before every scenario
Scenario: brief description of this scenario
# steps for this scenario
Scenario: a different scenario
# steps for this other scenario
```
> There is also a variant of `Scenario` called `Scenario Outline` along with `Examples`, useful for [data-driven tests](#data-driven-tests).
### Given-When-Then
The business of web-services testing requires access to low-level aspects such as HTTP headers, URL-paths, query-parameters, complex JSON or XML payloads and response-codes. And Karate gives you control over these aspects with the small set of keywords focused on HTTP such as [`url`](#url), [`path`](#path), [`param`](#param), etc.
Karate does not attempt to have tests be in "natural language" like how Cucumber tests are [traditionally expected to be](https://docs.cucumber.io/gherkin/reference/). That said, the syntax is very concise, and the convention of every step having to start with either `Given`, `And`, `When` or `Then`, makes things very readable. You end up with a decent approximation of BDD even though web-services by nature are "headless", without a UI, and not really human-friendly.
#### Cucumber vs Karate
> Karate was based on Cucumber-JVM until version 0.8.0 but the parser and engine were [re-written from scratch](https://github.com/karatelabs/karate/issues/444) in 0.9.0 onwards. So we use the same [Gherkin](https://docs.cucumber.io/gherkin/) syntax - but the similarity ends there.
If you are familiar with Cucumber (JVM), you may be wondering if you need to write [step-definitions](https://docs.cucumber.io/gherkin/step-organization/). The answer is **no**.
Karate's approach is that all the step-definitions you need in order to work with HTTP, JSON and XML have been already implemented. And since you can easily extend Karate [using JavaScript](#call), there is no need to compile Java code any more.
The following table summarizes some key differences between Cucumber and Karate.
:white_small_square: | Cucumber | Karate
-------------------- | -------- | ------
**Step Definitions Built-In** | **No**. You need to keep implementing them as your functionality grows. [This can get very tedious](https://thepracticaldeveloper.com/2017/08/03/microservices-end-to-end-tests-with-cucumber-and-spring-boot/), especially since for [dependency-injection](https://docs.cucumber.io/cucumber/state/#dependency-injection), you are [on your own](http://angiejones.tech/rest-assured-with-cucumber-using-bdd-for-web-services-automation?refreshed=y#comment-40). | :white_check_mark: **Yes**. No extra Java code needed.
**Single Layer of Code To Maintain** | **No**. There are 2 Layers. The [Gherkin](https://docs.cucumber.io/gherkin/reference/) spec or `*.feature` files make up one layer, and you will also have the corresponding Java step-definitions. | :white_check_mark: **Yes**. Only 1 layer of Karate-script (based on Gherkin).
**Readable Specification** | **Yes**. Cucumber will read like natural language _if_ you implement the step-definitions right. | :x: **No**. Although Karate is simple, and a [true DSL](https://ayende.com/blog/2984/dsl-vs-fluent-interface-compare-contrast), it is ultimately a [mini-programming language](https://hackernoon.com/yes-karate-is-not-true-bdd-698bf4a9be39). But it is [perfect for testing web-services](https://stackoverflow.com/a/47799207/143475) at the level of HTTP requests and responses.
**Re-Use Feature Files** | **No**. Cucumber does not support being able to call (and thus re-use) other `*.feature` files from a test-script. | :white_check_mark: [**Yes**](#calling-other-feature-files).
**Dynamic Data-Driven Testing** | **No**. Cucumber's [`Scenario Outline`](#the-cucumber-way) expects the `Examples` to contain a fixed set of rows. | :white_check_mark: **Yes**. Karate's support for calling other `*.feature` files allows you to use a [JSON array as the data-source](#data-driven-features) and you can [use JSON](https://twitter.com/KarateDSL/status/1051433711814627329) or [even CSV](#csv-files) directly in a data-driven `Scenario Outline`.
**Parallel Execution** | **No**. There are some challenges (especially with reporting) and you can find various discussions and third-party projects on the web that attempt to close this gap | :white_check_mark: [**Yes**](#parallel-execution). Karate runs even `Scenario`-s in parallel, not just `Feature`-s.
**Run 'Set-Up' Routines Only Once** | **No**. Cucumber has a limitation where `Background` steps are re-run for every `Scenario` and worse - even for every `Examples` row within a `Scenario Outline`. This has been a [highly-requested open issue](https://github.com/cucumber/cucumber-jvm/issues/515) for a *long* time. | :white_check_mark: [**Yes**](#hooks).
**Embedded JavaScript Engine** | **No**. And you have to roll your own approach to environment-specific configuration and worry about [dependency-injection](https://docs.cucumber.io/cucumber/state/#dependency-injection). | :white_check_mark: **Yes**. Easily define all environments in a [single file](#configuration) and share variables across all scenarios. Full script-ability via [JS](#calling-javascript-functions) or [Java interop](#java-interop).
One nice thing about the design of the Gherkin syntax is that script-steps are treated the same no matter whether they start with the keyword `Given`, `And`, `When` or `Then`. What this means is that you are free to use whatever makes sense for you. You could even have all the steps start with `When` and Karate won't care.
In fact Gherkin supports the [catch-all symbol '`*`'](https://www.relishapp.com/cucumber/cucumber/docs/gherkin/using-star-notation-instead-of-given-when-then) - instead of forcing you to use `Given`, `When` or `Then`. This is perfect for those cases where it really doesn't make sense - for example the [`Background`](#script-structure) section or when you use the [`def`](#def) or [`set`](#set) syntax. When eyeballing a test-script, think of the `*` as a 'bullet-point'.
You can read more about the Given-When-Then convention at the [Cucumber reference documentation](https://docs.cucumber.io/gherkin/reference/). Since Karate uses Gherkin, you can also employ [data-driven](#data-driven-tests) techniques such as expressing data-tables in test scripts. Another good thing that Karate inherits is the nice IDE support for Cucumber that [IntelliJ](https://www.jetbrains.com/idea/help/cucumber.html) and [Eclipse](https://cucumber.io/cucumber-eclipse/) have. So you can do things like right-click and run a `*.feature` file (or scenario) without needing to use a JUnit runner.
For a detailed discussion on BDD and how Karate relates to Cucumber, please refer to this blog-post: [Yes, Karate is not *true* BDD](https://medium.com/@ptrthomas/yes-karate-is-not-true-bdd-698bf4a9be39). It is the opinion of the author of Karate that *true* BDD is un-necessary over-kill for API testing, and this is explained more in [this answer](https://stackoverflow.com/a/47799207/143475) on [Stack Overflow](https://stackoverflow.com/questions/tagged/karate).
With the formalities out of the way, let's dive straight into the syntax.
# Setting and Using Variables
## `def`
### Set a named variable
```cucumber
# assigning a string value:
Given def myVar = 'world'
# using a variable
Then print myVar
# assigning a number (you can use '*' instead of Given / When / Then)
* def myNum = 5
```
Note that `def` will over-write any variable that was using the same name earlier. Keep in mind that the start-up [configuration routine](#configuration) could have already initialized some variables before the script even started. For details of scope and visibility of variables, see [Script Structure](#script-structure).
> Note that `url` and `request` are not allowed as variable names. This is just to reduce confusion for users new to Karate who tend to do `* def request = {}` and expect the [`request`](#request) body or similarly, the [`url`](#url) to be set.
The examples above are simple, but a variety of expression 'shapes' are supported on the right hand side of the `=` symbol. The section on [Karate Expressions](#karate-expressions) goes into the details.
## `assert`
### Assert if an expression evaluates to `true`
Once defined, you can refer to a variable by name. Expressions are evaluated using the embedded JavaScript engine. The assert keyword can be used to assert that an expression returns a boolean value.
```cucumber
Given def color = 'red '
And def num = 5
Then assert color + num == 'red 5'
```
Everything to the right of the `assert` keyword will be evaluated as a single expression.
Something worth mentioning here is that you would hardly need to use `assert` in your test scripts. Instead you would typically use the [`match`](#match) keyword, that is designed for performing powerful assertions against JSON and XML response payloads.
## `print`
### Log to the console
You can use `print` to log variables to the console in the middle of a script. For convenience, you can have multiple expressions separated by commas, so this is the recommended pattern:
```cucumber
* print 'the value of a is:', a
```
Similar to [`assert`](#assert), the expressions on the right-hand-side of a `print` have to be valid JavaScript. JsonPath and [Karate expressions](#karate-expressions) are not supported.
If you use commas (instead of concatenating strings using `+`), Karate will 'pretty-print' variables, which is what you typically want when dealing with [JSON or XML](#native-data-types).
```cucumber
* def myJson = { foo: 'bar', baz: [1, 2, 3] }
* print 'the value of myJson is:', myJson
```
Which results in the following output:
```
20:29:11.290 [main] INFO com.intuit.karate - [print] the value of myJson is: {
"foo": "bar",
"baz": [
1,
2,
3
]
}
```
> Since XML is represented internally as a JSON-like or map-like object, if you perform string concatenation when printing, you will *not* see XML - which can be confusing at first. Use the comma-delimited form (see above) or the JS helper (see below).
The built-in [`karate` object](#the-karate-object) is explained in detail later, but for now, note that this is also injected into `print` (and even `assert`) statements, and it has a helpful [`pretty`](#karate-pretty) method, that takes a JSON argument and a [`prettyXml`](#karate-prettyxml) method that deals with XML. So you could have also done something like:
```cucumber
* print 'the value of myJson is:\n' + karate.pretty(myJson)
```
Also refer to the [`configure`](#configure) keyword on how to switch on pretty-printing of all HTTP requests and responses.
# 'Native' data types
Native data types mean that you can insert them into a script without having to worry about enclosing them in strings and then having to 'escape' double-quotes all over the place. They seamlessly fit 'in-line' within your test script.
## JSON
Note that the parser is 'lenient' so that you don't have to enclose all keys in double-quotes.
```cucumber
* def cat = { name: 'Billie', scores: [2, 5] }
* assert cat.scores[1] == 5
```
> Some characters such as the hyphen `-` are not permitted in 'lenient' JSON keys (because they are interpreted by the JS engine as a 'minus sign'). In such cases, you *have* to use string quotes: `{ 'Content-Type': 'application/json' }`
When asserting for expected values in JSON or XML, always prefer using [`match`](#match) instead of [`assert`](#assert). Match failure messages are much more descriptive and useful, and you get the power of [embedded expressions](#embedded-expressions) and [fuzzy matching](#fuzzy-matching).
```cucumber
* def cats = [{ name: 'Billie' }, { name: 'Bob' }]
* match cats[1] == { name: 'Bob' }
```
Karate's native support for JSON means that you can assign parts of a JSON instance into another variable, which is useful when dealing with complex [`response`](#response) payloads.
```cucumber
* def first = cats[0]
* match first == { name: 'Billie' }
```
For manipulating or updating JSON (or XML) using path expressions, refer to the [`set`](#set) keyword.
## XML
```cucumber
Given def cat = Billie25
# sadly, xpath list indexes start from 1
Then match cat/cat/scores/score[2] == '5'
# but karate allows you to traverse xml like json !!
Then match cat.cat.scores.score[1] == 5
```
### Embedded Expressions
Karate has a very useful payload 'templating' approach. Variables can be referred to within JSON, for example:
```cucumber
Given def user = { name: 'john', age: 21 }
And def lang = 'en'
When def session = { name: '#(user.name)', locale: '#(lang)', sessionUser: '#(user)' }
```
So the rule is - if a string value within a JSON (or XML) object declaration is enclosed between `#(` and `)` - it will be evaluated as a JavaScript expression. And any variables which are alive in the context can be used in this expression. Here's how it works for XML:
```cucumber
Given def user = john
And def lang = 'en'
When def session = #(lang)#(user)
```
This comes in useful in some cases - and avoids needing to use the [`set`](#set) keyword or [JavaScript functions](#javascript-functions) to manipulate JSON. So you get the best of both worlds: the elegance of JSON to express complex nested data - while at the same time being able to dynamically plug values (that could even be other JSON or XML 'trees') into a 'template'.
Note that embedded expressions will be evaluated even when you [`read()` from a JSON or XML file](#reading-files). This is super-useful for re-use and data-driven tests.
A few special built-in variables such as `$` (which is a [reference to the JSON root](#referring-to-self)) - can be mixed into JSON embedded expressions.
A [special case](#remove-if-null) of embedded expressions can remove a JSON key (or XML element / attribute) if the expression evaluates to `null`.
#### Rules for Embedded Expressions
* They work only within JSON or XML
* and when on the Right Hand Side of a
* [`def`](#def)
* [`match`](#match)
* [`configure`](#configure)
* and when you [`read()`](#reading-files) a JSON or XML file
* the expression *has* to start with `#(` and end with `)`
Because of the last rule above, note that string-concatenation may not work quite the way you expect:
```cucumber
# wrong !
* def foo = { bar: 'hello #(name)' }
# right !
* def foo = { bar: '#("hello " + name)' }
```
Observe how you can achieve string concatenation if you really want, because any valid JavaScript expression can be stuffed within an embedded expression. You could always do this in two steps:
```cucumber
* def temp = 'hello ' + name
* def foo = { bar: '#(temp)' }
```
As a convenience, embedded expressions are supported on the Right Hand Side of a [`match`](#match) statement even for "quoted string" literals:
```cucumber
* def foo = 'a1'
* match foo == '#("a" + 1)'
```
And do note that in Karate 1.0 onwards, ES6 string-interpolation within "backticks" is supported:
```cucumber
* param filter = `ORDER_DATE:"${todaysDate}"`
```
### Enclosed JavaScript
An alternative to embedded expressions (for JSON only) is to enclose the entire payload within parentheses - which tells Karate to evaluate it as pure JavaScript. This can be a lot simpler than embedded expressions in many cases, and JavaScript programmers will feel right at home.
The example below shows the difference between embedded expressions and enclosed JavaScript:
```cucumber
When def user = { name: 'john', age: 21 }
And def lang = 'en'
* def embedded = { name: '#(user.name)', locale: '#(lang)', sessionUser: '#(user)' }
* def enclosed = ({ name: user.name, locale: lang, sessionUser: user })
* match embedded == enclosed
```
> So how would you choose between the two approaches to create JSON ? [Embedded expressions](#embedded-expressions) are useful when you have complex JSON [`read`](#reading-files) from files, because you can auto-replace (or even [remove](#remove-if-null)) data-elements with values dynamically evaluated from variables. And the JSON will still be 'well-formed', and editable in your IDE or text-editor. Embedded expressions also make more sense in [validation](#ignore-or-validate) and [schema-like](#schema-validation) short-cut situations. It can also be argued that the `#` symbol is easy to spot when eyeballing your test scripts - which makes things more readable and clear.
### Multi-Line Expressions
The keywords [`def`](#def), [`set`](#set), [`match`](#match), [`request`](#request) and [`eval`](#eval) take multi-line input as the last argument. This is useful when you want to express a one-off lengthy snippet of text in-line, without having to split it out into a separate [file](#reading-files). Note how triple-quotes (`"""`) are used to enclose content. Here are some examples:
```cucumber
# instead of:
* def cat = Billie25
# this is more readable:
* def cat =
"""
Billie
2
5
"""
# example of a request payload in-line
Given request
"""
12341234
"""
# example of a payload assertion in-line
Then match response ==
"""
{ id: { domain: "DOM", type: "entityId", value: "#ignore" },
created: { on: "#ignore" },
lastUpdated: { on: "#ignore" },
entityState: "ACTIVE"
}
"""
```
## `table`
### A simple way to create JSON Arrays
Now that we have seen how JSON is a 'native' data type that Karate understands, there is a very nice way to create JSON using the support for expressing [data-tables](http://www.thinkcode.se/blog/2014/06/30/cucumber-data-tables).
```cucumber
* table cats
| name | age |
| 'Bob' | 2 |
| 'Wild' | 4 |
| 'Nyan' | 3 |
* match cats == [{name: 'Bob', age: 2}, {name: 'Wild', age: 4}, {name: 'Nyan', age: 3}]
```
The [`match`](#match) keyword is explained later, but it should be clear right away how convenient the `table` keyword is. JSON can be combined with the ability to [call other `*.feature` files](#data-driven-features) to achieve dynamic data-driven testing in Karate.
Notice that in the above example, string values within the table need to be enclosed in quotes. Otherwise they would be evaluated as expressions - which does come in useful for some dynamic data-driven situations:
```cucumber
* def one = 'hello'
* def two = { baz: 'world' }
* table json
| foo | bar |
| one | { baz: 1 } |
| two.baz | ['baz', 'ban'] |
* match json == [{ foo: 'hello', bar: { baz: 1 } }, { foo: 'world', bar: ['baz', 'ban'] }]
```
Yes, you can even nest chunks of JSON in tables, and things work as you would expect.
Empty cells or expressions that evaluate to `null` will result in the key being omitted from the JSON. To force a `null` value, wrap it in parentheses:
```cucumber
* def one = { baz: null }
* table json
| foo | bar |
| 'hello' | |
| one.baz | (null) |
| 'world' | null |
* match json == [{ foo: 'hello' }, { bar: null }, { foo: 'world' }]
```
An alternate way to create data is using the [`set` multiple](#set-multiple) syntax. It is actually a 'transpose' of the `table` approach, and can be very convenient when there are a large number of keys per row or if the nesting is complex. Here is an example of what is possible:
```cucumber
* set search
| path | 0 | 1 | 2 |
| name.first | 'John' | 'Jane' | |
| name.last | 'Smith' | 'Doe' | 'Waldo' |
| age | 20 | | |
* match search[0] == { name: { first: 'John', last: 'Smith' }, age: 20 }
* match search[1] == { name: { first: 'Jane', last: 'Doe' } }
* match search[2] == { name: { last: 'Waldo' } }
```
## `text`
### Don't parse, treat as raw text
Not something you would commonly use, but in some cases you need to disable Karate's default behavior of attempting to parse anything that looks like JSON (or XML) when using [multi-line](#multi-line-expressions) / string [expressions](#karate-expressions). This is especially relevant when manipulating [GraphQL](http://graphql.org) queries - because although they look suspiciously like JSON, they are not, and tend to confuse Karate's internals. And as shown in the example below, having text 'in-line' is useful especially when you use the `Scenario Outline:` and `Examples:` for [data-driven tests](#data-driven-tests) involving place-holder substitutions in strings.
```cucumber
Scenario Outline:
# note the 'text' keyword instead of 'def'
* text query =
"""
{
hero(name: "") {
height
mass
}
}
"""
Given path 'graphql'
And request { query: '#(query)' }
And header Accept = 'application/json'
When method post
Then status 200
Examples:
| name |
| John |
| Smith |
```
Note that if you did not need to inject [`Examples:`](#data-driven-tests) into 'placeholders' enclosed within `<` and `>`, [reading from a file](#reading-files) with the extension `*.txt` may have been sufficient.
For placeholder-substitution, the [`replace`](#replace) keyword can be used instead, but with the advantage that the text can be read from a file or dynamically created.
Karate is a great fit for testing GraphQL because of how easy it is to deal with dynamic and deeply nested JSON responses. Refer to this example for more details: [`graphql.feature`](karate-demo/src/test/java/demo/graphql/graphql.feature).
## `replace`
### Text Placeholder Replacement
> Modifying existing JSON and XML is __natively__ supported by Karate via the [`set`](#set) keyword, and `replace` is primarily intended for dealing with raw strings. But when you deal with complex, nested JSON (or XML) - it may be easier in some cases to use `replace`, especially when you want to substitute multiple placeholders with one value, and when you don't need array manipulation. Since `replace` auto-converts the result to a string, make sure you perform [type conversion](#type-conversion) back to JSON (or XML) if applicable.
Karate provides an elegant 'native-like' experience for placeholder substitution within strings or text content. This is useful in any situation where you need to concatenate dynamic string fragments to form content such as GraphQL or SQL.
The placeholder format defaults to angle-brackets, for example: ``. Here is how to replace one placeholder at a time:
```cucumber
* def text = 'hello world'
* replace text.foo = 'bar'
* match text == 'hello bar world'
```
Karate makes it really easy to substitute multiple placeholders in a single, readable step as follows:
```cucumber
* def text = 'hello world bye'
* replace text
| token | value |
| one | 'cruel' |
| two | 'good' |
* match text == 'hello cruel world good bye'
```
Note how strings have to be enclosed in quotes. This is so that you can mix expressions into text replacements as shown below. This example also shows how you can use a custom placeholder format instead of the default:
```cucumber
* def text = 'hello world ${two} bye'
* def first = 'cruel'
* def json = { second: 'good' }
* replace text
| token | value |
| one | first |
| ${two} | json.second |
* match text == 'hello cruel world good bye'
```
Refer to this file for a detailed example: [`replace.feature`](karate-core/src/test/java/com/intuit/karate/core/replace.feature)
## YAML Files
For those who may prefer [YAML](http://yaml.org) as a simpler way to represent data, Karate allows you to read YAML content from a [file](#reading-files) - and it will be auto-converted into JSON.
```cucumber
# yaml from a file (the extension matters), and the data-type of 'bar' would be JSON
* def bar = read('data.yaml')
```
### `yaml`
A very rare need is to be able to convert a string which happens to be in YAML form into JSON, and this can be done via the `yaml` type cast keyword. For example - if a response data element or downloaded file is YAML and you need to use the data in subsequent steps. Also see [type conversion](#type-conversion).
```cucumber
* text foo =
"""
name: John
input:
id: 1
subType:
name: Smith
deleted: false
"""
# yaml to json type conversion
* yaml foo = foo
* match foo ==
"""
{
name: 'John',
input: {
id: 1,
subType: { name: 'Smith', deleted: false }
}
}
"""
```
## CSV Files
Karate can read `*.csv` files and will auto-convert them to JSON. A header row is always expected. See the section on [reading files](#reading-files) - and also this example [`dynamic-csv.feature`](karate-demo/src/test/java/demo/outline/dynamic-csv.feature), which shows off the convenience of [dynamic `Scenario Outline`-s](#dynamic-scenario-outline).
In rare cases you may want to use a csv-file as-is and *not* auto-convert it to JSON. A good example is when you want to use a CSV file as the [request-body](#request) for a file-upload. You could get by by renaming the file-extension to say `*.txt` but an alternative is to use the [`karate.readAsString()`](#read-file-as-string) API.
### `csv`
Just like [`yaml`](#yaml), you may occasionally need to [convert a string](#type-conversion) which happens to be in CSV form into JSON, and this can be done via the `csv` keyword.
```cucumber
* text foo =
"""
name,type
Billie,LOL
Bob,Wild
"""
* csv bar = foo
* match bar == [{ name: 'Billie', type: 'LOL' }, { name: 'Bob', type: 'Wild' }]
```
## JavaScript Functions
JavaScript Functions are also 'native'. And yes, functions can take arguments.
> Standard JavaScript syntax rules apply, but the right-hand-side should begin with the `function` keyword if declared *in-line*. When using stand-alone `*.js` files, you can have a comment before the `function` keyword, and you can use `fn` as the function name, so that your IDE does not complain about JavaScript syntax errors, e.g. `function fn(x){ return x + 1 }`
```cucumber
* def greeter = function(title, name) { return 'hello ' + title + ' ' + name }
* assert greeter('Mr.', 'Bob') == 'hello Mr. Bob'
```
> When JavaScript executes in Karate, the built-in [`karate` object](#the-karate-object) provides some commonly used utility functions. And with [Karate expressions](#karate-expressions), you can "dive into" JavaScript without needing to define a function - and [conditional logic](#conditional-logic) is a good example.
### Java Interop
For more complex functions you are better off using the [multi-line](#multi-line-expressions) 'doc-string' approach. This example actually calls into existing Java code, and being able to do this opens up a whole lot of possibilities. The JavaScript interpreter will try to convert types across Java and JavaScript as smartly as possible. For e.g. JSON objects become Java `Map`-s, JSON arrays become Java `List`-s, and Java Bean properties are accessible (and update-able) using 'dot notation' e.g. '`object.name`'
```cucumber
* def dateStringToLong =
"""
function(s) {
var SimpleDateFormat = Java.type('java.text.SimpleDateFormat');
var sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZ");
return sdf.parse(s).time; // '.getTime()' would also have worked instead of '.time'
}
"""
* assert dateStringToLong("2016-12-24T03:39:21.081+0000") == 1482550761081
```
> More examples of Java interop and how to invoke custom code can be found in the section on [Calling Java](#calling-java).
The `call` keyword provides an [alternate way of calling JavaScript functions](#calling-javascript-functions) that have only one argument. The argument can be provided after the function name, without parentheses, which makes things slightly more readable (and less cluttered) especially when the solitary argument is JSON.
```cucumber
* def timeLong = call dateStringToLong '2016-12-24T03:39:21.081+0000'
* assert timeLong == 1482550761081
# a better example, with a JSON argument
* def greeter = function(name){ return 'Hello ' + name.first + ' ' + name.last + '!' }
* def greeting = call greeter { first: 'John', last: 'Smith' }
```
## Reading Files
Karate makes re-use of payload data, utility-functions and even other test-scripts as easy as possible. Teams typically define complicated JSON (or XML) payloads in a file and then re-use this in multiple scripts. Keywords such as [`set`](#set) and [`remove`](#remove) allow you to to 'tweak' payload-data to fit the scenario under test. You can imagine how this greatly simplifies setting up tests for boundary conditions. And such re-use makes it easier to re-factor tests when needed, which is great for maintainability.
> Note that the [`set` (multiple)](#set-multiple) keyword can build complex, nested JSON (or XML) from scratch in a data-driven manner, and you may not even need to read from files for many situations. Test data can be within the main flow itself, which makes scripts highly readable.
Reading files is achieved using the built-in JavaScript function called `read()`. By default, the file is expected to be in the same folder (package) and side-by-side with the `*.feature` file. But you can prefix the name with `classpath:` in which case the ['root' folder](#classpath) would be `src/test/java` (assuming you are using the [recommended folder structure](#folder-structure)).
Prefer [`classpath:`](#classpath) when a file is expected to be heavily re-used all across your project. And yes, relative paths will work.
```cucumber
# json
* def someJson = read('some-json.json')
* def moreJson = read('classpath:more-json.json')
# xml
* def someXml = read('../common/my-xml.xml')
# import yaml (will be converted to json)
* def jsonFromYaml = read('some-data.yaml')
# csv (will be converted to json)
* def jsonFromCsv = read('some-data.csv')
# string
* def someString = read('classpath:messages.txt')
# javascript (will be evaluated)
* def someValue = read('some-js-code.js')
# if the js file evaluates to a function, it can be re-used later using the 'call' keyword (or invoked just like normal js)
* def someFunction = read('classpath:some-reusable-code.js')
* def someCallResult = call someFunction
* def sameCallResult = someFunction()
# the following short-cut is also allowed
* def someCallResult = call read('some-js-code.js')
```
You can also [re-use other `*.feature`](#calling-other-feature-files) files from test-scripts:
```cucumber
# perfect for all those common authentication or 'set up' flows
* def result = call read('classpath:some-reusable-steps.feature')
```
When a *called* feature depends on some side-by-side resources such as JSON or JS files, you can use the `this:` prefix to ensure that relative paths work correctly - because by default Karate calculates relative paths from the "root" feature or the top-most "caller".
```cucumber
* def data = read('this:payload.json')
```
If a file does not end in `.json`, `.xml`, `.yaml`, `.js`, `.csv` or `.txt`, it is treated as a stream - which is typically what you would need for [`multipart`](#multipart-field) file uploads.
```cucumber
* def someStream = read('some-pdf.pdf')
```
> The `.graphql` and `.gql` extensions are also recognized (for GraphQL) but are handled the same way as `.txt` and treated as a string.
For JSON and XML files, Karate will evaluate any [embedded expressions](#embedded-expressions) on load. This enables more concise tests, and the file can be re-usable in multiple, data-driven tests.
Since it is internally implemented as a JavaScript function, you can mix calls to `read()` freely wherever JavaScript expressions are allowed:
```cucumber
* def someBigString = read('first.txt') + read('second.txt')
```
> Tip: you can even use JS expressions to dynamically choose a file based on some condition: `* def someConfig = read('my-config-' + someVariable + '.json')`. Refer to [conditional logic](#conditional-logic) for more ideas.
And a very common need would be to use a file as the [`request`](#request) body:
```cucumber
Given request read('some-big-payload.json')
```
Or in a [`match`](#match):
```cucumber
And match response == read('expected-response-payload.json')
```
The rarely used `file:` prefix is also supported. You could use it for 'hard-coded' absolute paths in dev mode, but is obviously not recommended for CI test-suites. A good example of where you may need this is if you programmatically write a file to the `target` folder, and then you can read it like this:
```cucumber
* def payload = read('file:target/large.xml')
```
### Path Prefixes
To summarize the possible prefixes:
Prefix | Description
------ | -----------
`classpath:` | relative to the [classpath](#classpath), recommended for re-usable features
`file:` | do not use this unless you know what you are doing, see above
`this:` | when in a *called* feature, ensure that files are resolved relative to the current feature file
Take a look at the [Karate Demos](karate-demo) for real-life examples of how you can use files for validating HTTP responses, like this one: [`read-files.feature`](karate-demo/src/test/java/demo/read/read-files.feature).
### Read File As String
In some rare cases where you don't want to auto-convert JSON, XML, YAML or CSV, and just get the raw string content (without having to re-name the file to end with `.txt`) - you can use the [`karate.readAsString()`](#karate-readasstring) API. Here is an example of using a CSV file as the request-body:
```cucumber
Given path 'upload'
And header Content-Type = 'text/csv'
And request karate.readAsString('classpath:my.csv')
When method post
Then status 202
```
## Compare Image
Karate provides a flexible way to compare two images to determine if they are the same or similar. This is especially useful when capturing screenshots during tests and comparing against baseline images that are known to be correct.
> A stand-alone example can be found here: [`examples/image-comparison`](examples/image-comparison) along with a [video explanation](https://youtu.be/wlvmNBraP60).
Below is a simple example that will compare a `baseline` image to a more recent `latest` image. An image comparison UI will also be embedded into the Karate HTML report with detailed information about any differences between the two images.
```cucumber
* compareImage { baseline: 'screenshots/login.png', latest: '/tmp/login.png' }
```
You can also compare images using Karate [path prefixes](#path-prefixes) (e.g. `classpath:`, `this:`, `file:`) or byte arrays:
```cucumber
* def latestImgBytes = karate.readAsBytes('login.png')
* compareImage { baseline: 'classpath:screenshots/login.png', latest: '#(latestImgBytes)' }
```
You may configure the following image comparison options using the `configure` action:
```cucumber
* configure imageComparison = { /* image comparison options ... */ }
```
Image comparison configuration options:
| Key | Type | Default | Description |
|-----------------------|-------------|------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `allowScaling` | boolean | `false` | When `true` we will scale latest images to match the dimensions of the baseline when they are not equal |
| `engine` | string | 'resemble' | Comparison engine(s) to use. Valid options are `resemble` and `ssim` separated by either `,` or | |
| `failureThreshold` | number | `0.0` | Precentage of `latest` image pixels allowed to differ from `baseline` before we consider the comparison as failed |
| `mismatchShouldPass` | boolean | `false` | When `true` all image comparisons will pass (even when difference is >= `failureThreshold`). Note: failures will result in image comparison UI *always* being embedded in Karate HTML reports regardless of `hideUiOnSuccess` setting. |
| `onShowRebase` | string (js) | `null` | Function to be called when displaying image comparison rebase in Karate HTML reports (e.g. to customize rebase filename and/or output) |
| `onShowConfig` | string (js) | `null` | Function to be called when displaying image comparison configuration in Karate HTML reports (e.g. to customize configuration output) |
| `hideUiOnSuccess` | boolean | `false` | When `true` the comparison UI will *NOT* be embedded in Karate HTML reports for all non-failed image comparisons |
Examples:
```cucumber
# use only 'ssim' (structural similarity) engine
* configure imageComparison = { engine: 'ssim' }
# always use both 'resemble' and 'ssim' engines but only evaluate the lowest mismatch percentage against our `failureThreshold`
* configure imageComparison = { engine: 'resemble,ssim' }
# prefer 'resemble' and fallback to 'ssim' engine only if the resemble mismatch percentage is >= `failureThreshold`
* configure imageComparison = { engine: 'resemble|ssim' }
# only consider the comparison as failed when 2% or more pixels are different from the baseline
* configure imageComparison = { failureThreshold: 2 }
# consider image comparisons that fail due to too many mismatched pixels as passed (especially useful when you are first starting without any baseline images)
* configure imageComparison = { mismatchShouldPass: true }
# custom JS function called in Karate HTML image comparison UI when the user clicks the `Rebase` button
* text onShowRebaseFn =
"""
function (config, downloadLatestFn) {
// trigger download of latest image with custom file name
downloadLatestFn('custom_latest.png')
return 'this text will be displayed to the user when they click the rebase button'
}
"""
* configure imageComparison = { onShowRebase: '#(onShowRebaseFn)' }
# custom JS function called in Karate HTML image comparison UI when the user clicks the `Show config` button
* text onShowConfigFn =
"""
function (customConfigJson, config) {
return 'this text will be displayed above the image comparison config\n' + customConfigJson
}
"""
* configure imageComparison = { onShowConfig: '#(onShowConfigFn)' }
# don't embed the image comparison UI when the latest image is the same / similar to the baseline (e.g. to save space and speed up report loading)
* configure imageComparison = { hideUiOnSuccess: true }
```
Image comparison engines can also be customized:
```cucumber
* compareImage { baseline: 'baseline.png', latest: 'latest.png', options: { /* engine options ... */ } }
```
Image comparison configuration options:
| Key | Engines | Type | Default | Description |
|--------------------------|----------------|-----------------|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `ignoredBoxes` | resemble, ssim | array of object | `null` | Array of rectangles that should be ignored during image comparison |
| `ignore` | resemble | string | 'less' | Resemble ignore preset. Valid options are `nothing`, `less`, `antialiasing`, `colors`, `alpha` (see [presets](https://github.com/t12y/resemble/blob/0c55a1849232b68aa4c16f1d7a949cc429a6af8e/src/main/java/io/github/t12y/resemble/Options.java#L20)). |
| `ignoreAreasColoredWith` | resemble | object | `null` | Resemble option to ignore a specific color |
| `ignoreColors` | resemble | boolean | `false` | When `true` only pixel brightness is compared |
| `ignoreAntialiasing` | resemble | boolean | `false` | When `true` only pixel brightness is compared for pixels determined to be antialiased |
| `tolerances` | resemble | object | `null` | Resemble option to override preset tolerances for color and brightness |
| `ssim` | ssim | string | 'WEBER' | SSIM algorithm. Valid options are `FAST` or `WEBER`