Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/viclovsky/swagger-coverage

Tool which generates full picture of coverage of API tests based on OAS (Swagger) v2 and v3
https://github.com/viclovsky/swagger-coverage

coverage oas swagger

Last synced: 1 day ago
JSON representation

Tool which generates full picture of coverage of API tests based on OAS (Swagger) v2 and v3

Awesome Lists containing this project

README 

        
[license]: http://www.apache.org/licenses/LICENSE-2.0 "Apache License 2.0"

[release]: https://github.com/viclovsky/swagger-coverage/releases/latest "Latest release"

[release-badge]: https://img.shields.io/github/release/viclovsky/swagger-coverage.svg?style=flat

[maven]: https://repo.maven.apache.org/maven2/com/github/viclovsky/swagger-coverage-commandline/ "Maven Central"

[maven-badge]: https://img.shields.io/maven-central/v/com.github.viclovsky/swagger-coverage-commandline.svg?style=flat



[![Build Status](https://github.com/viclovsky/swagger-coverage/workflows/Build/badge.svg)](https://github.com/viclovsky/swagger-coverage/actions)

[![release-badge][]][release]

[![maven-badge][]][maven] 



# swagger-coverage

Swagger-coverage gives a full picture about coverage of API tests (regression) based on OAS (Swagger). 

By saying coverage we mean not a broad theme functionality, but presence (or absence) of calls defined by API methods, parameters, return codes or other conditions which corresponds specification of API.



![Swagger Coverage Report](.github/swagger-coverage.png)



## How it works

Producing coverage report consists of two parts. Firstly, during test execution, filter/interceptor/proxy save information of calls in swagger format in specific folder on executing tests.

The next stage is to compare saved result with generated conditions from current API specification and builds report.  



## How to use and examples

You can use swagger-coverage with any language and framework. You need to have proxy/filter/interceptor that accumulates data in swagger format. 

Swagger-coverage have rest-assured integration from the box.



> There is also a Karate integration, which has its own [manual](/swagger-coverage-karate/README.md).



Add filter dependency:

```xml

 

     com.github.viclovsky

     swagger-coverage-rest-assured

     ${latest-swagger-coverage-version}

 

```

or if use gradle, it can be added like



```

compile "com.github.viclovsky:swagger-coverage-rest-assured:$latest-swagger-coverage-version"

```



Just add filter into test client SwaggerCoverageRestAssured (SwaggerCoverageV3RestAssured for v3). For instance, as presented below:

```java

RestAssured.given().filter(new SwaggerCoverageRestAssured())

```



* Download and run command line.

Download zip archive and unpack it. Don't forget to replace {latest-swagger-coverage-version} to latest version. 

```

wget https://github.com/viclovsky/swagger-coverage/releases/download/{latest-swagger-coverage-version}/swagger-coverage-{latest-swagger-coverage-version}.zip

unzip swagger-coverage-commandline-{latest-swagger-coverage-version}.zip

```



Here is help of unzip swagger-commandline



```

./swagger-coverage-commandline --help



  Options:

  * -s, --spec

      Path to local or URL to remote swagger specification.

  * -i, --input

      Path to folder with generated files with coverage.

    -c, --configuration

      Path to file with report configuration.

    --help

      Print commandline help.

    -q, --quiet

      Switch on the quiet mode.

      Default: false

    -v, --verbose

      Switch on the verbose mode.

      Default: false

```



To compare result of API tests with current API specification and build report call command line tool after running tests like that:



```

./swagger-coverage-commandline -s swagger.json -i swagger-coverage-output

```



Output of the command:

```

19:21:21 INFO  OperationSwaggerCoverageCalculator - Empty coverage:

...

19:21:21 INFO  OperationSwaggerCoverageCalculator - Partial coverage:

...

19:21:21 INFO  OperationSwaggerCoverageCalculator - Full coverage:

...

19:21:21 INFO  OperationSwaggerCoverageCalculator - Conditions: 874/2520

19:21:21 INFO  OperationSwaggerCoverageCalculator - Empty coverage 49.284 %

19:21:21 INFO  OperationSwaggerCoverageCalculator - Partial coverage 12.034 %

19:21:21 INFO  OperationSwaggerCoverageCalculator - Full coverage 38.682 %

19:21:21 INFO  FileSystemResultsWriter - Write html report in file '.../swagger-coverage-report.html'

```

Results (swagger-coverage-report.html/swagger-coverage-results.json) will be created after running of swagger-coverage.



## Configuration options

Swagger-coverage report can be configured by json-file. 

You can control list of coverage, which be generated and checked for results.



## Rules configuration options

Options for different rules are placed in "rules" section. 

You can disable some rules or change their behavior.



#### Checking response http-status

This rule create condition for every status from *responses*-section of swagger specification.

Condition mark *covered* when report generator find specific status in results files.

Options for this rules are placed in *status* subsection in *rules* sections.



You can setup next options:



**enable** - *true/false*. You can disable this rule. Default value is *true*.



**filter** - *[val1,val2]*. Rule will ignore all status, which not in filter list.



**ignore** - *[val1,val2]*. Rule will ignore all status, which in ignore list.



```` 

{

  "rules" : {

    "status": {

      "enable": true,

      "ignore": ["400","500"],

      "filter": ["200"]

    },



    ....

  },

  

  ....

}

````



#### Checking the list of declared and received statuses

This rule create condition for comparing declared and received status. 

Condition marked as *covered* when result not contains any of undeclared status.

*Uncovered* state of this condition indicates missed status in original swagger-documentation 

or server errors.

Options for this rules are placed in *only-declared-status* subsection in *rules* sections.



You can setup next options:



**enable** - *true/false*. You can disable this rule. Default value is *true*.



````

{

  "rules" : {



    ....



    "only-declared-status" : {

      "enable" : true

    }

  },



   ....

}

````



#### Excluding deprecated operations from the coverage report statistic

This rule is created for cases when you don't want to measure coverage of deprecated operations, but only for actual ones. 


If an operation is deprecated then it will be excluded from *Full*, *Partial*, and *Empty* categories and won't affect the "Operations coverage summary"



Options for this rule are placed in "*exclude-deprecated*" subsection in *rules* sections.



You can set up next options:



**enable** - *true/false*. 


By default, this rule is not enabled. Add it to the config file with *true* value to enable this rule, like in the example below:



````

{

  "rules" : {



    ....



    "exclude-deprecated" : {

      "enable" : true

    }

  },



   ....

}

````



If you need you can add your rules for generation of conditions. So, please, send your PRs.



## Result writer configuration

Options for report generation setting are placed in *writers* sections.



#### HTML report writer

Options for html-report placed in subsection *html* of *writers* sections.



You can setup next options:



**locale** - two latter language code. Now supported only *en/ru*.



**filename** - filename for html report.



**numberFormat** - [Extended Java decimal format](https://freemarker.apache.org/docs/ref_builtins_number.html#topic.extendedJavaDecimalFormat) to control how numbers are displayed in the report.



````

{

  ....



  "writers": {

      "html": {

        "locale": "ru",

        "filename":"report.html",

        "numberFormat": "0.##"

      }

  }

}

````



#### Report customization

To customize your http report with your own template set full path to the template like below:

````

{

  ....

  

  "writers": {

    "html": {

      ....

      "customTemplatePath": "/full/path/to/report_custom.ftl"

    }

  }

}

````



[Look here](https://github.com/swagger-api/swagger-parser/blob/master/modules/swagger-parser-core/src/main/java/io/swagger/v3/parser/core/models/ParseOptions.java) to see all available options.



## Demo

I have prepared several tests. Thus you are able to have a look and touch swagger-coverage. Just run ```run.sh``` script.



## Important remark

Swagger-coverage works fine with clients which were generated from swagger (for example: https://github.com/OpenAPITools/openapi-generator). 

Because all methods/parameters which will be saved are 100% compatible with current API specification. 



## Requirements 



For a moment swagger-coverage  is compatible only with OpenApi specifications v2 & v3. It is possible that swagger-coverage will support other versions.



## Pull Requests

My project is open for any enhancement. So, your help is much appreciated. Please, feel free to open your pull request or issue and I will consider it in several days.



## Created & Maintained By

[Victor Orlovsky](https://github.com/viclovsky)



## Contributing to swagger-coverage

Thanks to all people who contributed. Especially 



* [@TemaMak](https://github.com/TemaMak) 

* [@Emilio-Pega](https://github.com/Emilio-Pega)



who have contributed significant improvements to swagger-coverage.



## License

Swagger coverage is released under version 2.0 of the [Apache License](http://www.apache.org/licenses/LICENSE-2.0)