https://github.com/martinmoene/catch-lest-other-comparison
Tabularised feature comparison between Catch, doctest and lest C++ test frameworks
https://github.com/martinmoene/catch-lest-other-comparison
catch comparison doctest lest tdd-framework
Last synced: 3 months ago
JSON representation
Tabularised feature comparison between Catch, doctest and lest C++ test frameworks
- Host: GitHub
- URL: https://github.com/martinmoene/catch-lest-other-comparison
- Owner: martinmoene
- License: mit
- Created: 2017-07-15T08:26:20.000Z (about 8 years ago)
- Default Branch: master
- Last Pushed: 2021-07-12T08:39:42.000Z (about 4 years ago)
- Last Synced: 2025-04-04T00:24:57.956Z (6 months ago)
- Topics: catch, comparison, doctest, lest, tdd-framework
- Homepage:
- Size: 48.8 KB
- Stars: 30
- Watchers: 4
- Forks: 3
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Catch–doctest–lest feature comparison
Tabularised feature comparison between Catch, doctest and lest C++ test frameworks.
[Skip to the feature comparison](#table).
Note: This is an initial draft, it is incomplete and likely contains errors.
Ideas, additions, corrections, signaling omissions, etc. welcome!
---
**The frameworks in this comparison share the following properties:**
- The frameworks are available as single-file header-only library.
- Test cases are described by a string, they are not identified by a function name.
- There is a single assertion macro for expressions.
Expressions are decomposed and original expression code can be shown alongside the values they represent.
- Fixtures can be created inside a test case. They are called sections or sub cases.
**Advent of Catch** Expression-decomposing assertions and function-level fixtures (sections) were introduced by Kevlin Henney in his talk *Rethinking Unit Testing in C++* at ACCU London in May 2010 [[1]](#KH1). At that time, Phil Nash was working on a testing framework called *YACUTS* (Yet Another C++ Unit Test System) that would bring similar capabilities, be it a little more intrusively [[2]](#PN1). Kevlin's talk inspired Phil to develop ***Catch*** [[3]](#PN2).
**Kevlin's emphasis on simplicity** in testing and the desire to understand how expression decomposition works inspired Martin Moene to write ***lest*** [[4]](#MM1). The code of *lest* is intentionally kept small, so that it should be fairly easy to gain insight in how things work.
**Accompanying test code with the code to test** was one of the drivers for Viktor Kirilov to write ***doctest*** [[5]](#VK1). Others are to enable using *doctest* with many different compilers and to shine in compile-time and run-time performance. At CppCon 2017 Viktor gave a presentation that explains many techniques used in doctest and Catch [[23]](#VK2).
Feature | [doctest](#DOT)| [Catch-1](#CA1)| [Catch-2](#CA2)| [lest](#LST)| Notes |
----------------------------------|:------:|:------:|:------:|:------:|-------|
**License** | MIT |BSL-1.0 |BSL-1.0 |BSL-1.0 | |
| | | | | |
**C++ standard related** | | | | | |
C++ standard | C++98 | C++98 | C++11 | C++11 | |
Can work without exceptions |✓| - | - | - | reduced functionality |
Requires RTTI | - | - | - | - | |
| | | | | |
**Fields of use** | | | | | |
Desktop |✓|✓|✓|✓| |
Embedded (tiny) | ? | ? | ? | ? | |
Embedded (largish) | ? | ? | ? |✓| |
Application, tool | ? |✓|✓| ? | |
Library, framework | ? |✓|✓|✓| |
Operating System | ? | ? | ? |✓| lest: IncludeOS |
| | | | | |
**Documentation of framework** | | | | | see [[22]](#WGD) |
First contact | ? | ? | ? | ? | new users |
Education | ? | ? | ? | ? | new & existing users |
Support | ? | ? | ? | ? | experienced users |
Troubleshooting | ? | ? | ? | ? | annoyed users |
Internals | ? | ? | ? | ? | fellow developers |
Reference | ? | ? | ? | ? | everyone|
Try it online |✓|✓|✓|✓| on wandbox |
Forum | - |✓|✓| - | Catch: Google group |
Chat |✓| - | - | - | doctest: Gitter |
StackOverflow tag | - |✓|✓| - | see [[19]](#SOC) |
| | | | | |
**Test organisation** | | | | | |
Suites of tests |✓| - | - | - | |
Fixtures (sections) |✓|✓|✓|✓| doctest: sub case |
Fixtures (class-based) |✓|✓|✓| - | |
Type-parameterised tests |✓|✓|✓| - | |
Value-parameterised tests | - | - | - | - | |
Auto-registration of tests |✓|✓|✓|✓| |
Tabularised tests | - | - | - |✓| array of lambdas |
Allow tests accompany code |✓|✓|✓| - | Catch: not recommended... |
Allow tests in header files |✓|✓|✓| - | ...to avoid long compilation times |
| | | | | |
**Assertions** | | | | | |
BDD style scenarios |✓|✓|✓|✓| |
Assert expressions |✓|✓|✓|✓| |
Assert exceptions |✓|✓|✓|✓| |
Assert abortion (death) | - | - | - | - | |
Assert assertions (death) | - | - | - | - | |
Expression decomposition |✓|✓|✓|✓| see [[2]](#PN1) |
Floating point comparison, approx |✓|✓|✓|✓| |
Floating point comparison, ULP | - | - |✓| - | see stf [[10]](#ULP)|
Hamcrest matchers (built-in) | - | some | some | • | see [[12](#C1M)][[13](#C2M)][[14](#HLM)][[15](#HCT)] |
Hamcrest matchers (user-definable)| - |✓|✓| - | |
Macros with and without prefix |✓|✓|✓|✓| CATCH_CHECK(), CHECK()|
| | | | | |
**Other test facilities** | | | | | |
Mocking support | • | • | • | • | via 3rd party |
Logging facility |✓|✓|✓| - | |
Logging levels | - | - | - | - | |
Checkpoints | - | - | - | - | see Boost.Test [[11]](#BTC) |
Test decorators |✓| - | - | - | see [[16](#DTD)] |
Test data generators | - | • | - | - | Cach-1: undocumented |
Property-based testing | - | - | - | - | |
Obtain name of current test | ? | • | ? | - | Catch-1: undocumented |
| | | | | |
**API / seams** | | | | | |
Reporting user-defined types |✓|✓|✓|✓| |
Reporter API | - |✓|✓| - | |
Event listeners | - |✓|✓| - | |
Run-time context |✓|✓|✓| - | |
| | | | | |
**Reporting formats** | | | | | |
Console, multi-line |✓|✓|✓| - | |
Console, single-line (compact) | - |✓|✓|✓| similar to compiler error |
JUnit | - |✓|✓| - | result at end|
XML | - |✓|✓| - | streaming|
TeamCity | - |✓|✓| - | |
TAP | - |✓|✓| - | |
Automake | - |✓|✓| - | |
| | | | | |
**Reporting options** | | | | | |
Colourised output (run-time) |✓|✓|✓| | |
Colourised output (compile-time) |✓| - | - |✓| |
Literal suffix u, l, f | - | - | - |✓| compile-time |
| | | | | |
**Test execution** | | | | | |
Concurrent execution of tests | - | - | - | - | |
Isolated execution of tests | - | - | - | - | |
Limit test execution time |✓| - | - | - | doctest: via test decorator |
Signal tests without assertions | - |✓|✓| - | |
Supports floating point exceptions| - | - | - | - | |
Supports Posix signals |✓|✓|✓| - | |
Supports Windows SEH |✓|✓|✓| - | |
| | | | | |
**Compile-time control** | | | | | |
Improve compilation speed |✓|✓|✓|- | reduced functionality |
Omit macros without prefix |✓|✓|✓|✓| |
Omit std::wstring | - | - | - |✓| |
Omit std::cout, cerr, clog | - |✓|✓| - | |
Specify terminal color system |✓|✓|✓| - | none, ansi, win32 |
Specify terminal width | - |✓|✓| - | |
Control handling of POSIX signals |✓|✓|✓| - | |
Control handling of SE on Windows |✓|✓|✓| - | |
Enable leak checking on Windows | - |✓|✓| - | uses CRT Debug Heap|
Enable use of \_\_COUNTER\_\_ | - |✓|✓| - | for internal unique names|
| | | | | |
**Run-time control, commandline** | | | | | |
Help screen |✓|✓|✓|✓| |
List selected suites |✓| - | - | - | |
List selected tests |✓|✓|✓|✓| |
List tags of selected tests | - |✓|✓|✓| |
Select tests by suite |✓| - | - | - | |
Select tests by name |✓|✓|✓|✓| |
Select tests by tag |✓|✓|✓|✓| |
Select tests by file |✓|✓|✓| - | Catch: via filename as tags |
Select section by name |✓|✓|✓| - | |
Select supports regexp |✓| - | - |✓| |
Omit assertions expected to throw | - |✓|✓| - | |
Count selected tests |✓| - | - |✓| |
Abort at N-th failure |✓|✓|✓|✓| lest: N is 1 |
Break into debugger |✓|✓|✓| - | |
Report passing tests |✓|✓|✓|✓| |
Time duration of tests |✓|✓|✓|✓| |
Control order of tests |✓|✓|✓|✓| |
Repeat tests | - |✓|✓|✓| |
| | | | | |
**IDE integration** | | | | | see [[17]](#IDE) |
Output-window-compatible reports |✓|✓|✓|✓| click through to source |
C++ Builder | ? | ? | ? | ? | |
CLion | ✓|✓|✓| - | |
Code::Blocks | ? | ? | ? | ? | |
CodeLite | ? | ? | ? | ? | |
Eclipse CDT | ? | ? | ? | ? | |
KDevelop | ? | ? | ? | ? | |
Qt Creator | ? | ? | ✓| ? | |
Visual-MinGW | ? | ? | ? | ? | |
Visual Studio | • | • | • | - | Catch and doctest: via ReSharper C++ |
Visual Studio Code | - | - | - | - | see [issue](https://github.com/Microsoft/vscode-cpptools/issues/1079)|
| | | | | |
**Platforms** | | | | | |
AIX / XL | ? | 11.1.0 | ? | ? | Catch-1: 2013 |
macOS / clang | ? | ? | ? | ? | |
macOS / gcc | 4.4-6 | ? | ? | ? | source: Travis |
macOS / XCode / clang | 6.4-8.3| 7.3-8 | 7.3-8 | ? | source: Travis |
macOS / XCode / gcc | 6.4-7.3| 7.3-8 | 7.3-8 | ? | source: Travis |
Linux / clang | 3.4-4 | 3.4-3.8| 3.5-3.8| 3.6-3.7| source: Travis |
Linux / gcc | 4.4-6 | 4.4-6 | 4.8-6 | 4.9-5 | source: Travis |
Solaris / SunPro | ? | 5.12.0 | ? | ? | Catch-1: 2013 |
Windows / clang | ? | ? | ? | 3.2+ | |
Windows / gcc | 7 | ? | ? | 4.8.1+ | |
Windows / Visual C++ | 9-15 | 10-15 | 14-15 | 12-15 | Catch-1: possibly from 8 |
Windows / Cygwin / clang | ? | ? | ? | ? | |
Windows / Cygwin / gcc | ? | ? | ? | ? | |
Windows / msys2 / clang | ? | ? | ? | ? | |
Windows / msys2 / gcc | ? | ? | ? | ? | |
| | | | | |
**Quality of project** | | | | | note: challenge *42* |
Code of conduct | - |✓|✓| - | |
Follows CII best practices |[✓](https://bestpractices.coreinfrastructure.org/projects/503)|[partly](https://bestpractices.coreinfrastructure.org/projects/1224)|[partly](https://bestpractices.coreinfrastructure.org/projects/1224)|[partly](https://bestpractices.coreinfrastructure.org/projects/1127)| see [[21]](#CII)|
| | | | | |
**Quality of code** | | | | | |
Unit testing |✓|✓|✓|✓| details below |
CI testing |✓|✓|✓|✓| details below |
CI code analysis |✓| - | - | - | details below |
| | | | | |
**Testing of framework** | | | | | |
Unit tests |✓|✓|✓|✓| |
Tests itself |✓|✓|✓|✓| |
Baseline tests |✓|✓|✓| - | |
| | | | | |
**CI testing** | | | | | |
Travis |✓|✓|✓|✓| |
Appveyor |✓|✓|✓|✓| |
Coverage scan |✓| - |✓| - | |
Coverity scan |✓| - | - | - | |
| | | | | |
**CI static code analysis** | | | | | |
clang-tidy |✓| - | - | - | |
CppCheck |✓| - | - | - | |
CppCoreCheck | - | - | - | - | |
PVS-Studio | - | - | - | - | |
OCLint |✓| - | - | - | |
ReSharper C++ | - | - | - | - | |
Visual Studio |✓| - | - | - | |
| | | | | |
**Code-related** | | | | | |
Compile-time performance, relative|☆☆☆|☆☆| ? | ? | see [[20]](#DTB) |
Run-time performance, relative |☆☆☆|☆☆| ? | ? | see [[20]](#DTB) |
Compiler-warning friendly |☆☆☆|☆☆|☆☆|☆☆| |
Compiles with -Werror -Wall -Wextra|✓|✓|✓|✓| |
Can provide canned main() |✓|✓|✓| - | |
Only depends on C++ std library |✓|✓|✓|✓| |
Single-file header-only |✓|✓|✓|✓| |
Size, LOC | 4,500 | 9,100 | 8,800 | 1,000 | see [[18]](#LOC) |
| | | | | |
**Distribution of framework** | | | | | |
GitHub single-file download |✓|✓|✓|✓| from the landing page |
conan |✓|✓|✓| - | |
hunter |✓| ? | - | - | |
vcpkg |✓|✓|✓| - | |
Linux and other (search) | [search](https://repology.org/metapackage/doctest/information) | → | [search](https://repology.org/metapackage/catch/information) | [search](https://repology.org/metapackage/lest/information) | on [Repology](https://repology.org/) |
| | | | | |
**Feature** | [doctest](#DOT)| [Catch-1](#CA1)| [Catch-2](#CA2)| [lest](#LST)| Notes |
References
----------
[1] Kevlin Henney. [Rethinking Unit Testing in C++](http://accu.org/index.php/accu_branches/accu_london/accu_london_may_2010) ([Video](http://skillsmatter.com/podcast/agile-testing/kevlin-henney-rethinking-unit-testing-in-c-plus-plus)). Skills Matter, May 2010.
[2] Phil Nash. [The Ultimate C++ Unit Test Framework](http://www.levelofindirection.com/journal/2010/5/21/the-ultimate-c-unit-test-framework.html). Blog, May 2010.
[3] Phil Nash. [Unit Testing in C++ and Objective-C just got easier](http://www.levelofindirection.com/journal/2010/12/28/unit-testing-in-c-and-objective-c-just-got-easier.html). Blog announcement, December 2010.
[4] Martin Moene. [lest errors escape testing](http://martin-moene.blogspot.nl/2013/06/lest-tiny-c11-unittest-helper.html). Blog announcement, June 2013.
[5] Viktor Kirilov. [doctest – the Lightest C++ Unit Testing Framework](https://accu.org/index.php/journals/2343). Announcement in Overload 137, February 2017.
[6] Phil Nash. [Catch-1 on GitHub](https://github.com/catchorg/Catch2/tree/Catch1.x). 2010.
[7] Phil Nash. [Catch-2 on GitHub](https://github.com/catchorg/Catch2). 2017.
[8] Viktor Kirilov. [doctest on GitHub](https://github.com/onqtam/doctest/). 2014.
[9] Martin Moene. [lest on GitHub](https://github.com/martinmoene/lest/). 2013.
[10] Joel Falcou. [Design rationale for using ULP with stf](https://github.com/jfalcou/stf/blob/master/doc/rationale.md).
[11] Gennadiy Rozental and Raffi Enficiaud. [Checkpoints in Boost.Test](http://www.boost.org/doc/libs/1_64_0/libs/test/doc/html/boost_test/test_output/test_tools_support_for_logging/checkpoints.html).
[12] [Matchers in Catch-1](https://github.com/catchorg/Catch2/blob/Catch1.x/docs/matchers.md#built-in-matchers).
[13] [Matchers in Catch-2](https://github.com/catchorg/Catch2/blob/master/docs/matchers.md#built-in-matchers).
[14] [Matchers in lest: via hamlest](https://github.com/martinmoene/hamlest).
[15] [Hamcrest 1.3 Quick Reference (PDF)](http://www.marcphilipp.de/downloads/posts/2013-01-02-hamcrest-quick-reference/Hamcrest-1.3.pdf).
[16] [Decorators in doctest](https://github.com/onqtam/doctest/blob/master/doc/markdown/testcases.md#decorators).
[17] Wikipedia. [List of C++ IDEs](https://en.wikipedia.org/wiki/Comparison_of_integrated_development_environments#C.2FC.2B.2B).
[18] [CodeBlocks IDE](http://www.codeblocks.org/). Non-empty, non-comment lines, via File > Properties.
[19] StackOverflow. [Questions tagged catch-unit-test](https://stackoverflow.com/questions/tagged/catch-unit-test).
[20] Viktor Kirilov. [doctest Benchmarks](https://github.com/onqtam/doctest/blob/master/doc/markdown/benchmarks.md#benchmarks). May 2017.
[21] Linux Foundation (LF). [CII Best Practices](https://bestpractices.coreinfrastructure.org/).
[22] Jacob Kaplan-Moss. [Writing great documentation](https://www.slideshare.net/jacobian/writing-great-documentation-codeconf-2011).
[23] Viktor Kirilov. [Mix Tests and Production Code With Doctest - Implementing and Using the Fastest Modern C++ Testing Framework](https://www.youtube.com/watch?time_continue=1&v=eH1CxEC29l8) ([slides](http://slides.com/onqtam/2017_cppcon_doctest)). [CppCon 2017](https://cppcon.org/).