Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/mrzechonek/tut-framework

C++ Template Unit Test Framework
https://github.com/mrzechonek/tut-framework

c-plus-plus unit-testing

Last synced: about 4 hours ago
JSON representation

C++ Template Unit Test Framework

Awesome Lists containing this project

README 

        
Documentation TUT How-To minimum steps to make TUT work for you



What is TUT



TUT is a pure C++ unit test framework. Its name - TUT - stands for 

Template Unit Tests.



Features



TUT provides all features required for unit testing:



    * Similar tests can be grouped together into test groups. Each 

      test group has its unique name and is located in a separate 

      compilation unit. One group can contain almost unlimited number 

      of tests (actually, the limit is the compiler template 

      recursion depth).

    * User can run all the tests (regression), or just some selected 

      groups or even some tests in these groups.

    * TUT provides special template functions to check the condition 

      validity at run-time and to force test failure if required. 

      Since C++ doesn't provide a facility for obtaining stack trace 

      of the throwed exception and TUT avoids macros, those functions 

      accept string marker to allow users easely determine the source 

      of exception.

    * TUT contains callback that can be implemented by the calling code 

      to integrate with an IDE, for example. Callbacks tell listener 

      when a new test run started, when test runner switches to the 

      next tests group, when a test was completed (and what result it 

      has), and when test run was finished. The callbacks allow users 

      to produce their own visualization format for test process and results.

    * Being a template library, it doesn't need compilation; just 

      include the  header into the test modules.



TUT tests organization



Test application



C++ produces executable code, so tests have to be compiled into a single 

binary called test application. The application can be built in automated 

mode to perform nightly tests. They also can be built manually when a 

developer hunts for bugs.



The test application contains tests, organized into test groups.



Test groups



The functionality of a tested application can be divided into a few separate 

function blocks (e.g. User Rights, Export, Processing, ...). It is natural 

to group together tests for each block. TUT invokes this test group. Each 

test group has a unique human-readable name and normally is located in a 

separate file.



Tests



Each single test usually checks only one specific element of functionality. 

For example, for a container a test could check whether size() call 

returns zero after the successful call to the clear() method.



Writing simple test



Preamble



You are going to create a new class for your application. You decided to 

write tests for the class to be sure it works while you are developing or, 

possibly, enhancing it. Let's consider your class is shared pointer: 

std::auto_ptr-alike type that shares the same object among instances.



Prior to test writing, you should decide what to test. Maximalist's 

approach requires to write so many tests that altering any single 

line of your production code will break at least one of them. 

Minimalist's approach allows one to write tests only for the most 

general or the most complex use cases. The truth lies somewhere in 

between, but only you, developer, know where. You should prepare 

common successful and unsuccessful scenarios, and the scenarios for 

testing any other functionality you believe might be broken in some way.



For our shared_ptr we obviosly should test constructors, assignment operators, referencing and passing ownership.



Skeleton



If you don't have any implemented class to test yet, it would be good to 

implement it as a set of stubs for a first time. Thus you'll get an 

interface, and be able to write your tests. Yes, that's correct: you 

should write your tests before writing code! First of all, writing tests 

often helps to understand oddities in the current interface, and fix it. 

Secondly, with the stubs all your tests will fail, so you'll be sure 

they do their job.



Creating Test Group



Since we're writing unit tests, it would be a good idea to group the 

tests for our class in one place to be able to run them separately. 

It's also natural in C++ to place all the grouped tests into one 

compilation unit (i.e. source file). So, to begin, we should create 

a new file. Let's call it test_shared_ptr.cpp. (Final variant of the 

test group can be found in examples/shared_ptr subdirectory of the 

distribution package)



// test_shared_ptr.cpp

#include 



namespace tut

{

}



As you see, you need to include TUT header file (as expected) and 

use namespace tut for tests. You may also use anonymous namespace if 

your compiler allows it (you will need to instantiate methods from 

tut namespace and some compilers refuse to place such instantiations 

into the anonymous namespace).



A test group in TUT framework is described by the special template 

test_group. The template parameter T is a type that will hold all 

test-specific data during the test execution. Actually, the data 

stored in T are member data of the test. Test object is inherited 

from T, so any test can refer to the data in T as its member data.



For simple test groups (where all data are stored in test local 

variables) type T is an empty struct.



#include 



namespace tut

{

  struct shared_ptr_data

  { 

  };

}



But when tests have complex or repeating creation phase, you may put 

data members into the T and provide constructor (and, if required, 

destructor) for it. For each test, a new instance of T will be 

created. To prepare your test for execution TUT will use default 

constructor. Similarly, after the test has been finished, TUT 

calls the destructor to clean up T. I.e.:



#include 



namespace tut

{

  struct complex_data

  {

    connection* con;

    complex_data(){ con = db_pool.get_connection(); }

    ~complex_data(){ db_pool.release_connection(con); }

  };



  // each test from now will have con data member initialized

  // by constructor:

  ...

  con->commit();

  ...

}



What will happen if the constructor throws an exception? TUT will treat 

it as if test itself failed with exception, so this test will 

not be executed. You'll see an exception mark near the test, and 

if the constructor throwed something printable, a certain message will appear.



Exception in destructor is threated a bit different. Reaching destruction 

phase means that the test is passed, so TUT marks test with warning 

status meaning that test itself was OK, but something bad has happend 

after the test.



Well, all we have written so far is just a type declaration. To work 

with a group we have to have an object, so we must create the test group 

object. Since we need only one test group object for each unit, we can 

(and should, actually) make this object static. To prevent name clash with 

other test group objects in the namespace tut, we should provide a 

descriptive name, or, alternatively, we may put it into the anonymous 

namespace. The former is more correct, but a descriptive name usually works 

well too, unless you're too terse in giving names to objects.



#include 



namespace tut

{

    struct shared_ptr_data

    {

    };



    typedef test_group tg;

    tg shared_ptr_group("shared_ptr");

}



As you see, any test group accepts a single parameter - its human-readable 

name. This name is used to identify the group when a programmer wants to 

execute all tests or a single test within the group. So this name shall 

also be descriptive enough to avoid clashes. Since we're writing tests 

for a specific unit, it's enough to name it after the unit name.



Test group constructor will be called at unspecified moment at the test 

application startup. The constructor performs self-registration; it calls 

tut::runner and asks it to store the test group object name and location. 

Any other test group in the system undergoes the same processing, i.e. 

each test group object registers itself. Thus, test runner can iterate 

all test groups or execute any test group by its name.



Newly created group has no tests associated with it. To be more precise, 

it has predefined set of dummy tests. By default, there are 50 tests in a 

group, including dummy ones. To create a test group with higher volume 

(e.g. when tests are generated by a script and their number is higher) 

we must provide a higher border of test group size when it is instantiated:



#include 



namespace tut

{

    struct huge_test_data

    {

    };



    // test group with maximum 500 tests

    typedef test_group testgroup;

    testgroup huge_test_testgroup("huge group");

}



Note also, that your compiler will possibly need a command-line switch 

or pragma to enlarge recursive instantiation depth. For g++, for 

example, you should specify at least --ftemplate-depth-501 to increase 

the depth. For more information see your compiler documentation.



Creating Tests



Now it's time to fill our test group with content.



In TUT, all tests have unique numbers inside the test group. Some 

people believe that textual names better describe failed tests in 

reports. I agree; but in reality C++ templates work good with numbers 

because they are compile-time constants and refuse to do the same 

with strings, since strings are in fact addresses of character 

buffers, i.e. run-time data.



As I mentioned above, our test group already has a few dummy tests; 

and we can replace any of them with something real just by writing 

our own version:



#include 



namespace tut

{

    struct shared_ptr_data{};



    typedef test_group testgroup;

    typedef testgroup::object testobject;

    testgroup shared_ptr_testgroup("shared_ptr");



    template<>

    template<>

    void testobject::test<1>()

    {

        // do nothing test

    }

}



So far this test does nothing, but it's enough to illustrate the concept.



All tests in the group belong to the type test_group::object. This 

class is directly inherited from our test data structure. In our case, it is



class object : public shared_ptr_data { ... }

        

This allows to access members of the data structure directly, since at 

the same time they are members of the object type. We also typedef the 

type with testobject for brevity.



We mark our test with number 1. Previously, test group had a dummy test 

with the same number, but now, since we've defined our own version, it 

replaces the dummy test as more specialized one. It's how C++ template 

ordering works.



The test we've written always succeeds. Successful test returns with no 

exceptions. Unsuccessful one either throws an exception, or fails at 

fail() or ensure() methods (which anyway just throw the exception when failed).



First real test



Well, now we know enough to write the first real working test. This test 

will create shared_ptr instances and check their state. We will define a 

small structure (keepee) to use it as shared_ptr stored object type.



#include 

#include 



namespace tut

{

    struct shared_ptr_data

    {

        struct keepee{ int data; };

    };



    typedef test_group testgroup;

    typedef testgroup::object testobject;

    testgroup shared_ptr_testgroup("shared_ptr");



    /**

     * Checks default constructor.

     */

    template<>

    template<>

    void testobject::test<1>()

    {

        shared_ptr def;

        ensure("null",def.get() == 0);

    }

};



That's all! The first line creates shared_ptr. If constructor throws 

an exception, test will fail (exceptions, including '...', are catched 

by the TUT framework). If the first line succeeds, we must check 

whether the kept object is null one. To do this, we use test object 

member function ensure(), which throws std::logic_error with a given 

message if its second argument is not true. Finally, if destructor of 

shared_ptr fails with exception, TUT also will report this test as failed.



It's equally easy to write a test for the scenario where we expect to get 

an exception: let's consider our class should throw an exception if it 

has no stored object, and the operator -> is called.



/**

 * Checks operator -> throws instead of returning null.

 */

template<>

template<>

void testobject::test<2>()

{

    try

    {

        shared_ptr sp;

        sp->data = 0;

        fail("exception expected");

    }

    catch( const std::runtime_error& ex )

    {

        // ok

    }

}

        



Here we expect the std::runtime_error. If operator doesn't throw it, 

we'll force the test to fail using another member function: fail(). It 

just throws std::logic_error with a given message. If operator throws 

anything else, our test will fail too, since we intercept only 

std::runtime_error, and any other exception means the test has failed.



NB: our second test has number 2 in its name; it can, actually, be any 

in range 1..Max; the only requirement is not to write tests with the 

same numbers. If you did, compiler will force you to fix them anyway.



And finally, one more test to demonstrate how to use the 

ensure_equals template member function:



/**

 * Checks keepee counting.

 */

template<>

template<>

void testobject::test<3>()

{

    shared_ptr sp1(new keepee());

    shared_ptr sp2(sp1);

    ensure_equals("second copy at sp1",sp1.count(),2);

    ensure_equals("second copy at sp2",sp2.count(),2);

}



The test checks if the shared_ptr correctly counts references during 

copy construction. What's interesting here is the template member 

ensure_equals. It has an additional functionality comparing with similar 

call ensure("second_copy",sp1.count()==2); it uses operator == to check 

the equality of the two passed parameters and, what's more important, it 

uses std::stream to format the passed parameters into a human-readable 

message (smth like: "second copy: expected 2, got 1"). It means that 

ensure_equals cannot be used with the types that don't have operator <<; 

but for those having the operator it provides much more informational message.



In contrast to JUnit assertEquals, where the expected value goes before 

the actual one, ensure_equals() accepts the expected after the actual 

value. I believe it's more natural to read ensure_equals("msg", count, 5) 

as "ensure that count equals to 5" rather than JUnit's 

"assert that 5 is the value of the count".



Running tests



Tests are already written, but an attempt to run them will be unsuccessful. 

We need a few other bits to complete the test application.



First of all, we need a main() method, simply because it must be in all 

applications. Secondly, we need a test runner singleton. Remember I said 

each test group should register itself in singleton? So, we need that 

singleton. And, finally, we need a kind of a callback handler to visualize 

our test results.



The design of TUT doesn't strictly set a way the tests are visualized; 

instead, it provides an opportunity to get the test results by means of 

callbacks. Moreover it allows user to output the results in any format he 

prefers. Of course, there is a "reference implementation" in the 

example/subdirectory of the project.



Test runner singleton is defined in tut/tut.hpp, so all we need to activate it 

is to declare an object of the type tut::test_runner_singleton in the main 

module with a special name tut::runner.



Now, with the test_runner we can run tests. Singleton has method get() 

returning a reference to an instance of the test_runner class, which in 

turn has methods run_tests() to run all tests in all groups, 

run_tests(const std::string& groupname) to run all tests in a given group 

and run_test(const std::string& grp,int n) to run one test in the specified group.



// main.cpp

#include 



namespace tut

{

    test_runner_singleton runner;

}



int main()

{

    // run all tests in all groups

    runner.get().run_tests();



    // run all tests in group "shared_ptr"

    runner.get().run_tests("shared_ptr");



    // run test number 5 in group "shared_ptr"

    runner.get().run_test("shared_ptr",5);



    return 0;

}

  

It's up to user to handle command-line arguments or GUI messages and map those 

arguments/messages to actual calls to test runner. Again, as you see, TUT 

doesn't restrict user here.



But, the last question is still unanswered: how do we get our test results? 

The answer lies inside tut::callback interface. User shall create its subclass, 

and write up to three simple methods. He also can omit any method since they 

have default no-op implementation. Each corresponding method is called in the 

following cases:



    * a new test run started;

    * test finished;

    * test run finished.



Here is a minimal implementation:



class visualizator : public tut::callback

{

public:

  void run_started(){ }



  void test_completed(const tut::test_result& tr)

  {

      // ... show test result here ...

  }



  void run_completed(){ }

};        



The most important is the test_completed() method; its parameter has type 

test_result, and contains everything about the finished test, from its group 

name and number to the exception message, if any. Member result is an enum 

that contains status of the test: ok, fail or ex. Take a look at the 

examples/basic/main.cpp for more complete visualizator.



Visualizator should be passed to the test_runner before run. Knowing that, 

we are ready to write the final version of our main module:



// main.cpp

#include 



namespace tut

{

  test_runner_singleton runner;

}



class callback : public tut::callback

{

public:

  void run_started(){ std::cout << "\nbegin"; }



  void test_completed(const tut::test_result& tr)

  {

    std::cout << tr.test_pos << "=" << tr.result << std::flush;

  }



  void run_completed(){ std::cout << "\nend"; }

};



int main()

{

  callback clbk;

  runner.get().set_callback(&clbk);



  // run all tests in all groups

  runner.get().run_tests();

  return 0;

}



That's it. You are now ready to link and run our test application. Do it as often as possible; 

once a day is a definite must. I hope, TUT will help you to make your application more 

robust and relieve your testing pain. Feel free to send your questions, suggestions and 

critical opinions to me; I'll do my best to address them asap.