Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/cryptocode/marble

A metamorphic testing library for Zig
https://github.com/cryptocode/marble

metamorphic-testing testing testing-tool testing-tools zig zig-library zig-package

Last synced: 3 months ago
JSON representation

A metamorphic testing library for Zig

Awesome Lists containing this project

README 

        



Marble is a [metamorphic testing](https://en.wikipedia.org/wiki/Metamorphic_testing) library for Zig.



This library tracks Zig master and was last tested on `0.12.0-dev.4472+1c7798a3c`



Metamorphic testing is a powerful technique that provides additional test coverage by applying a number of transformations to test input, and then checking if certain relations still hold between the outputs. Marble will automatically run through all possible combinations of these transformations.



Here's a [great introduction by](https://www.cockroachlabs.com/blog/metamorphic-testing-the-database/) Cockroach Labs. I highly recommend reading before using this library.



The repository contains a few [test examples](https://github.com/cryptocode/marble/blob/main/src/example_tests.zig)



## Resources

* [Hillel Wayne's blog post on Metamorphic Testing (highly recommended)](https://www.hillelwayne.com/post/metamorphic-testing/)

* [Test your Machine Learning Algorithm with Metamorphic Testing](https://medium.com/trustableai/testing-ai-with-metamorphic-testing-61d690001f5c)

* [Original paper by T.Y. Chen et al](https://www.cse.ust.hk/~scc/publ/CS98-01-metamorphictesting.pdf)

* [Case study T.Y. Chen et al](http://grise.upm.es/rearviewmirror/conferencias/jiisic04/Papers/25.pdf)

* [Metamorphic Testing and Beyond T.Y. Chen et al](https://www.cs.hku.hk/data/techreps/document/TR-2003-06.pdf)

* [Survey on Metamorphic Testing](http://www.cs.ecu.edu/reu/reufiles/read/metamorphicTesting-16.pdf)

* [Performance Metamorphic Testing](http://www.lsi.us.es/~jtroya/publications/NIER17_at_ICSE17.pdf)

* [Experiences from Three Fuzzer Tools](https://johnwickerson.github.io/papers/dreamingup_MET21.pdf)

* [Monarch, a similar library for Rust](https://github.com/zmitchell/monarch/blob/master/src/runner.rs)



## Building



To build and run test examples:



```bash

zig build

zig build test

```



## Importing the library

Add Marble as a Zig package in your build file, or simply import it directly after vendoring/adding a submodule:



```zig

const marble = @import("marble/main.zig");

```



## Writing tests



A metamorphic Zig test looks something like this:



```zig

const SinusTest = struct {

    const tolerance = std.math.epsilon(f64) * 20;



    /// This test has a single value, but you could also design the test to take an

    /// array as input. The transformations, check and execute functions would then

    /// loop through them all. Alternatively, the test can be run multiple times

    /// with different inputs.

    value: f64,



    /// The mathematical property "sin(x) = sin(Ο€ βˆ’ x)" must hold

    pub fn transformPi(self: *SinusTest) void {

        self.value = std.math.pi - self.value;

    }



    /// Adding half the epsilon must still cause the relation to hold given the tolerance

    pub fn transformEpsilon(self: *SinusTest) void {

        self.value = self.value + std.math.epsilon(f64) / 2.0;

    }



    /// A metamorphic relation is a relation between outputs in different executions.

    /// This relation must hold after every execution of transformation combinations.

    pub fn check(_: *SinusTest, original_output: f64, transformed_output: f64) bool {

        return std.math.approxEqAbs(f64, original_output, transformed_output, tolerance);

    }



    /// Called initially to compute the baseline output, and after every transformation combination

    pub fn execute(self: *SinusTest) f64 {

        return std.math.sin(self.value);

    }

};



test "sinus" {

    var i: f64 = 1;

    while (i < 100) : (i += 1) {

        var t = SinusTest{ .value = i };

        try std.testing.expect(try marble.run(SinusTest, &t, .{}));

    }

}

```



You will get compile time errors if the requirements for a metamorphic test are not met.



In short, you must provide a `value` field, a `check` function, an `execute` function and one or more `transform...` functions.



### Writing transformations

Add one or more functions starting with `transform...`



Marble will execute all combinations of the transformation functions. After every

combination, `execute` is called followed by `check`.



Transformations should change the `value` property - Marble will remember what it was originally. The transformations must be such that `check`

succeeds. That is, the relations between the inital output and the transformed output must still hold.



### Checking if relations still hold

You must provide a `check` function to see if one or more relations hold, and return true if so. If false is returned, the test fails with a print-out of the current transformation-combination.



Relation checks may be conditional; check out the tests for examples on how this works.



### Executing

You must provide an `execute` function that computes a result based on the current value. The simplest form will simply return the current value, but you can

do any arbitrary operation here. This function is called before any transformations to form a baseline. This baseline is passed as the first argument to `check`



### Optional before/after calls



Before and after the test, and every combination, `before(...)` and `after(...)` is called if present. This is useful to reset state, initialize test cases, and perform clean-up.



### What happens during a test run?



Using the example above, the following pseudocode runs will be performed:



```

baseline = execute()



// First combination

transformPi()

out = execute()

check(baseline, out)



// Second combination

transformEpsilon()

out = execute()

check(baseline, out)



// Third combination

transformPi()

transformEpsilon()

out = execute()

check(baseline, out)

```



### Configuring runs



The `run` function takes a `RunConfiguration`:



```zig

/// If set to true, only run each transformation once separately

skip_combinations: bool = false,



/// If true, print detailed information during the run

verbose: bool = false,

```



### Error reporting



If a test fails, the current combination being executed is printed. For instance, the following tells us that the combination of `transformAdditionalTerm` and `transformCase` caused the metamorphic relation to fail:



```

Test [2/2] test "query"... Test case failed with transformation(s):

  >> transformAdditionalTerm

  >> transformCase

```



### Terminology



* Source test case output: The output produced by `execute()` on the initial input. This is also known as the baseline.

* Derived test case output: The output produced by `execute()` after applying a specific combination of transformations.

* Metamorphic relation: A property that must hold when considering a source test case and a derived test case.