Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/lindig/quest

A tool for testing C compilers automatically
https://github.com/lindig/quest

Last synced: 4 days ago
JSON representation

A tool for testing C compilers automatically

Awesome Lists containing this project

README 

        

# Quest - A tool for testing C compilers



This is Quest, a tool that generates C code for testing C compilers. A test

is run by compiling it with the compiler under test and subsequently

executing the binary that the compiler produced. A compiler bug manifests

itself by an assertion failure during the execution of the binary.



The code generated tests one specific part of a C compiler: how it

implements passing values to a function by function call and returning a

value from it using the return statement. Stated simpler, the generated

code by Quests checks that a value passed to a function is indeed received

unaltered from the caller and likewise when a valuereturned to the caller.

Conceptually, the code generated by Quest looks like this:



    static int   i = 1;

    static char  c = 'c'; 

    static float r = 3.14; 



    static float f(int ii, char cc)

    {

        assert(ii == i);

        assert(cc == c);

        return r;

    }



    static void test(void)

    {

        float rr;

        rr = f(i ,c);

        assert(rr == r); 

    }



    int main(int argc, char **argv)

    {

        test();

    }



By compiling and runnung the code with a C compiler we can test that values

are passed and returned to and from a function as expected:



    $ quest > testcase.c

    $ gcc -o testcase testcase.c

    $ ./testcase



If the compiler fails to generate correct code, executing the code will

lead to an assertion failure that would highlight the problem. Quest

generates tests in a random yet controlled way. Generating a test case,

compiling and running it can be done in a loop until a bug is found.



# The Issue



Why is compiling function calls even an issue? What looks simple on the

surface of C source code is quite complicated inside a compiler. How

values are supposed to be passed between functions is governed by a calling

convention. Such a convention specifies how values are passed in registers

and on the stack using a specific alignment. C calling conventions in

particular are complicted when structured values are passed, like

structures, or var args.  As a consequence, even mature compilers exhibit

bugs and Quest helps to find them. The beauty is that we don't have to know

the details of a calling convention in order to test it: generating test

cases and executing them is enough. 



Directory doc/ contains a [paper](doc/lindig-aadebug-2005.pdf) whith more

details:



    Christian Lindig. Random  Testing of C Calling Conventions. In

    Jong Deok Choi and Raimondas Lencevicius, editors, Sixth

    Internation Symposium on  Automated and Analysis-Driven

    Debugging", ACM Press, Monterey, CA,  USA, September 2005. 



Directory [bugs](bugs) contains a few compiler bugs that were uncovered

with Quest in the past. This provides also examples for code generated

by Quest.



# Installation from Opam 



Quest is available from OCaml's package manager

[Opam](http://opam.ocaml.org):



    $ opam install quest



This will install the quest binary. 



# Installation from GitHub



Quest is implemened in OCaml as a literate program. It should build on Unix

systems without a problem and does not depend on libraries outside the

OCaml standard libraries and its tools.



    $ opam install lipsum

    $ make PREFIX=$HOME     # see the Makefile



# Quick Start



To be written. See [doc/quest.pod](doc/quest.pod) for now.



# Documentation



* The manual page quest.pod in doc/ explains Quest in detail. 



* If you have a Quest binary you can view the manual page also with

    

    quest -man



  which emits it to stdout. A short summary of options is available

  with:

   

    quest -help



* The Lua 2.5 reference manual in doc/ introduces the Lua scripting

  language. It is only required to define new generators beyond the

  existing ones.



# Discussion



Recently testing calling concentions was discussed in a thread on Hacker

News:



* [Calling conventions are hard – fuzz them](https://news.ycombinator.com/item?id=10246697)



# Quest Features



* Quest pretty prints C code to keep it readable.



* Quest can split test cases into two files such that they may be

  compiled by different compilers to check their interoperability.



* Neither Quest itself, nor the generated code requires any special

  run-time libraries or configuration files. 



* Quest is developed on Mach OS X but contains no

  system-specific source code. However, the build process needs to be

  adapted for other systems.



* Quest is controlled by an embedded Lua interpreter. This makes it easy to

  tailor the generation of test cases.



# Author



Christian Lindig . 

Please report bugs on GitHub https://github.com/lindig/quest



# Opam

quest - generates C code for testing a C compiler's calling convention



Quest generates C code for testing C compilers. A test is run by compiling

the generated code it with the compiler under test and subsequently

executing the binary that the compiler produced. A compiler bug manifests

itself by an assertion failure during the execution of the binary. This

leads to automatic testing: generate and execute tests until an assertion

failure is found.



The code generated by Quest tests one specific part of a C compiler: how it

implements passing values to a function by function call and returning a

value from it using the return statement. Stated simpler, the generated

code by Quests checks that a value passed to a function is indeed received

unaltered from the caller and likewise when a valuereturned to the caller.