Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/mirage/alcotest

A lightweight and colourful test framework
https://github.com/mirage/alcotest

ocaml unit-testing

Last synced: 22 days ago
JSON representation

A lightweight and colourful test framework

Awesome Lists containing this project

README 

        


A lightweight and colourful test framework.






Alcotest exposes a simple interface to perform unit tests. It exposes

a simple `TESTABLE` module type, a `check` function to assert test

predicates and a `run` function to perform a list of `unit -> unit`

test callbacks.



Alcotest provides a quiet and colorful output where only faulty runs

are fully displayed at the end of the run (with the full logs ready to

inspect), with a simple (yet expressive) query language to select the

tests to run. See [the manpage](./alcotest-help.txt) for details.



The API documentation can be found [here][docs]. For information on

contributing to Alcotest, see [CONTRIBUTING.md](./CONTRIBUTING.md).



[![OCaml-CI Build Status](https://img.shields.io/endpoint?url=https://ci.ocamllabs.io/badge/mirage/alcotest/main&logo=ocaml)](https://ci.ocamllabs.io/github/mirage/alcotest)

[![Alcotest Documentation](https://img.shields.io/badge/doc-online-blue.svg)][docs]



[docs]: https://mirage.github.io/alcotest/alcotest/Alcotest/index.html






### Examples



A simple example (taken from `examples/simple.ml`):






Generated by the following test suite specification:



```ocaml

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)



(* A module with functions to test *)

module To_test = struct

  let lowercase = String.lowercase_ascii

  let capitalize = String.capitalize_ascii

  let str_concat = String.concat ""

  let list_concat = List.append

end



(* The tests *)

let test_lowercase () =

  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")



let test_capitalize () =

  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")



let test_str_concat () =

  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])



let test_list_concat () =

  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])



(* Run it *)

let () =

  let open Alcotest in

  run "Utils" [

      "string-case", [

          test_case "Lower case"     `Quick test_lowercase;

          test_case "Capitalization" `Quick test_capitalize;

        ];

      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];

      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];

    ]

```



The result is a self-contained binary which displays the test

results. Use `dune exec examples/simple.exe -- --help` to see the

runtime options.



Here's an example of a of failing test suite:






By default, only the first failing test log is printed to the console

(and all test logs are captured on disk). Pass `--show-errors` to

print all error messages.



### Using Alcotest with opam and Dune



Add `(alcotest :with-test)` to the `depends` stanza of your

`dune-project` file, or `"alcotest" {with-test}` to your opam file.

Use the `with-test` [package variable][with-test] to declare your

tests opam dependencies. Call opam to install them:



```sh-session

$ opam install --deps-only --with-test .

```



You can then [declare your test][tests] and link with Alcotest: `(test

(libraries alcotest …) …)`, and run your tests:



```sh-session

$ dune runtest

```



[with-test]: https://opam.ocaml.org/doc/Manual.html#pkgvar-with-test

[tests]: https://dune.readthedocs.io/en/stable/tests.html#custom-tests



### Selecting tests to execute



You can filter which tests to run by supplying a regular expression

matching the names of the tests to execute, or by passing a regular

expression _and_ a comma-separated list of test numbers (or ranges of

test numbers, e.g. `2,4..9`):



```sh-session

$ ./simple.native test '.*concat*'

Testing Utils.

[SKIP]     string-case            0   Lower case.

[SKIP]     string-case            1   Capitalization.

[OK]       string-concat          0   String mashing.

[OK]       list-concat            0   List mashing.

The full test results are available in `_build/_tests`.

Test Successful in 0.000s. 2 tests run.



$ ./simple.native test 'string-case' '1..3'

Testing Utils.

[SKIP]     string-case            0   Lower case.

[OK]       string-case            1   Capitalization.

[SKIP]     string-concat          0   String mashing.

[SKIP]     list-concat            0   List mashing.

The full test results are available in `_build/_tests`.

Test Successful in 0.000s. 1 test run.

```



Note that you cannot filter by test case name (i.e. `Lower case` or

`Capitalization`), you must filter by test name & number instead.



See the [examples][] directory for more examples.



[examples]: https://github.com/mirage/alcotest/tree/main/examples



### Quick and Slow tests



In general you should use `` `Quick`` tests: tests that are ran on any

invocations of the test suite. You should only use `` `Slow`` tests

for stress tests that are ran only on occasion (typically before a

release or after a major change). These slow tests can be suppressed

by passing the `-q` flag on the command line, e.g.:



```sh-session

$ ./test.exe -q # run only the quick tests

$ ./test.exe    # run quick and slow tests

```



### Passing custom options to the tests



In most cases, the base tests are `unit -> unit` functions. However,

it is also possible to pass an extra option to all the test functions

by using `'a -> unit`, where `'a` is the type of the extra parameter.



In order to do this, you need to specify how this extra parameter is

read on the command-line, by providing a [Cmdliner term][] for

command-line arguments which explains how to parse and serialize

values of type `'a` (*note:* do not use positional arguments, only

optional arguments are supported).



[Cmdliner term]: https://erratique.ch/software/cmdliner/doc/Cmdliner/Term/index.html



For instance:



```ocaml

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42



let int =

  let doc = "What is your preferred number?" in

  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")



let () =

  Alcotest.run_with_args "foo" int [

    "all", ["nice", `Quick, test_nice]

  ]

```



Will generate `test.exe` such that:



```sh-session

$ test.exe test

test.exe: required option -n is missing



$ test.exe test -n 42

Testing foo.

[OK]                all          0   int.

```



### Lwt



Alcotest provides an `Alcotest_lwt` module that you could use to wrap

Lwt test cases. The basic idea is that instead of providing a test

function in the form `unit -> unit`, you provide one with the type

`unit -> unit Lwt.t` and alcotest-lwt calls `Lwt_main.run` for you.



However, there are a couple of extra features:



- If an async exception occurs, it will cancel your test case for you

  and fail it (rather than exiting the process).



- You get given a switch, which will be turned off when the test case

  finishes (or fails). You can use that to free up any resources.



For instance:



```ocaml

let free () = print_endline "freeing all resources"; Lwt.return ()



let test_lwt switch () =

  Lwt_switch.add_hook (Some switch) free;

  Lwt.async (fun () -> failwith "All is broken");

  Lwt_unix.sleep 10.



let () =

  Lwt_main.run @@ Alcotest_lwt.run "foo" [

    "all", [

      Alcotest_lwt.test_case "one" `Quick test_lwt

    ]

  ]

```



Will generate:



```sh-session

$ test.exe

Testing foo.

[ERROR]             all          0   one.

-- all.000 [one.] Failed --

in _build/_tests/all.000.output:

freeing all resources

[failure] All is broken

```



### Comparison with other testing frameworks



The README is pretty clear about that:



> Alcotest is a lightweight and colourful test framework.



Alcotest is the only testing framework using colors!



More seriously, Alcotest is similar to [ounit][] but it fixes a few of

the problems found in that library:



- Alcotest has a nicer output, it is easier to see what failed and

  what succeeded and to read the log outputs of the failed tests;



- Alcotest uses combinators to define pretty-printers and comparators

  between the things to test.



Other nice tools doing different kind of testing also exist:



- [qcheck][] does random generation and property testing (e.g. Quick

  Check);



- [crowbar][] and [bun][] are similar to qcheck, but use

  compiler-directed randomness, i.e. they take advantage of the AFL

  support the OCaml compiler;



- [ppx_inline_tests][] allows to write tests in the same file as

  your source-code; they will be run only in a special mode of

  compilation.



[ounit]: https://github.com/gildor478/ounit

[qcheck]: https://github.com/c-cube/qcheck

[crowbar]: https://github.com/stedolan/crowbar

[bun]: https://github.com/yomimono/ocaml-bun

[ppx_inline_tests]: https://github.com/janestreet/ppx_inline_test