Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/galoisinc/oughta

A Haskell library for testing programs that output text
https://github.com/galoisinc/oughta

haskell haskell-library testing

Last synced: 4 months ago
JSON representation

A Haskell library for testing programs that output text

Awesome Lists containing this project

README 

          
# Oughta



Oughta is a Haskell library for testing programs that output text. The testing

paradigm essentially combines golden testing with `grep`.



More precisely, Oughta provides a DSL to build *recognizers* (i.e., parsers that

simply accept or reject an string). The inputs to Oughta are a string (usually,

the output of the program under test) and a separate, generally quite short

program written in the Oughta DSL.



The simplest DSL procedure is `check`, which checks that the output contains a

string. For example, the following test would pass:



Program output:

```

Hello, world!

```

DSL program:

```

check "Hello"

check "world"

```



Oughta draws inspiration from [DejaGNU] (which is used to test [GCC]), LLVM's

[FileCheck], and Rust's [compiletest].



[DejaGNU]: https://www.gnu.org/software/dejagnu/

[GCC]: https://gcc.gnu.org/

[FileCheck]: https://llvm.org/docs/CommandGuide/FileCheck.html

[compiletest]: https://rustc-dev-guide.rust-lang.org/tests/compiletest.html



## Example



Let's say that you have decided to write the first ever Haskell implementation

of a POSIX shell, `hsh`. Here's how to test it with Oughta.



If the input to the program under test is a file format that supports comments,

it is often convenient to embed Oughta DSL programs in the input itself. For

example, here's a test for `echo`:



```sh

# check "Hello, world!"

echo 'Hello, world!'

```



Using this strategy, each test case is a single file. You can use the

`directory` package to discover tests from a directory, and `tasty{,-hunit}` to

run tests:



```haskell

module Main (main) where



import Control.Monad (filterM, forM)

import Data.ByteString qualified as BS

import Oughta qualified

import System.Directory qualified as Dir

import Test.Tasty qualified as TT

import Test.Tasty.HUnit qualified as TTH



-- Code under test:

-- runScript :: ByteString -> (ByteString, ByteString)

-- Returns (stdout, stderr)

import Hsh (runScript)



test :: FilePath -> IO ()

test sh = do

  content <- BS.readFile sh

  (stdout, _stderr) <- runScript content

  let comment = "# "  -- shell script start-of-line comment

  let prog = Oughta.fromLineComments sh comment content

  Oughta.check' prog (Oughta.Output stdout)



main :: IO ()

main = do

  let dir = "test-data/"

  entries <- map (dir >) <$> Dir.listDirectory dir

  files <- filterM Dir.doesFileExist entries

  let shs = List.filter ((== ".sh") . FilePath.takeExtension) files

  let mkTest path = TTH.testCase path (test path)

  let tests = map mkTest shs

  TT.defaultMain (TT.testGroup "hsh tests" tests)

```



Now you can just toss `.sh` scripts into `test-data/` and have them picked up

as tests.



What if you wanted to test both stdout and stderr? You can use two different

styles of comments:

```haskell

test :: FilePath -> IO ()

test sh = do

  -- snip --

  let stdoutComment = "# STDOUT: "

  let prog = Oughta.fromLineComments sh stdoutComment content

  Oughta.check' prog (Oughta.Output stdout)



  let stderrComment = "# STDERR: "

  let prog' = Oughta.fromLineComments sh stderrComment content

  Oughta.check' prog' (Oughta.Output stderr)

```



Test cases would then look like so:



```sh

# STDOUT: check "Hello, stdout!"

echo 'Hello, stdout!'

# STDERR: check "Hello, stderr!"

echo 'Hello, stderr!' 1>&2

```



## Cookbook



This section demonstrates how to accomplish common tasks with the Oughta DSL.

The DSL is just [Lua][lua], extended with an API for easy parsing.



[Lua]: https://www.lua.org/



Writing tests in Lua offers considerable flexibility and expressive power.

However, with great power comes with great responsibility. Tests should be

as simple as possible, and some repetition should be accepted for the sake of

readability. It is often appropriate to just make a sequence of API calls with

literal strings as arguments.



Oughta is used to test itself. See the test suite for additional examples.



### Long matches



Match large blocks of text with Lua's multi-line string syntax:

```

some

multi-line

    text

```

```lua

check [[

some

multi-line

    text

]]

```



### Repetition



Match repetitive text using a variable:



```

HAL: Affirmative, Dave. I read you.

Dave: Open the pod bay doors, HAL.

HAL: I'm sorry, Dave. I'm afraid I can't do that.

```

```lua

name="Dave"

check("Affirmative, " .. name)

check("I'm sorry, " .. name)

```

or with a `for`-loop:

```

Step 1: Learn about Oughta

Step 2: Use it to test your project

Step 3: Enjoy!

```

```lua

for i=1,3 do

  check("Step %d".format(i))

end

```



### Checking for generated text



To check that some dynamically-generated text appears elsewhere in the output,

use the `string` library and the `text` variable:

```

Generating a random number...

Generated 37106428!

Printing 37106428...

```

```lua

check "Generated "

num=string.find(text, "^%d+")

check(string.format("Printing %d...", num))

```



The above example borders on *too much* logic for a test case. Use discretion

when writing tests!



## API reference



The Lua API is *stateful*. It keeps track of a global variable `text` that

is initialized to the output of the program under test. Various API functions

cause the API to seek forward in `text`. This is analogous to working file-like

objects in languages like C or Python. `text` should not be updated from Lua

code; such updates will be ignored by Oughta.



### High-level API



Checking functions:



- `check(s: String)`: Find `s` in `text`. Seek to after the end of `s`. Like

  LLVM FileCheck's `CHECK`, or [Expect's][expect] `expect`.

- `check_not(s: String)`: Assert that `s` is *not* in `text`. Do not seek. Like

  LLVM FileCheck's `CHECK-NOT`.

- `checkln(s: String)`: `checkln(s)` is equivalent to `check(s .. "\n")`.

- `here(s: String)`: Check that `text` beings with `s`. Seek to after the end

  of `s`.

- `hereln(String)`: `hereln(s)` is equivalent to `here(s .. "\n")`.



[expect]: https://core.tcl-lang.org/expect/index



Other utilities:



- `col() -> Int`: Get the current line number of `text` in the output.

- `file() -> String`: Get the file name of the test case.

- `line() -> Int`: Get the current line number of `text` in the output.

- `src_line(n: Int) -> Int`: Get the line number of the Lua code at stack level

  `n`.



### Low-level API



- `fail()`: Fail to match at this point in `text`.

- `match(n: Integer)`: Consume `n` bytes of `text`, treating them as a match.

- `seek(n: Integer)`: Seek forward `n` bytes in `text`.

- `reset(name: String, s: String)`: Set `text` to `s` and reset the program

  state (e.g., location tracking). Treat `name` as the file name for `s` in

  user-facing output.



## Motivation



The overall Oughta paradigm is a form of [data driven testing]. See that blog

post for considerable motivation and discussion.



[data driven testing]: https://matklad.github.io/2021/05/31/how-to-test.html#Data-Driven-Testing



In comparison to golden testing, Oughta-style tests are *coarser*. They only

check particular parts of the program's output. This can cause less churn in

the test suite when the program output changes in ways that are not relevant

to the properties being tested. The fineness of golden testing can force

devlopers to adopt [complex] [workarounds], these can sometimes be obviated by

Oughta-style testing.



[complex]: https://rustc-dev-guide.rust-lang.org/tests/ui.html#normalization

[workarounds]: https://github.com/GaloisInc/crucible/blob/dc0895f4435dc19a8cceee3272c9a508221bce51/crux-llvm/test/Test.hs#L226-L311



However, it is more complex. For example, it requires learning the Oughta DSL.

It can also cause unexpected successes, e.g., if the program output contains the

pattern being checked, but not in the proper place.



Why build a Haskell library when LLVM already provides their FileCheck tool?

There are a variety of reasons:



1. Ease of adoption: external runtime test dependencies are painful

2. Speed: use as a library avoids file I/O, spawning shells, etc.

3. Flexibility: FileCheck can be used on tools without command-line interfaces



## Versioning policy



Oughta conforms to the [Haskell Package Versioning Policy][pvp]. Both the Lua

and Haskell APIs are considered to be part of the public interface.



[pvp]: https://pvp.haskell.org/



## GHC support policy



We support at least three versions of GHC at a time. We are not aggressive about

dropping older versions, but will generally do so for versions outside of the

support window if either (1) maintaining that support would require significant

effort, such as significant numbers of C pre-processor `ifdef` sections or Cabal

`if` sections, or (2) the codebase could benefit significantly from features

that are only available on more recent versions of GHC. We try to support new

versions as soon as they are supported by the libraries that we depend on.