{"id":27644952,"url":"https://github.com/galoisinc/oughta","last_synced_at":"2025-10-16T05:26:24.971Z","repository":{"id":289323743,"uuid":"970341454","full_name":"GaloisInc/oughta","owner":"GaloisInc","description":"A Haskell library for testing programs that output text","archived":false,"fork":false,"pushed_at":"2025-04-23T17:16:59.000Z","size":26,"stargazers_count":1,"open_issues_count":5,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-24T00:59:12.243Z","etag":null,"topics":["haskell","haskell-library","testing"],"latest_commit_sha":null,"homepage":"https://galoisinc.github.io/oughta/","language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GaloisInc.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-21T21:37:21.000Z","updated_at":"2025-04-23T17:17:01.000Z","dependencies_parsed_at":"2025-04-22T18:42:59.093Z","dependency_job_id":null,"html_url":"https://github.com/GaloisInc/oughta","commit_stats":null,"previous_names":["galoisinc/oughta"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GaloisInc%2Foughta","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GaloisInc%2Foughta/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GaloisInc%2Foughta/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GaloisInc%2Foughta/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GaloisInc","download_url":"https://codeload.github.com/GaloisInc/oughta/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250540990,"owners_count":21447427,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["haskell","haskell-library","testing"],"created_at":"2025-04-24T00:59:15.311Z","updated_at":"2025-10-16T05:26:24.949Z","avatar_url":"https://github.com/GaloisInc.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Oughta\n\nOughta is a Haskell library for testing programs that output text. The testing\nparadigm essentially combines golden testing with `grep`.\n\nMore precisely, Oughta provides a DSL to build *recognizers* (i.e., parsers that\nsimply accept or reject an string). The inputs to Oughta are a string (usually,\nthe output of the program under test) and a separate, generally quite short\nprogram written in the Oughta DSL.\n\nThe simplest DSL procedure is `check`, which checks that the output contains a\nstring. For example, the following test would pass:\n\nProgram output:\n```\nHello, world!\n```\nDSL program:\n```\ncheck \"Hello\"\ncheck \"world\"\n```\n\nOughta draws inspiration from [DejaGNU] (which is used to test [GCC]), LLVM's\n[FileCheck], and Rust's [compiletest].\n\n[DejaGNU]: https://www.gnu.org/software/dejagnu/\n[GCC]: https://gcc.gnu.org/\n[FileCheck]: https://llvm.org/docs/CommandGuide/FileCheck.html\n[compiletest]: https://rustc-dev-guide.rust-lang.org/tests/compiletest.html\n\n## Example\n\nLet's say that you have decided to write the first ever Haskell implementation\nof a POSIX shell, `hsh`. Here's how to test it with Oughta.\n\nIf the input to the program under test is a file format that supports comments,\nit is often convenient to embed Oughta DSL programs in the input itself. For\nexample, here's a test for `echo`:\n\n```sh\n# check \"Hello, world!\"\necho 'Hello, world!'\n```\n\nUsing this strategy, each test case is a single file. You can use the\n`directory` package to discover tests from a directory, and `tasty{,-hunit}` to\nrun tests:\n\n```haskell\nmodule Main (main) where\n\nimport Control.Monad (filterM, forM)\nimport Data.ByteString qualified as BS\nimport Oughta qualified\nimport System.Directory qualified as Dir\nimport Test.Tasty qualified as TT\nimport Test.Tasty.HUnit qualified as TTH\n\n-- Code under test:\n-- runScript :: ByteString -\u003e (ByteString, ByteString)\n-- Returns (stdout, stderr)\nimport Hsh (runScript)\n\ntest :: FilePath -\u003e IO ()\ntest sh = do\n  content \u003c- BS.readFile sh\n  (stdout, _stderr) \u003c- runScript content\n  let comment = \"# \"  -- shell script start-of-line comment\n  let prog = Oughta.fromLineComments sh comment content\n  Oughta.check' prog (Oughta.Output stdout)\n\nmain :: IO ()\nmain = do\n  let dir = \"test-data/\"\n  entries \u003c- map (dir \u003c/\u003e) \u003c$\u003e Dir.listDirectory dir\n  files \u003c- filterM Dir.doesFileExist entries\n  let shs = List.filter ((== \".sh\") . FilePath.takeExtension) files\n  let mkTest path = TTH.testCase path (test path)\n  let tests = map mkTest shs\n  TT.defaultMain (TT.testGroup \"hsh tests\" tests)\n```\n\nNow you can just toss `.sh` scripts into `test-data/` and have them picked up\nas tests.\n\nWhat if you wanted to test both stdout and stderr? You can use two different\nstyles of comments:\n```haskell\ntest :: FilePath -\u003e IO ()\ntest sh = do\n  -- snip --\n  let stdoutComment = \"# STDOUT: \"\n  let prog = Oughta.fromLineComments sh stdoutComment content\n  Oughta.check' prog (Oughta.Output stdout)\n\n  let stderrComment = \"# STDERR: \"\n  let prog' = Oughta.fromLineComments sh stderrComment content\n  Oughta.check' prog' (Oughta.Output stderr)\n```\n\nTest cases would then look like so:\n\n```sh\n# STDOUT: check \"Hello, stdout!\"\necho 'Hello, stdout!'\n# STDERR: check \"Hello, stderr!\"\necho 'Hello, stderr!' 1\u003e\u00262\n```\n\n## Cookbook\n\nThis section demonstrates how to accomplish common tasks with the Oughta DSL.\nThe DSL is just [Lua][lua], extended with an API for easy parsing.\n\n[Lua]: https://www.lua.org/\n\nWriting tests in Lua offers considerable flexibility and expressive power.\nHowever, with great power comes with great responsibility. Tests should be\nas simple as possible, and some repetition should be accepted for the sake of\nreadability. It is often appropriate to just make a sequence of API calls with\nliteral strings as arguments.\n\nOughta is used to test itself. See the test suite for additional examples.\n\n### Long matches\n\nMatch large blocks of text with Lua's multi-line string syntax:\n```\nsome\nmulti-line\n    text\n```\n```lua\ncheck [[\nsome\nmulti-line\n    text\n]]\n```\n\n### Repetition\n\nMatch repetitive text using a variable:\n\n```\nHAL: Affirmative, Dave. I read you.\nDave: Open the pod bay doors, HAL.\nHAL: I'm sorry, Dave. I'm afraid I can't do that.\n```\n```lua\nname=\"Dave\"\ncheck(\"Affirmative, \" .. name)\ncheck(\"I'm sorry, \" .. name)\n```\nor with a `for`-loop:\n```\nStep 1: Learn about Oughta\nStep 2: Use it to test your project\nStep 3: Enjoy!\n```\n```lua\nfor i=1,3 do\n  check(\"Step %d\".format(i))\nend\n```\n\n### Checking for generated text\n\nTo check that some dynamically-generated text appears elsewhere in the output,\nuse the `string` library and the `text` variable:\n```\nGenerating a random number...\nGenerated 37106428!\nPrinting 37106428...\n```\n```lua\ncheck \"Generated \"\nnum=string.find(text, \"^%d+\")\ncheck(string.format(\"Printing %d...\", num))\n```\n\nThe above example borders on *too much* logic for a test case. Use discretion\nwhen writing tests!\n\n## API reference\n\nThe Lua API is *stateful*. It keeps track of a global variable `text` that\nis initialized to the output of the program under test. Various API functions\ncause the API to seek forward in `text`. This is analogous to working file-like\nobjects in languages like C or Python. `text` should not be updated from Lua\ncode; such updates will be ignored by Oughta.\n\n### High-level API\n\nChecking functions:\n\n- `check(s: String)`: Find `s` in `text`. Seek to after the end of `s`. Like\n  LLVM FileCheck's `CHECK`, or [Expect's][expect] `expect`.\n- `check_not(s: String)`: Assert that `s` is *not* in `text`. Do not seek. Like\n  LLVM FileCheck's `CHECK-NOT`.\n- `checkln(s: String)`: `checkln(s)` is equivalent to `check(s .. \"\\n\")`.\n- `here(s: String)`: Check that `text` beings with `s`. Seek to after the end\n  of `s`.\n- `hereln(String)`: `hereln(s)` is equivalent to `here(s .. \"\\n\")`.\n\n[expect]: https://core.tcl-lang.org/expect/index\n\nOther utilities:\n\n- `col() -\u003e Int`: Get the current line number of `text` in the output.\n- `file() -\u003e String`: Get the file name of the test case.\n- `line() -\u003e Int`: Get the current line number of `text` in the output.\n- `src_line(n: Int) -\u003e Int`: Get the line number of the Lua code at stack level\n  `n`.\n\n### Low-level API\n\n- `fail()`: Fail to match at this point in `text`.\n- `match(n: Integer)`: Consume `n` bytes of `text`, treating them as a match.\n- `seek(n: Integer)`: Seek forward `n` bytes in `text`.\n- `reset(name: String, s: String)`: Set `text` to `s` and reset the program\n  state (e.g., location tracking). Treat `name` as the file name for `s` in\n  user-facing output.\n\n## Motivation\n\nThe overall Oughta paradigm is a form of [data driven testing]. See that blog\npost for considerable motivation and discussion.\n\n[data driven testing]: https://matklad.github.io/2021/05/31/how-to-test.html#Data-Driven-Testing\n\nIn comparison to golden testing, Oughta-style tests are *coarser*. They only\ncheck particular parts of the program's output. This can cause less churn in\nthe test suite when the program output changes in ways that are not relevant\nto the properties being tested. The fineness of golden testing can force\ndevlopers to adopt [complex] [workarounds], these can sometimes be obviated by\nOughta-style testing.\n\n[complex]: https://rustc-dev-guide.rust-lang.org/tests/ui.html#normalization\n[workarounds]: https://github.com/GaloisInc/crucible/blob/dc0895f4435dc19a8cceee3272c9a508221bce51/crux-llvm/test/Test.hs#L226-L311\n\nHowever, it is more complex. For example, it requires learning the Oughta DSL.\nIt can also cause unexpected successes, e.g., if the program output contains the\npattern being checked, but not in the proper place.\n\nWhy build a Haskell library when LLVM already provides their FileCheck tool?\nThere are a variety of reasons:\n\n1. Ease of adoption: external runtime test dependencies are painful\n2. Speed: use as a library avoids file I/O, spawning shells, etc.\n3. Flexibility: FileCheck can be used on tools without command-line interfaces\n\n## Versioning policy\n\nOughta conforms to the [Haskell Package Versioning Policy][pvp]. Both the Lua\nand Haskell APIs are considered to be part of the public interface.\n\n[pvp]: https://pvp.haskell.org/\n\n## GHC support policy\n\nWe support at least three versions of GHC at a time. We are not aggressive about\ndropping older versions, but will generally do so for versions outside of the\nsupport window if either (1) maintaining that support would require significant\neffort, such as significant numbers of C pre-processor `ifdef` sections or Cabal\n`if` sections, or (2) the codebase could benefit significantly from features\nthat are only available on more recent versions of GHC. We try to support new\nversions as soon as they are supported by the libraries that we depend on.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgaloisinc%2Foughta","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgaloisinc%2Foughta","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgaloisinc%2Foughta/lists"}