{"id":13726200,"url":"https://github.com/janestreet/ppx_expect","last_synced_at":"2025-04-04T13:08:15.272Z","repository":{"id":41322011,"uuid":"49583340","full_name":"janestreet/ppx_expect","owner":"janestreet","description":"Cram like framework for OCaml","archived":false,"fork":false,"pushed_at":"2024-12-10T15:25:51.000Z","size":439,"stargazers_count":156,"open_issues_count":16,"forks_count":28,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-28T12:03:51.507Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"OCaml","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/janestreet.png","metadata":{"files":{"readme":"README.mdx","changelog":"CHANGES.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-01-13T15:42:08.000Z","updated_at":"2025-03-12T17:06:27.000Z","dependencies_parsed_at":"2024-02-01T20:07:37.517Z","dependency_job_id":"1d11a699-ed18-4e27-9dc3-194313179b7c","html_url":"https://github.com/janestreet/ppx_expect","commit_stats":null,"previous_names":[],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janestreet%2Fppx_expect","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janestreet%2Fppx_expect/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janestreet%2Fppx_expect/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janestreet%2Fppx_expect/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/janestreet","download_url":"https://codeload.github.com/janestreet/ppx_expect/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247176628,"owners_count":20896502,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T01:02:55.535Z","updated_at":"2025-04-04T13:08:15.237Z","avatar_url":"https://github.com/janestreet.png","language":"OCaml","funding_links":[],"categories":["Testing","OCaml","Software"],"sub_categories":[],"readme":"expect-test - a Cram-like framework for OCaml\n=============================================\n\n\n# Introduction\n\nExpect-test is a framework for writing tests in OCaml, similar to\n[Cram](https://bitheap.org/cram/).\n\nExpect-tests mimic the (now less idiomatic)\n[inline test](https://github.com/janestreet/ppx_inline_test)\nframework in providing a\n`let%expect_test` construct.\n\nThe body of an expect-test can contain output-generating code, interleaved with\n`[%expect]` extension expressions to denote the expected output.\n\nWhen run, expect-tests pass iff the output [_matches_](#matching-behavior) the expected\noutput. If a test fails, the `inline_tests_runner` outputs a diff and creates a file with\nthe suffix \".corrected\" containing the actual output.\n\nHere is an example expect-test in `foo.ml`:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/foo.ml,part=addition --\u003e\n```ocaml\nopen! Core\n\nlet%expect_test \"addition\" =\n  printf \"%d\" (1 + 2);\n  [%expect {| 4 |}]\n;;\n```\n\nWhen the test runs, the `inline_tests_runner` creates `foo.ml.corrected` with contents:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/foo.ml.corrected.expected,part=addition --\u003e\n```ocaml\nopen! Core\n\nlet%expect_test \"addition\" =\n  printf \"%d\" (1 + 2);\n  [%expect {| 3 |}]\n;;\n```\n\n`inline_tests_runner` also outputs:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/test-output.expected --\u003e\n```\n------ foo.ml\n++++++ foo.ml.corrected\nFile \"foo.ml\", line 6, characters 0-1:\n |open! Core\n |\n |let%expect_test \"addition\" =\n |  printf \"%d\" (1 + 2);\n-|  [%expect {| 4 |}]\n+|  [%expect {| 3 |}]\n |;;\n |\n```\n\nDiffs are shown in color if the `-use-color` flag is passed to the inline test runner\nexecutable.\n\n# Common usage\n\nEach `[%expect]` block matches all the output generated since the previous `[%expect]`\nblock (or the beginning of the test). In this way, when multiple `[%expect]` blocks are\ninterleaved with test code, they can help show which part of the test produced which\noutput.\n\nThe following test:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml,part=interleaved --\u003e\n```ocaml\nlet%expect_test \"interleaved\" =\n  let l = [ \"a\"; \"b\"; \"c\" ] in\n  printf \"A list [l]\\n\";\n  printf \"It has length %d\\n\" (List.length l);\n  [%expect {| A list [l] |}];\n  List.iter l ~f:print_string;\n  [%expect\n    {|\n    It has length 3\n    abc\n    |}]\n;;\n```\n\nis rewritten as\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=interleaved --\u003e\n```ocaml\nlet%expect_test \"interleaved\" =\n  let l = [ \"a\"; \"b\"; \"c\" ] in\n  printf \"A list [l]\\n\";\n  printf \"It has length %d\\n\" (List.length l);\n  [%expect\n    {|\n    A list [l]\n    It has length 3\n    |}];\n  List.iter l ~f:print_string;\n  [%expect {| abc |}]\n;;\n```\n\nWhen there is \"trailing\" output at the end of a `let%expect_test` (output that has yet to\nbe matched by some `[%expect]` block), a new `[%expect]` block is appended to the test\nwith the trailing output:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml,part=trailing --\u003e\n```ocaml\nlet%expect_test \"trailing output\" =\n  print_endline \"Hello\";\n  [%expect {| Hello |}];\n  print_endline \"world\"\n;;\n```\n\nbecomes:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=trailing --\u003e\n```ocaml\nlet%expect_test \"trailing output\" =\n  print_endline \"Hello\";\n  [%expect {| Hello |}];\n  print_endline \"world\";\n  [%expect {| world |}]\n;;\n```\n\n# Matching behavior\n\nYou might have noticed that the contents of the `[%expect]` blocks are not _exactly_ the\nprogram output; in some of the examples above, they contain a different number of leading\nand trailing newlines, and are indented to match the code indentation. We say the contents\nof a block `[%expect str]` (where `str` is some string literal) _match_ the output at that\nblock if the output, after we format it to standardize indentation and other whitespace,\nis identical to the contents of `str`\nafter it has been similarly formatted\n.\n\nThe formatting applied depends on the type of delimiter used in `str` (i.e. whether it a\n`\"quoted string\"` or a `{xxx| delimited string |xxx}`). To summarize:\n\n* Output containing only whitespace is formatted as `[%expect {| |}]` or `[%expect \"\"]`.\n* Output where only one line contains non-whitespace characters is formatted onto a single\n  line, as `[%expect {| output |}]` or `[%expect \"output\"]`.\n* Output where multiple lines contain non-whitespace characters is formatted so that:\n  - There is no trailing whitespace on lines with content.\n  - The relative indentation of the lines is preserved.\n  - In `{| delimited strings |}`, the least-indented line with content (the \"left margin\"\n    of the output) is aligned to be two spaces past the indentation of the `[%expect]`\n    block.\n  - In `\"quoted string\"`, the least-indented line is indented by exactly one space (this\n    plays the nicest with `ocamlformat`'s existing decisions about how to format string\n    literals).\n  - There is one empty line before and one empty line after the contents.\n\n\nHere is an example containing several cases of output that are subject to distinct\nformatting rules and how they appear in `[%expect]` and `[%expect_exact]` blocks:\n\n\u003cdetails\u003e\n\u003csummary\u003eExpand examples\u003c/summary\u003e\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=matching --\u003e\n```ocaml\nlet%expect_test \"matching behavior --- no content\" =\n  printf \"     \";\n  [%expect {| |}];\n  printf \"     \";\n  [%expect \"\"];\n  printf \"     \";\n  [%expect_exact {|     |}];\n  printf \"     \";\n  [%expect_exact \"     \"]\n;;\n\nlet%expect_test \"matching behavior --- one line of content\" =\n  printf \"\\n   This is one line\\n\\n\";\n  [%expect {| This is one line |}];\n  printf \"\\n   This is one line\\n\\n\";\n  [%expect \"This is one line\"];\n  printf \"\\n   This is one line\\n\\n\";\n  [%expect_exact\n    {|\n   This is one line\n\n|}];\n  printf \"\\n   This is one line\\n\\n\";\n  [%expect_exact \"\\n   This is one line\\n\\n\"]\n;;\n\nlet%expect_test \"matching behavior --- multiple lines of content\" =\n  printf\n    {|\nOnce upon a midnight dreary,\n  while I pondered, weak and weary,\nOver many a quaint and curious\n  volume of forgotten lore |};\n  [%expect\n    {|\n    Once upon a midnight dreary,\n      while I pondered, weak and weary,\n    Over many a quaint and curious\n      volume of forgotten lore\n    |}];\n  printf\n    {|\nOnce upon a midnight dreary,\n  while I pondered, weak and weary,\nOver many a quaint and curious\n  volume of forgotten lore |};\n  [%expect\n    \" \\n\\\n    \\ Once upon a midnight dreary,\\n\\\n    \\   while I pondered, weak and weary,\\n\\\n    \\ Over many a quaint and curious\\n\\\n    \\   volume of forgotten lore\\n\\\n    \\ \"];\n  printf\n    {|\nOnce upon a midnight dreary,\n  while I pondered, weak and weary,\nOver many a quaint and curious\n  volume of forgotten lore |};\n  [%expect_exact\n    {|\nOnce upon a midnight dreary,\n  while I pondered, weak and weary,\nOver many a quaint and curious\n  volume of forgotten lore |}];\n  printf\n    {|\nOnce upon a midnight dreary,\n  while I pondered, weak and weary,\nOver many a quaint and curious\n  volume of forgotten lore |};\n  [%expect_exact\n    \"\\n\\\n     Once upon a midnight dreary,\\n\\\n    \\  while I pondered, weak and weary,\\n\\\n     Over many a quaint and curious\\n\\\n    \\  volume of forgotten lore \"]\n;;\n```\n\u003c/details\u003e\n\nExpect-test is by default permissive about this formatting, so that a\n`[%expect]` block that is correct modulo formatting is\naccepted. However, passing `-expect-test-strict-indentation=true` to\nthe ppx driver makes the test runner issue corrections for blocks that\ndo not satisfy the indentation rules.\nFor example, the following:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml,part=bad-format --\u003e\n```ocaml\nlet%expect_test \"bad formatting\" =\n  printf \"a\\n    b\";\n  [%expect\n    {|\na\n    b |}]\n;;\n```\nis corrected to:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=bad-format --\u003e\n```ocaml\nlet%expect_test \"bad formatting\" =\n  printf \"a\\n    b\";\n  [%expect\n    {|\n    a\n        b\n    |}]\n;;\n```\n\n(to add the required indentation and trailing newline)\n\n\n# Reachability\n\n## Expects reached from multiple places\n\nA `[%expect]` extension can be encountered multiple times if it is in e.g. a functor or a\nfunction:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml,part=function --\u003e\n```ocaml\nlet%expect_test \"function\" =\n  let f output =\n    print_string output;\n    [%expect {| hello world |}]\n  in\n  f \"hello world\";\n  f \"hello world\"\n;;\n```\n\nThe test passes if the `[%expect]` block matches the output each time it is encountered,\nas described in the section on [matching behavior](#matching-behavior).\n\nIf the outputs are not consistent, then the corrected file contains a report of all of the\noutputs that were captured, in the order that they were captured at runtime.\n\nFor example, calling `f` in the snippet above with inconsistent arguments will produce:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=broken-function --\u003e\n```ocaml\nlet%expect_test \"function\" =\n  let f output =\n    print_string output;\n    [%expect\n      {|\n      (* expect_test: Test ran multiple times with different test outputs *)\n      ============================ Output 1 / 4 ============================\n      hello world\n      ============================ Output 2 / 4 ============================\n      goodbye world\n      ============================ Output 3 / 4 ============================\n      once upon\n      a midnight dreary\n      ============================ Output 4 / 4 ============================\n      hello world\n      |}]\n  in\n  f \"hello world\";\n  f \"goodbye world\";\n  f \"once upon\\na midnight dreary\";\n  f \"hello world\"\n;;\n```\n\n\n## Unreached expects\n\nEvery `[%expect]` and `[%expect_exact]` block in a `let%expect_test` must be reached at\nleast once if that test is ever run. Failure for control flow to reach a block is _not_\ntreated like recording empty output at a block. The extension expression\n`[%expect.unreachable]` is used to indicate that some part of an expect test shouldn't be\nreached; if control flow reaches that point anyway, the corrected file replaces the\n`[%expect.unreachable]` with a plain old expect containing the output collected until that\npoint. Conversely, if control flow never reaches some `[%expect]` block, that block is\nturned into a `[%expect.unreachable]`. For example:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml,part=unreachable --\u003e\n```ocaml\nlet%expect_test \"unreachable\" =\n  let interesting_bool = 3 \u003e 5 in\n  printf \"%b\\n\" interesting_bool;\n  if interesting_bool\n  then [%expect {| true |}]\n  else (\n    printf \"don't reach\\n\";\n    [%expect.unreachable])\n;;\n```\nbecomes:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=unreachable --\u003e\n```ocaml\nlet%expect_test \"unreachable\" =\n  let interesting_bool = 3 \u003e 5 in\n  printf \"%b\\n\" interesting_bool;\n  if interesting_bool\n  then [%expect.unreachable]\n  else (\n    printf \"don't reach\\n\";\n    [%expect\n      {|\n      false\n      don't reach\n      |}])\n;;\n```\n\nNote that, for an expect block that is sometimes reachable and sometimes not, that block\npasses if the output captured at that block matches every time the block is encountered.\nFor example, the following test passes:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=sometimes-reachable --\u003e\n```ocaml\nmodule Test (B : sig\n    val interesting_opt : int option\n  end) =\nstruct\n  let%expect_test \"sometimes reachable\" =\n    match B.interesting_opt with\n    | Some x -\u003e\n      printf \"%d\\n\" x;\n      [%expect {| 5 |}]\n    | None -\u003e [%expect {| |}]\n  ;;\nend\n\nmodule _ = Test (struct\n    let interesting_opt = Some 5\n  end)\n\nmodule _ = Test (struct\n    let interesting_opt = None\n  end)\n\nmodule _ = Test (struct\n    let interesting_opt = Some 5\n  end)\n```\n\n# Exceptions\n\nWhen an exception is raised by the body of an expect-test, the `inline_test_runner` shows\nit (and, if relevant, any output generated by the test that had not yet been captured) in\na `[@@expect.uncaught_exn]` attribute attached to the corresponding `let%expect_test`.\n`[%expect]` blocks in the test are treated according to the usual rules: those reached\nbefore the exception is raised capture output as usual, and those that \"would have\" been\nreached after are marked as unreachable:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml,part=exn --\u003e\n```ocaml\nlet%expect_test \"exception\" =\n  Printexc.record_backtrace false;\n  printf \"start!\";\n  [%expect {| |}];\n  let sum = 2 + 2 in\n  if sum \u003c\u003e 3\n  then (\n    printf \"%d\" sum;\n    failwith \"nope\");\n  printf \"done!\";\n  [%expect {| done! |}]\n;;\n```\n\nbecomes:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=exn --\u003e\n```ocaml\nlet%expect_test \"exception\" =\n  Printexc.record_backtrace false;\n  printf \"start!\";\n  [%expect {| start! |}];\n  let sum = 2 + 2 in\n  if sum \u003c\u003e 3\n  then (\n    printf \"%d\" sum;\n    failwith \"nope\");\n  printf \"done!\";\n  [%expect.unreachable]\n[@@expect.uncaught_exn\n  {|\n  (Failure nope)\n  Trailing output\n  ---------------\n  4\n  |}]\n;;\n```\n\nUnlike `[%expect]` blocks, which might be reached on some runs of a test and not others, a\ntest with an `[@@expect.uncaught_exn]` attribute _must_ raise every time it is run.\nChanging the `None` branch of the functorized test from [before](#unreached-expects) to\nraise gives:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=sometimes-raises --\u003e\n```ocaml\nmodule Test' (B : sig\n    val interesting_opt : int option\n  end) =\nstruct\n  let%expect_test \"sometimes raises\" =\n    match B.interesting_opt with\n    | Some x -\u003e\n      printf \"%d\\n\" x;\n      [%expect {| 5 |}]\n    | None -\u003e failwith \"got none!\"\n  [@@expect.uncaught_exn\n    {|\n    (* expect_test: Test ran multiple times with different uncaught exceptions *)\n    =============================== Output 1 / 3 ================================\n    \u003cexpect test ran without uncaught exception\u003e\n    =============================== Output 2 / 3 ================================\n    (Failure \"got none!\")\n    =============================== Output 3 / 3 ================================\n    \u003cexpect test ran without uncaught exception\u003e\n    |}]\n  ;;\nend\n\nmodule _ = Test' (struct\n    let interesting_opt = Some 5\n  end)\n\nmodule _ = Test' (struct\n    let interesting_opt = None\n  end)\n\nmodule _ = Test' (struct\n    let interesting_opt = Some 5\n  end)\n```\n\n\n# Output capture\n\nThe extension point `[%expect.output]` evaluates to a `string` with the output that would\nhave been captured had an `[%expect]` node been there instead.\n\nOne idiom for testing non-deterministic output is to capture the output using\n`[%expect.output]` and post-process it:\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=output-capture --\u003e\n```ocaml\n(* Suppose we want to test code that attaches a timestamp to everything it prints *)\nlet print_message s = printf \"%s: %s\\n\" (Time_float.to_string_utc (Time_float.now ())) s\n\nlet%expect_test \"output capture\" =\n  (* A simple way to clean up the non-determinism is to 'X' all digits *)\n  let censor_digits s = String.map s ~f:(fun c -\u003e if Char.is_digit c then 'X' else c) in\n  print_message \"Hello\";\n  [%expect.output] |\u003e censor_digits |\u003e print_endline;\n  [%expect {| XXXX-XX-XX XX:XX:XX.XXXXXXZ: Hello |}];\n  print_message \"world\";\n  [%expect.output] |\u003e censor_digits |\u003e print_endline;\n  [%expect {| XXXX-XX-XX XX:XX:XX.XXXXXXZ: world |}]\n;;\n```\n\nOther uses of `[%expect.output]` include:\n\n* Sorting lines of output printed in nondeterministic order.\n* Passing output that is known to be a sexp to `t_of_sexp` and performing tests on the\n  resulting structure.\n* Performing some sort of additional validation on the output before printing it to a\n  normal `[%expect]` block.\n\n# Configuration\n\nExpect-test exposes hooks for configuring how the bodies of expect tests are run, which\ncan be used to set up and tear down test environments, sanitize output, or embed\n`[%expect]` expressions in a monadic computation, like a `Deferred.t`.\n\nEach `let%expect_test` reads these configurations from the module named\n`Expect_test_config` in the scope of that let binding. The default module in scope defines\nno-op hooks that the user can override. To do so, first include the existing\n`Expect_test_config`, then override a subset of the following interface:\n\n```ocaml\nmodule type Expect_test_config = sig\n  (** The type of the expression on the RHS of a [let%expect_test]\n      binding is [unit IO.t] *)\n  module IO : sig\n      type 'a t\n\n      val return : 'a -\u003e 'a t\n  end\n\n  (** Run an IO operation until completion *)\n  val run : (unit -\u003e unit IO.t) -\u003e unit\n\n  (** [sanitize] can be used to map all output strings, e.g. for cleansing. *)\n  val sanitize : string -\u003e string\n\n  (** This module type actually contains other definitions, but they\n      are for internal testing of [ppx_expect] only. *)\nend\n```\n\nFor example, `Async` exports an `Expect_test_config` equivalent to:\n\n```ocaml skip\nmodule Expect_test_config = struct\n  include Expect_test_config\n\n  module IO = Async_kernel.Deferred\n\n  let run f = Async_unix.Thread_safe.block_on_async_exn f\nend\n```\n\nIf we want to consistently apply the same sanitization to all of the output in our expect\ntest, like we did in the timestamp example above, we can override\n`Expect_test_config.sanitize`. This cleans up the testing code and removes the need to use\n`[%expect.output]`.\n\n\u003c!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=sanitization --\u003e\n```ocaml\n(* Suppose we want to test code that attaches a timestamp to everything it prints *)\nlet print_message s = printf \"%s: %s\\n\" (Time_float.to_string_utc (Time_float.now ())) s\n\nmodule Expect_test_config = struct\n  include Expect_test_config\n\n  (* A simple way to clean up the non-determinism is to 'X' all digits *)\n  let sanitize s = String.map s ~f:(fun c -\u003e if Char.is_digit c then 'X' else c)\nend\n\nlet%expect_test \"sanitization\" =\n  print_message \"Hello\";\n  [%expect {| XXXX-XX-XX XX:XX:XX.XXXXXXZ: Hello |}];\n  print_message \"world\";\n  [%expect {| XXXX-XX-XX XX:XX:XX.XXXXXXZ: world |}]\n;;\n```\n\n# Build system integration\n\nFollow the same rules as for\n[ppx_inline_test](https://github.com/janestreet/ppx_inline_test?tab=readme-ov-file#building-and-running-the-tests-outside-of-jane-street-with-dune).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjanestreet%2Fppx_expect","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjanestreet%2Fppx_expect","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjanestreet%2Fppx_expect/lists"}