{"id":19112152,"url":"https://github.com/ocaml-multicore/multicoretests","last_synced_at":"2025-04-30T22:09:45.095Z","repository":{"id":36954482,"uuid":"415139457","full_name":"ocaml-multicore/multicoretests","owner":"ocaml-multicore","description":"PBT testsuite and libraries for testing multicore OCaml","archived":false,"fork":false,"pushed_at":"2025-04-30T20:14:48.000Z","size":3441,"stargazers_count":40,"open_issues_count":57,"forks_count":16,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-30T22:09:36.829Z","etag":null,"topics":["multicore-ocaml","property-based-testing","property-testing","quickcheck"],"latest_commit_sha":null,"homepage":"https://ocaml-multicore.github.io/multicoretests/","language":"OCaml","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ocaml-multicore.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-10-08T22:11:24.000Z","updated_at":"2025-04-30T20:14:48.000Z","dependencies_parsed_at":"2023-12-11T11:25:10.424Z","dependency_job_id":"2fca67b9-5e71-4c7d-bc85-3c5a72cde984","html_url":"https://github.com/ocaml-multicore/multicoretests","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ocaml-multicore%2Fmulticoretests","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ocaml-multicore%2Fmulticoretests/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ocaml-multicore%2Fmulticoretests/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ocaml-multicore%2Fmulticoretests/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ocaml-multicore","download_url":"https://codeload.github.com/ocaml-multicore/multicoretests/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251789612,"owners_count":21644085,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["multicore-ocaml","property-based-testing","property-testing","quickcheck"],"created_at":"2024-11-09T04:31:51.278Z","updated_at":"2025-04-30T22:09:45.088Z","avatar_url":"https://github.com/ocaml-multicore.png","language":"OCaml","readme":"Multicore tests\n===============\n\n[![OPAM installation](https://github.com/ocaml-multicore/multicoretests/actions/workflows/opam.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/opam.yml)\n\n[![Linux 5.3.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-530-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-530-trunk.yml)\n[![macOS-Intel 5.3.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/macosx-intel-530-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/macosx-intel-530-trunk.yml)\n[![macOS-ARM64 5.3.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/macosx-arm64-530-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/macosx-arm64-530-trunk.yml)\n[![Linux 5.3.0+trunk-bytecode](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-530-trunk-bytecode.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-530-trunk-bytecode.yml)\n[![Linux 5.3.0+trunk-debug](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-530-trunk-debug.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-530-trunk-debug.yml)\n[![Linux 5.3.0+trunk-musl](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-530-trunk-musl.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-530-trunk-musl.yml)\n[![Linux 32-bit 5.3.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-530-trunk-32bit.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-530-trunk-32bit.yml)\n[![Linux FP 5.3.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-530-trunk-fp.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-530-trunk-fp.yml)\n[![Linux ARM64 5.3.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-530-trunk-arm64.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-530-trunk-arm64.yml)\n[![MinGW 5.3.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/mingw-530-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/mingw-530-trunk.yml)\n[![MinGW 5.3.0+trunk-bytecode](https://github.com/ocaml-multicore/multicoretests/actions/workflows/mingw-530-trunk-bytecode.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/mingw-530-trunk-bytecode.yml)\n[![Cygwin 5.3.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/cygwin-530-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/cygwin-530-trunk.yml)\n[![MSVC 5.3.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/msvc-530-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/msvc-530-trunk.yml)\n[![MSVC 5.3.0+trunk-bytecode](https://github.com/ocaml-multicore/multicoretests/actions/workflows/msvc-530-trunk-bytecode.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/msvc-530-trunk-bytecode.yml)\n\n[![Linux 5.4.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-540-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-540-trunk.yml)\n[![macOS-Intel 5.4.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/macosx-intel-540-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/macosx-intel-540-trunk.yml)\n[![macOS-ARM64 5.4.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/macosx-arm64-540-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/macosx-arm64-540-trunk.yml)\n[![Linux 5.4.0+trunk-bytecode](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-540-trunk-bytecode.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-540-trunk-bytecode.yml)\n[![Linux 5.4.0+trunk-debug](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-540-trunk-debug.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-540-trunk-debug.yml)\n[![Linux 5.4.0+trunk-musl](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-540-trunk-musl.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-540-trunk-musl.yml)\n[![Linux 32-bit 5.4.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-540-trunk-32bit.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-540-trunk-32bit.yml)\n[![Linux FP 5.4.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-540-trunk-fp.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-540-trunk-fp.yml)\n[![Linux ARM64 5.4.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-540-trunk-arm64.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-540-trunk-arm64.yml)\n[![MinGW 5.4.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/mingw-540-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/mingw-540-trunk.yml)\n[![MinGW 5.4.0+trunk-bytecode](https://github.com/ocaml-multicore/multicoretests/actions/workflows/mingw-540-trunk-bytecode.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/mingw-540-trunk-bytecode.yml)\n[![Cygwin 5.4.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/cygwin-540-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/cygwin-540-trunk.yml)\n[![MSVC 5.4.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/msvc-540-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/msvc-540-trunk.yml)\n[![MSVC 5.4.0+trunk-bytecode](https://github.com/ocaml-multicore/multicoretests/actions/workflows/msvc-540-trunk-bytecode.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/msvc-540-trunk-bytecode.yml)\n\n[![Linux 5.5.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-550-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-550-trunk.yml)\n[![macOS-Intel 5.5.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/macosx-intel-550-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/macosx-intel-550-trunk.yml)\n[![macOS-ARM64 5.5.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/macosx-arm64-550-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/macosx-arm64-550-trunk.yml)\n[![Linux 5.5.0+trunk-bytecode](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-550-trunk-bytecode.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-550-trunk-bytecode.yml)\n[![Linux 5.5.0+trunk-debug](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-550-trunk-debug.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-550-trunk-debug.yml)\n[![Linux 5.5.0+trunk-musl](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-550-trunk-musl.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-550-trunk-musl.yml)\n[![Linux 32-bit 5.5.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-550-trunk-32bit.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-550-trunk-32bit.yml)\n[![Linux FP 5.5.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-550-trunk-fp.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-550-trunk-fp.yml)\n[![Linux ARM64 5.5.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-550-trunk-arm64.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/linux-550-trunk-arm64.yml)\n[![MinGW 5.5.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/mingw-550-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/mingw-550-trunk.yml)\n[![MinGW 5.5.0+trunk-bytecode](https://github.com/ocaml-multicore/multicoretests/actions/workflows/mingw-550-trunk-bytecode.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/mingw-550-trunk-bytecode.yml)\n[![Cygwin 5.5.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/cygwin-550-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/cygwin-550-trunk.yml)\n[![MSVC 5.5.0+trunk](https://github.com/ocaml-multicore/multicoretests/actions/workflows/msvc-550-trunk.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/msvc-550-trunk.yml)\n[![MSVC 5.5.0+trunk-bytecode](https://github.com/ocaml-multicore/multicoretests/actions/workflows/msvc-550-trunk-bytecode.yml/badge.svg)](https://github.com/ocaml-multicore/multicoretests/actions/workflows/msvc-550-trunk-bytecode.yml)\n\nProperty-based tests of (parts of) the OCaml multicore compiler and run time.\n\nThis project contains\n- a randomized test suite of OCaml 5.x, packaged up in `multicoretests.opam`\n- two reusable testing libraries:\n  - `Lin` packaged up in `qcheck-lin.opam` and\n  - `STM` packaged up in `qcheck-stm.opam`\n\nAll of the above build on [QCheck](https://github.com/c-cube/qcheck),\na black-box, property-based testing library in the style of QuickCheck.\n\nThe two libraries are [already quite helpful](https://tarides.com/blog/2022-12-22-ocaml-5-multicore-testing-tools)\n\n\n\nInstallation instructions, and running the tests\n================================================\n\nThe multicore test suite requires OCaml 5.x:\n```\nopam update\nopam switch create 5.3.0\n```\n\nIt is designed to stress test OCaml 5's multicore features. As such, it requires\na multicore machine to run successfully and will fail on a single-CPU setup. We\nrecommend running the test suite with 3 or more (virtual) CPUs.\n\n\nInstalling the libraries\n------------------------\n\nThe two testing libraries are available as packages `qcheck-lin`\nand `qcheck-stm` from the opam repository. The full versions require\nOCaml 5.x and reduced, non-`Domain` versions are available for OCaml\n4.12.x to 4.14.x. They can be installed in the usual way:\n```\nopam install qcheck-lin\nopam install qcheck-stm\n```\n\nBleeding edge users can `pin` and install the latest `main` as follows:\n```\nopam pin -y https://github.com/ocaml-multicore/multicoretests.git#main\n```\n\nTo use the `Lin` library in parallel mode on a Dune project, add the\nfollowing dependency to your dune rule:\n```\n  (libraries qcheck-lin.domain)\n```\nUsing the `STM` library in sequential mode requires the dependency\n`(libraries qcheck-stm.sequential)` and the parallel mode similarly\nrequires the dependency `(libraries qcheck-stm.domain)`.\n\n\nRunning the test suite\n----------------------\n\nWe have not released the test suite on the [opam\nrepository](https://github.com/ocaml/opam-repository) at this point.\nThe test suite can be built and run from a clone of this repository\nwith the following commands:\n```\nopam install . --deps-only --with-test\ndune build\ndune runtest -j1 --no-buffer --display=quiet\n```\n\nAs some of the tests are negative, e.g., confirming known domain unsafety by\nfinding a counterexample, we recommend running only one property-based test at a\ntime (`-j1`) to prevent contention between the individual tests for CPU\nresources.\n\nIndividual tests can be run by invoking `dune exec`. For example:\n```\n$ dune exec src/atomic/stm_tests.exe -- -v\nrandom seed: 51501376\ngenerated error fail pass / total     time test name\n[✓] 1000    0    0 1000 / 1000     0.2s sequential atomic test\n[✓] 1000    0    0 1000 / 1000   180.8s parallel atomic test\n================================================================================\nsuccess (ran 2 tests)\n```\n\nSee [src/README.md](src/README.md) for an overview of the current\nPBTs of OCaml 5.x.\n\nIt is also possible to run the test suite in the CI, by altering\n[.github/workflows/common.yml](.github/workflows/common.yml) to target\na particular compiler PR:\n```\n      COMPILER_REF:             'refs/pull/12345/head'\n```\n\nor a particular branch of a particular fork:\n```\n      COMPILER_REPO:            'login/ocaml'\n      COMPILER_REF:             'refs/heads/test-me'\n```\n\nSince [ocaml/ocaml#13458](https://github.com/ocaml/ocaml/pull/13458)\nthe test suite can be triggered on an ocaml/ocaml PR (or on a fork of it)\nby adding the `run-multicoretests` label.\n\n\nA Linearization Tester\n======================\n\nThe `Lin` module lets a user test an API for *sequential consistency*,\ni.e., it performs a sequence of random commands in parallel, records\nthe results, and checks whether the observed results can be linearized\nand reconciled with some sequential execution. The library offers an\nembedded, combinator DSL to describe signatures succinctly. As an\nexample, the required specification to test (a small part of) the\n`Hashtbl` module is as follows:\n\n``` ocaml\nmodule HashtblSig =\nstruct\n  type t = (char, int) Hashtbl.t\n  let init () = Hashtbl.create ~random:false 42\n  let cleanup _ = ()\n\n  open Lin\n  let a,b = char_printable,nat_small\n  let api =\n    [ val_ \"Hashtbl.add\"    Hashtbl.add    (t @-\u003e a @-\u003e b @-\u003e returning unit);\n      val_ \"Hashtbl.remove\" Hashtbl.remove (t @-\u003e a @-\u003e returning unit);\n      val_ \"Hashtbl.find\"   Hashtbl.find   (t @-\u003e a @-\u003e returning_or_exc b);\n      val_ \"Hashtbl.mem\"    Hashtbl.mem    (t @-\u003e a @-\u003e returning bool);\n      val_ \"Hashtbl.length\" Hashtbl.length (t @-\u003e returning int); ]\nend\n\nmodule HT = Lin_domain.Make(HashtblSig)\n;;\nQCheck_base_runner.run_tests_main [\n  HT.lin_test `Domain ~count:1000 ~name:\"Lin Hashtbl test\";\n]\n```\n\nThe first line indicates the type of the system under test along with\nbindings `init` and `cleanup` for setting it up and tearing it down.\nThe `api` then contains a list of type signature descriptions using\ncombinators `unit`, `bool`, `int`, `returning`, `returning_or_exc`,\n... in the style of [Ctypes](https://github.com/ocamllabs/ocaml-ctypes).\nThe functor `Lin_domain.Make` expects a description of the tested\ncommands and outputs a module with a QCheck test `lin_test` that\nperforms the linearization test.\n\nThe QCheck linearization test iterates a number of test\ninstances. Each instance consists of a \"sequential prefix\" of calls to\nthe above commands, followed by a `spawn` of two parallel `Domain`s\nthat each call a sequence of operations. `Lin` chooses the individual\noperations and arguments arbitrarily and records their results. The\nframework then performs a search for a sequential interleaving of the\nsame calls, and succeeds if it finds one.\n\nSince `Hashtbl`s are not safe for parallelism, if you run\n`dune exec doc/example/lin_tests.exe` the output can produce the\nfollowing output, where each tested command is annotated with its result:\n```\n\nMessages for test Lin Hashtbl test:\n\n  Results incompatible with sequential execution\n\n                                    |\n                                    |\n                 .------------------------------------.\n                 |                                    |\n     Hashtbl.add t 'a' 0  : ()            Hashtbl.add t 'a' 0  : ()\n       Hashtbl.length t  : 1                Hashtbl.length t  : 1\n```\n\nIn this case, the test tells us that there is no sequential\ninterleaving of these calls which would return `1` from both calls to\n`Hashtbl.length`. For example, in the following sequential interleaving\nthe last call should return `2`:\n``` ocaml\n Hashtbl.add t 'a' 0;;\n let res1 = Hashtbl.length t;;\n Hashtbl.add t 'a' 0;;\n let res2 = Hashtbl.length t;;\n```\n\nSee [src/atomic/lin_tests.ml](src/atomic/lin_tests.ml) for\nanother example of testing the `Atomic` module.\n\n\nA Parallel State-Machine Testing Library\n========================================\n\n`STM` contains a revision of [qcstm](https://github.com/jmid/qcstm)\nextended to run parallel state-machine tests akin to [Erlang\nQuickCheck, Haskell Hedgehog, ScalaCheck, ...](https://github.com/jmid/pbt-frameworks).\nTo do so, the `STM` library also performs a sequence of random\noperations in parallel and records the results. In contrast to `Lin`,\n`STM` then checks whether the observed results are linearizable by\nreconciling them with a sequential execution of a `model` description.\nThe `model` expresses the intended meaning of each tested command. As\nsuch, it requires more of the user compared to `Lin`. The\ncorresponding code to describe a `Hashtbl` test using `STM` is\ngiven below:\n\n``` ocaml\nopen QCheck\nopen STM\n\n(** parallel STM tests of Hashtbl *)\n\nmodule HashtblModel =\nstruct\n  type sut = (char, int) Hashtbl.t\n  type state = (char * int) list\n  type cmd =\n    | Add of char * int\n    | Remove of char\n    | Find of char\n    | Mem of char\n    | Length [@@deriving show { with_path = false }]\n\n  let init_sut () = Hashtbl.create ~random:false 42\n  let cleanup (_:sut) = ()\n\n  let arb_cmd (s:state) =\n    let char =\n      if s=[]\n      then Gen.printable\n      else Gen.(oneof [oneofl (List.map fst s); printable]) in\n    let int = Gen.nat in\n    QCheck.make ~print:show_cmd\n      (Gen.oneof\n         [Gen.map2 (fun k v -\u003e Add (k,v)) char int;\n          Gen.map  (fun k   -\u003e Remove k) char;\n          Gen.map  (fun k   -\u003e Find k) char;\n          Gen.map  (fun k   -\u003e Mem k) char;\n          Gen.return Length; ])\n\n  let next_state (c:cmd) (s:state) = match c with\n    | Add (k,v) -\u003e (k,v)::s\n    | Remove k  -\u003e List.remove_assoc k s\n    | Find _\n    | Mem _\n    | Length    -\u003e s\n\n  let run (c:cmd) (h:sut) = match c with\n    | Add (k,v) -\u003e Res (unit, Hashtbl.add h k v)\n    | Remove k  -\u003e Res (unit, Hashtbl.remove h k)\n    | Find k    -\u003e Res (result int exn, protect (Hashtbl.find h) k)\n    | Mem k     -\u003e Res (bool, Hashtbl.mem h k)\n    | Length    -\u003e Res (int,  Hashtbl.length h)\n\n  let init_state = []\n\n  let precond (_:cmd) (_:state) = true\n  let postcond (c:cmd) (s:state) (res:res) = match c,res with\n    | Add (_,_), Res ((Unit,_),_)\n    | Remove _,  Res ((Unit,_),_) -\u003e true\n    | Find k,    Res ((Result (Int,Exn),_),r) -\u003e r = (try Ok (List.assoc k s) with Not_found -\u003e Error Not_found)\n    | Mem k,     Res ((Bool,_),r) -\u003e r = List.mem_assoc k s\n    | Length,    Res ((Int,_),r)  -\u003e r = List.length s\n    | _ -\u003e false\nend\n\nmodule HT_seq = STM_sequential.Make(HashtblModel)\nmodule HT_dom = STM_domain.Make(HashtblModel)\n;;\nQCheck_base_runner.run_tests_main\n  (let count = 200 in\n   [HT_seq.agree_test     ~count ~name:\"Hashtbl test sequential\";\n    HT_dom.agree_test_par ~count ~name:\"Hashtbl test parallel\"; ])\n```\n\nAgain this requires a type `sut` for the system under test, and\nbindings `init_sut` and `cleanup` for setting it up and tearing it\ndown. The type `cmd` describes the tested commands.\n\nThe type `state = (char * int) list` describes with a pure association\nlist the internal state of a `Hashtbl`. The `init_state` represents\nthe empty `Hashtbl` mode and the state transition function\n`next_state` describes how it changes across each `cmd`. For\nexample, `Add (k,v)` appends the key-value pair onto the association\nlist.\n\n`arb_cmd` is a generator of `cmd`s, taking `state` as a parameter.\nThis allows for `state`-dependent `cmd` generation, which we use\nto increase the chance of producing a `Remove 'c'`, `Find 'c'`, ...\nfollowing an `Add 'c'`. Internally `arb_cmd` uses QCheck combinators\n`Gen.return`, `Gen.map`, and `Gen.map2` to generate one of\n5 different commands.\n\n`run` executes the tested `cmd` over the `sut` and wraps the result up\nin a result type `res` offered by `STM`. Combinators `unit`, `bool`,\n`int`, ... allow to annotate the result with the expected type.\n`postcond` expresses a post-condition by matching the received `res`,\nfor a `cmd` with the corresponding answer from the `model`. For\nexample, this compares the Boolean result `r` from `Hashtbl.mem` with\nthe result from `List.mem_assoc`. Similarly `precond` expresses a\npre-condition.\n\nThe module is phrased as functors:\n - the functor `STM_sequential.Make` produces a module with a function\n   `agree_test` to test whether the model agrees with the `sut` across\n   a sequential run of an arbitrary command sequence and\n - the functor `STM_domain.Make` produces a module with a function\n   `agree_test_par` which tests in parallel by `spawn`ing two domains\n   with `Domain` similarly to `Lin` and searches for a sequential\n   interleaving over the model.\n\nWhen running the above with the command `dune exec doc/example/stm_tests.exe`\none may obtain the following output:\n```\nMessages for test Hashtbl test parallel:\n\n  Results incompatible with linearized model\n\n                                  |\n                                  |\n               .------------------------------------.\n               |                                    |\n     (Add ('e', 5268)) : ()                (Add ('!', 4)) : ()\n           Length : 1                           Length : 1\n```\n\nThis illustrates how two hashtable `Add` commands may interfere when\nexecuted in parallel, leaving only `1` entry in the resulting\n`Hashtbl` - which is not reconcilable with the declarative model\ndescription.\n\n\nThe above examples are available from the [doc/example](doc/example)\ndirectory. The [doc](doc) directory also contains our 2022 OCaml\nWorkshop paper presenting the project in a bit more detail.\n\n\n\nRepeatability Efforts\n=====================\n\nBoth `Lin` and `STM` perform randomized property-based testing with\nQCheck. When rerunning a test to shrink/reduce the test input, QCheck\nthus starts from the same `Random` seed to limit non-determinism.\nThis is however not suffient for multicore programs where CPU\nscheduling and garbage collection may hinder reproducibility.\n\n`Lin` and `STM` primarily uses test repetition to increase\nreproducibility and it is sufficient that only a single repetition\ntriggers an issue. Currently repeating a non-deterministic QCheck\nproperty can be done in two different ways:\n - a `repeat`-combinator lets you test a property, e.g., 50 times\n   rather than just 1. (Pro: a failure is found faster, Con: wasted,\n   repetitive testing when there are no failures)\n - [a `QCheck` PR](https://github.com/c-cube/qcheck/pull/212) extends `Test.make` with a `~retries` parameter causing\n   it to only perform repetition during shrinking. (Pro: each test is\n   cheaper so we can run more, Con: more tests are required to trigger a race)\n\n\nIssues\n======\n\nTearing in `Array.sub` and `Array.copy` (new, fixed, runtime)\n-------------------------------------------------------------\n\nA `Lin` `Dynarray` test observed rare unsuspected values (and even rarer crashes)\non the CI's musl build, which turned out to be [due to tearing in the `memcpy`\nunderlying `Array.sub` and`Array.copy`](https://github.com/ocaml/ocaml/pull/13950).\n\n\nReplacing blocking functions by non-blocking ones caused deadlocks (new, fixed, runtime)\n----------------------------------------------------------------------------------------\n\n[A recently merged PR](https://github.com/ocaml/ocaml/pull/13227) replacing blocking functions\nby unblocking ones caused [a regression in the form of deadlocks in the parallel `Sys` STM test](https://github.com/ocaml/ocaml/issuess/13713).\n\n\nUnboxed `Dynarray` STM tests segfaults (new, fixed, runtime)\n------------------------------------------------------------\n\nEarly `STM` tests of [the unboxed `Dynarray` PR](https://github.com/ocaml/ocaml/pull/12885) triggered\n[segfaults caused by mixing flat float arrays with boxed arrays](https://github.com/ocaml/ocaml/pull/12885#discussion_r1568976695).\n\n\nRace condition in backup thread logic (new, fixed, runtime)\n-----------------------------------------------------------\n\nAn assertion error revealed [a race condition between two atomic updates\nunderlying the coordination between a spawned domain and its backup thread](https://github.com/ocaml/ocaml/issues/13677).\n\n\nMarking color problem when adopting orphaned Ephemerons (new, runtime)\n----------------------------------------------------------------------\n\nAn assertion error during the upstreaming of [a mark-delay improvement](https://github.com/ocaml/ocaml/pull/13580), [revealed\na problem](https://github.com/ocaml/ocaml/pull/13580#issuecomment-2478454501) [with the marking color of orphaned Ephemerons]( https://github.com/ocaml-flambda/flambda-backend/pull/3332).\n\n\nParallel usage of `flush` may trigger `Sys_error` exception (new, runtime)\n--------------------------------------------------------------------------\n\nThe `Out_channel` tests found that [`flush` may raise a `Sys_error`\nexception when used in parallel with a `close`](https://github.com/ocaml/ocaml/issues/13586).\n\n\nRegistered bytecode fragments leading to bytecode crashes (new, fixed, runtime)\n-------------------------------------------------------------------------------\n\nBoth the `Gc` and `Domain.DLS` tests triggered crashes due to bytecode fragments in\ncallbacks [not being properly unregistered](https://github.com/ocaml/ocaml/pull/13549),\n[fixed in a separate PR](https://github.com/ocaml/ocaml/pull/13553).\n\n\nOut of date `Gc.control` documentation (new, fixed, stdlib)\n-----------------------------------------------------------\n\nTests of the `Gc` module revealed that `Gc.control` records contain\n[constant zero fields ignored by `Gc.set`](https://github.com/ocaml/ocaml/pull/13440)\n\n\nOut of date  `Gc.quick_stat` documentation (new, fixed, stdlib)\n---------------------------------------------------------------\n\nTests of the `Gc` module revealed that `Gc.quick_stat` did not\nreturn [a record with 4 zero fields as documented](https://github.com/ocaml/ocaml/pull/13424)\n\n\nShared heap assertion failure (known, fixed, runtime)\n-----------------------------------------------------\n\nNew GC tests offered a simple reproducer for consistently triggering\n[a shared heap assertion error](https://github.com/ocaml/ocaml/issues/13090)\n\n\nUnsafe GC interaction in `Gc.counters` binding (known, fixed, runtime)\n----------------------------------------------------------------------\n\nNew GC tests spotted an issue with unsafe root registration in\n`Gc.counters` in 5.2.0, [already fixed upstream](https://github.com/ocaml/ocaml/pull/13370)\n\n\nAssertion error `s-\u003erunning` in backup thread termination (new, fixed, runtime)\n-------------------------------------------------------------------------------\n\nTests of `In_channel` would trigger an occasional race in a debug\nassertion, due to a [TOCTOU](https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use)\nrace for [incoming interrupts during backup thread termination](https://github.com/ocaml/ocaml/issues/13386)\n\n\nParallel `Dynlink` tests under Windows could deadlock or crash (known, fixed, flexdll)\n--------------------------------------------------------------------------------------\n\nTests of `Dynlink` on Windows revealed that [the underlying FlexDLL was\nunsafe for parallel usage](https://github.com/ocaml/ocaml/issues/13046)\n\n\n`Sys.rename` regression under MinGW/MSVC (new, fixed, runtime)\n--------------------------------------------------------------\n\nEarlier fixes to bring Windows behaviour closer to other platforms introduced\n[an unfortunate cornercase regression](https://github.com/ocaml/ocaml/pull/13166)\nif attempting to `Sys.rename` a parent directory to an empty child directory\n\n\nRegression causing a Cygwin configure to fail (new, fixed, configure)\n---------------------------------------------------------------------\n\nA configure PR accidentally [introduced a regression causing a flexlink test to\nfail for a Cygwin build](https://github.com/ocaml/ocaml/pull/13009)\n\n\nCrash and hangs on MinGW (new, fixed, runtime)\n----------------------------------------------\n\n[We observed crashes and hangs of the `threadomain` test under\nMinGW](https://github.com/ocaml/ocaml/issues/12230), which turned out to be due\nto unsafe systhread yielding.\n\n\nRegression on output to closed `Out_channel`s (new, fixed, runtime)\n-------------------------------------------------------------------\n\nWhile revising out `Out_channel` tests we discovered [a regression when\noutputting to a closed `Out_channel`](https://github.com/ocaml/ocaml/issues/12898)\n\n\nFailure to build `dune` with trunk (new, fixed, dune)\n-----------------------------------------------------\n\nA change to the OCaml compiler's internals revealed that `dune` was [not\nusing `CAML_INTERNALS` according to the OCaml manual](https://github.com/ocaml/dune/pull/9733)\n\n\nHard abort regression on 'failure to create domains' (new, fixed, runtime)\n--------------------------------------------------------------------------\n\nThe tests found a regression where a failure to create a domain [would trigger an\nabort rather than an exception](https://github.com/ocaml/ocaml/pull/12855)\n\n\nAssertion failures in `runtime/domain.c` on trunk (new, fixed, runtime)\n-----------------------------------------------------------------------\n\nA PR merged to `trunk` [reintroduced off-by-one assertion errors in `caml_reset_young_limit`](\nhttps://github.com/ocaml/ocaml/pull/12824)\n\n\nAssertion failure triggered in runtime/memprof.c (new, fixed, runtime)\n----------------------------------------------------------------------\n\nThe `thread_joingraph` test triggered [an assertion boundary case in\n`caml_memprof_renew_minor_sample` from `memprof.c`](https://github.com/ocaml/ocaml/pull/12817)\n\n\nAssertion boundary case in `caml_reset_young_limit` (new, fixed, runtime)\n-------------------------------------------------------------------------\n\nThe `thread_joingraph` test triggered [an assertion boundary case in\n`caml_reset_young_limit` which was too strict](https://github.com/ocaml/ocaml/pull/12742)\n\n\nAssertion race condition in `install_backup_thread` (new, fixed, runtime)\n-------------------------------------------------------------------------\n\nA repro test case [submitted upstream from `multicoretests` to the ocaml\ncompiler test suite](https://github.com/ocaml/ocaml/pull/11749) and two separate\n`multicoretests` all [triggered an race condition in `install_backup_thread`](https://github.com/ocaml/ocaml/pull/12707)\n\n\nFloat register preservation on ppc64 (new, fixed, codegen)\n----------------------------------------------------------\n\nThe sequential `Float.Array` `STM` test revealed that a float register\nwas not properly preserved on ppc64, sometimes resulting in\n[random `float` values appearing](https://github.com/ocaml/ocaml/pull/12546)\n\n\nSignal-based overflow on ppc64 crash (new, fixed, codegen)\n----------------------------------------------------------\n\nThe sequential `STM` tests of `Array`, `Bytes`, and `Float.Array`\nwould [trigger segfaults on ppc64](https://github.com/ocaml/ocaml/issues/12482)\n\n\nFrame pointer `Effect` crashes (new, fixed, codegen)\n----------------------------------------------------\n\nNegative `Lin` `Effect` tests exercising exceptions for unhandled\n`Effect`s triggered a [crash on a frame pointer switch](https://github.com/ocaml/ocaml/pull/12535)\n\n\ns390x `Effect` crashes (new, fixed, codegen)\n--------------------------------------------\n\nNegative `Lin` `Effect` tests exercising exceptions for unhandled\n`Effect`s also triggered [a crash on the newly restored s390x backend](https://github.com/ocaml/ocaml/issues/12486)\n\n\n`Sys.rename` behaves differently on corner cases under MingW (new, fixed, stdlib)\n---------------------------------------------------------------------------------\n\nSequential `STM` tests targeting `Sys.rename` found [two corner cases\nwhere MingW behaves differently](https://github.com/ocaml/ocaml/issues/12073)\n\n\n`flexdll` contains a race condition in its handling of errors (new, fixed, flexdll)\n-----------------------------------------------------------------------------------\n\nParallel `Lin` tests of the `Dynlink` module found [a race\ncondition](https://github.com/ocaml/flexdll/pull/112) in accesses to\nthe global variables storing the last error.\n\n\n`Buffer.add_string` contained a race condition (new, fixed, stdlib)\n-------------------------------------------------------------------\n\nParallel `STM` tests of the `Buffer` module found a segfault, leading\nto the discovery of an [assertion failure](https://github.com/ocaml/ocaml/issues/12103)\nrevealing a race condition in the `add_string` function\n\n\nParallel `Weak` `Hashset` usage may crash the runtime (new, fixed, runtime)\n---------------------------------------------------------------------------\n\nParallel `STM` tests found a combination of `Weak` `Hashset` functions\nthat [may cause the run-time to `abort` or segfault](https://github.com/ocaml/ocaml/issues/11934)\n\n\n`Sys.readdir` on MingW disagrees with Linux behavior (new, fixed, stdlib)\n-------------------------------------------------------------------------\n\nSequential `STM` tests of `Sys` showed how `Sys.readdir` of a\nnon-existing directory on MingW Windows [returns an empty `array`, thus\ndisagreeing with the Linux and macOS behavior](https://github.com/ocaml/ocaml/issues/11829)\n\n\n`seek` on a closed `in_channel` may read uninitialized memory (new, fixed, runtime)\n-----------------------------------------------------------------------------------\n\nA failure of `Lin` `In_channel` tests revealed that `seek` on a closed\n`in_channel` [may read uninitialized memory](https://github.com/ocaml/ocaml/issues/11878)\n\n\nParallel usage of `Weak` could produce weird values (new, fixed, runtime)\n-------------------------------------------------------------------------\n\nRacing `Weak.set` and `Weak.get` [can in some cases produce strange values](https://github.com/ocaml/ocaml/pull/11749)\n\n\nBytecode interpreter can segfault on unhandled `Effect` (new, fixed, runtime)\n-----------------------------------------------------------------------------\n\nIn seldom cases the tests would [trigger a segfault in the bytecode interpreter when treating an unhandled `Effect`](https://github.com/ocaml/ocaml/issues/11669)\n\n\n`Ephemeron` can fail assert and abort (new, fixed, runtime)\n-----------------------------------------------------------\n\nIn some cases (even sequential) [the `Ephemeron` tests can trigger an assertion failure and abort](https://github.com/ocaml/ocaml/issues/11503).\n\n\nParallel usage of `Bytes.escaped` is unsafe (new, fixed, stdlib)\n----------------------------------------------------------------\n\nThe `Bytes` tests triggered a segfault which turned out to be caused by [an unsafe `Bytes.escaped` definition](https://github.com/ocaml/ocaml/issues/11508).\n\n\nInfinite loop in `caml_scan_stack` on ARM64 (known, fixed, runtime)\n-------------------------------------------------------------------\n\nThe tests triggered [an apparent infinite loop on ARM64 while amd64 would complete the tests as expected](https://github.com/ocaml/ocaml/issues/11425).\n\n\nUnsafe `Buffer` module (new, fixed, stdlib)\n-------------------------------------------\n\nThe tests found that the `Buffer` module implementation is [unsafe under parallel usage](https://github.com/ocaml/ocaml/issues/11279) - initially described in [multicoretests#63](https://github.com/ocaml-multicore/multicoretests/pull/63).\n\n\nMacOS segfault (new, fixed, runtime)\n------------------------------------\n\nThe tests found an issue causing [a segfault on MacOS](https://github.com/ocaml/ocaml/issues/11226).\n\n\n`In_channel` and `Out_channel` unsafety (new, fixed, runtime)\n-------------------------------------------------------------\n\nThe tests found a problem with `In_channel` and `Out_channel` which\ncould trigger segfaults under parallel usage. For details see\n[issue ocaml-multicore/multicoretests#13](https://github.com/ocaml-multicore/multicoretests/pull/13) and\n[this ocaml/ocaml#10960 comment](https://github.com/ocaml/ocaml/issues/10960#issuecomment-1087660763).\n\n\nCornercase issue in `Domainslib` (new, fixed, domainslib)\n---------------------------------------------------------\n\nThe tests found an issue in `Domainslib.parallel_for_reduce` which\n[would yield the wrong result for empty arrays](https://github.com/ocaml-multicore/domainslib/pull/67).\nAs of [domainslib#100](https://github.com/ocaml-multicore/domainslib/pull/100)\nthe `Domainslib` tests have been moved to the `Domainslib` repo.\n\n\nSpecification of `Lockfree.Ws_deque` (new, fixed, lockfree)\n-----------------------------------------------------------\n\nThe initial tests of `ws_deque` just applied the parallelism property `agree_prop_par`.\nHowever that is not sufficient, as only the original domain (thread)\nis allowed to call `push`, `pop`, ..., while a `spawn`ed domain\nshould call only `steal`.\n\nA custom, revised property test runs a `cmd` prefix, then\n`spawn`s a \"stealer domain\" with `steal`, ... calls, while the\noriginal domain performs calls across a broder random selection\n(`push`, `pop`, ...). As of\n[lockfree#43](https://github.com/ocaml-multicore/lockfree/pull/43)\nthis test has now been moved to the `lockfree` repo.\n\nHere is an example output illustrating how `size` may return `-1` when\nused in a \"stealer domain\". The first line in the `Failure` section lists\nthe original domain's commands and the second lists the stealer\ndomains commands (`Steal`,...). The second `Messages` section lists a\nrough dump of the corresponding return values: `RSteal (Some 73)` is\nthe result of `Steal`, ... Here it is clear that the spawned domain\nsuccessfully steals 73, and then observes both a `-1` and `0` result from\n`size` depending on timing. `Size` should therefore not be considered\nthreadsafe (none of the\n[two](https://www.dre.vanderbilt.edu/~schmidt/PDF/work-stealing-dequeue.pdf)\n[papers](https://hal.inria.fr/hal-00802885/document) make any such\npromises though):\n\n``` ocaml\n$ dune exec src/ws_deque_test.exe\nrandom seed: 55610855\ngenerated error  fail  pass / total     time test name\n[✗]   318     0     1   317 / 10000     2.4s parallel ws_deque test (w/repeat)\n\n--- Failure --------------------------------------------------------------------\n\nTest parallel ws_deque test (w/repeat) failed (8 shrink steps):\n\n Seq.prefix:  Parallel procs.:\n\n          []  [(Push 73); Pop; Is_empty; Size]\n\n              [Steal; Size; Size]\n\n\n+++ Messages ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n\nMessages for test parallel ws_deque test (w/repeat):\n\nResult observations not explainable by linearized model:\n\n  Seq.prefix:  Parallel procs.:\n\n          []  [RPush; (RPop None); (RIs_empty true); (RSize 0)]\n\n              [(RSteal (Some 73)); (RSize -1); (RSize 0)]\n\n================================================================================\nfailure (1 tests failed, 0 tests errored, ran 1 tests)\n```\n\n\nSegfault in Domainslib (known, fixed, domainslib)\n-------------------------------------------------\n\nTesting `Domainslib.Task`s with one dependency and with 2 work pools\nfound a [segfault in domainslib](https://github.com/ocaml-multicore/domainslib/issues/58).\nAs of [domainslib#100](https://github.com/ocaml-multicore/domainslib/pull/100)\nthe `domainslib/task_one_dep.ml` test in question has been moved to\nthe `Domainslib` repo.\n\n\n\n\nDead-lock in Domainslib (known, fixed, domainslib)\n--------------------------------------------------\n\nA reported deadlock in domainslib motivated the development of these tests:\n - https://github.com/ocaml-multicore/domainslib/issues/47\n - https://github.com/ocaml-multicore/ocaml-multicore/issues/670\n\nThe tests `domainslib/task_one_dep.ml` and\n`domainslib/task_more_deps.ml` were run with a timeout to prevent\ndeadlocking indefinitely. `domainslib/task_one_dep.ml` could trigger one\nsuch deadlock. As of [domainslib#100](https://github.com/ocaml-multicore/domainslib/pull/100)\nthese tests have been moved to the `Domainslib` repo.\n\n\nThe test exhibits no non-determistic behaviour when repeating the same\ntested property from within the QCheck test.\nHowever it fails (due to timeout) on the following test input:\n\n```ocaml\n$ dune exec -- src/task_one_dep.exe -v\nrandom seed: 147821373\ngenerated error fail pass / total     time test name\n[✗]   25    0    1   24 /  100    36.2s Task.async/await\n\n--- Failure --------------------------------------------------------------------\n\nTest Task.async/await failed (2 shrink steps):\n\n{ num_domains = 3; length = 6;\n  dependencies = [|None; (Some 0); None; (Some 1); None; None|] }\n================================================================================\nfailure (1 tests failed, 0 tests errored, ran 1 tests)\n```\n\nThis corresponds to the following program with 3+1 domains and 6 promises.\nIt loops infinitely with both bytecode/native:\n\n```ocaml\n...\nopen Domainslib\n\n(* a simple work item, from ocaml/testsuite/tests/misc/takc.ml *)\nlet rec tak x y z =\n  if x \u003e y then tak (tak (x-1) y z) (tak (y-1) z x) (tak (z-1) x y)\n           else z\n\nlet work () =\n  for _ = 1 to 200 do\n    assert (7 = tak 18 12 6);\n  done\n\nlet pool = Task.setup_pool ~num_additional_domains:3 ()\n\nlet p0 = Task.async pool work\nlet p1 = Task.async pool (fun () -\u003e work (); Task.await pool p0)\nlet p2 = Task.async pool work\nlet p3 = Task.async pool (fun () -\u003e work (); Task.await pool p1)\nlet p4 = Task.async pool work\nlet p5 = Task.async pool work\n\nlet () = List.iter (fun p -\u003e Task.await pool p) [p0;p1;p2;p3;p4;p5]\nlet () = Task.teardown_pool pool\n```\n\n---\n\nThis project has been created by \u003ca href=\"https://tarides.com/\"\u003eTarides\u003c/a\u003e.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Focaml-multicore%2Fmulticoretests","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Focaml-multicore%2Fmulticoretests","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Focaml-multicore%2Fmulticoretests/lists"}