{"id":22116327,"url":"https://github.com/jmid/pbt-frameworks","last_synced_at":"2026-01-06T03:13:59.427Z","repository":{"id":47409651,"uuid":"353416907","full_name":"jmid/pbt-frameworks","owner":"jmid","description":"An overview of property-based testing functionality","archived":false,"fork":false,"pushed_at":"2025-02-05T14:09:46.000Z","size":61,"stargazers_count":57,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-05T15:24:59.986Z","etag":null,"topics":["property-based-testing","property-testing","quickcheck"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jmid.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-31T16:12:28.000Z","updated_at":"2025-02-05T14:21:12.000Z","dependencies_parsed_at":"2025-02-05T15:22:17.972Z","dependency_job_id":"c2b1d41f-8472-4ab9-8fcb-f57b28f3c635","html_url":"https://github.com/jmid/pbt-frameworks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fpbt-frameworks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fpbt-frameworks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fpbt-frameworks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fpbt-frameworks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jmid","download_url":"https://codeload.github.com/jmid/pbt-frameworks/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245217792,"owners_count":20579297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["property-based-testing","property-testing","quickcheck"],"created_at":"2024-12-01T12:26:53.510Z","updated_at":"2026-01-06T03:13:59.408Z","avatar_url":"https://github.com/jmid.png","language":null,"readme":"Overview of Property-Based Testing Functionality\n================================================\n\nProperty-based testing (PBT) frameworks come with a number of\ndifferent features, but which library supports which features?\nFor a PBT newcomer, it can be hard to tell.\nStrictly speaking you don't even need a PBT framework. Property-based tests can be written from scratch\non a case-by-case basis using a random number generator. One can even test stateful code without a state-machine framework,\ne.g., [as outlined here](https://github.com/silentbicycle/theft/blob/master/doc/properties.md#testing-stateful-systems).\nHowever a framework provides reusable parts and infrastructure, thus paving the way for bigger developments, such as\n[property-based testing automotive software against the AUTOSAR specification](https://www.youtube.com/watch?v=zi0rHwfiX1Q).\n\n\n- When finding a counterexample shrinking is nice to get to the\n essence of an issue\n- To test an imperative API a framework with state machine support would be nice.\n- Integrated shrinking can be a nice feature for bigger developments\n where writing custom shrinkers may be out of the question.\n- ...\n\nThis overview is to help myself keep track. It has been compiled over a number of years\n[teaching PBT](https://janmidtgaard.dk/quickcheck/). As features are gradually added to\na framework the table's entries may unfortunately become outdated. YMMV.\n\nI'll be happy to accept PRs for updating entries and adding new frameworks.\n\n\n\n\nFramework functionality\n-----------------------\n\n| Framework | Language | Gen. EDSL | Shrinking | Int. shr. | State machine | Par. st. mach. | Cov. guidance |\n|----------------------------------------------------------------------|----------------|:----------------------:|:----------------------------------:|:------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------:|:--------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------:|\n| [theft](https://github.com/silentbicycle/theft) | C | ( :heavy_check_mark: ) | :heavy_check_mark: | [( :heavy_check_mark: )](https://github.com/silentbicycle/theft/blob/master/doc/shrinking.md#auto-shrinking) | | | [( :heavy_check_mark: )](https://github.com/silentbicycle/theft/issues/43) |\n| [DeepState](https://github.com/trailofbits/deepstate) | C / C++ | ( :heavy_check_mark: ) | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: |\n| [RapidCheck](https://github.com/emil-e/rapidcheck/) | C++ | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | [( :heavy_check_mark: )](https://github.com/emil-e/rapidcheck/issues/47) | |\n| [CsCheck](https://github.com/AnthonyLloyd/CsCheck) | C# / .Net | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |\n| [test.check](https://github.com/clojure/test.check) | Clojure | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ? | ? | |\n| [check-it](https://github.com/DalekBaldwin/check-it) | Common Lisp | :heavy_check_mark: | :heavy_check_mark: | | | | |\n| [cl-quickcheck](https://github.com/mcandre/cl-quickcheck) | Common Lisp | :heavy_check_mark: | | | | | |\n| [QuickChick](https://github.com/QuickChick/QuickChick) | Coq / Rocq | :heavy_check_mark: | :heavy_check_mark: | | | | :heavy_check_mark: |\n| [PropCheck](https://github.com/alfert/propcheck) | Elixir | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |\n| [StreamData](https://github.com/whatyouhide/stream_data) | Elixir | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | |\n| [elm test](https://github.com/elm-explorations/test) | Elm | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | |\n| [propcheck](https://github.com/Wilfred/propcheck) | Emacs Lisp | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | |\n| [QuickCheck](http://www.quviq.com/products/) | Erlang | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |\n| [PropEr](https://github.com/proper-testing/proper) | Erlang | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |\n| [FsCheck](https://fscheck.github.io/FsCheck/index.html) | F# / .Net | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | | |\n| [FSharp-Hedgehog](https://github.com/hedgehogqa/fsharp-hedgehog) | F# / .Net | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | |\n| [gopter](https://github.com/leanovate/gopter) | Go | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | | |\n| [Rapid](https://github.com/flyingmutant/rapid) | Go | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | |\n| [QuickCheck](https://github.com/nick8325/quickcheck) | Haskell | :heavy_check_mark: | :heavy_check_mark: | | [( :heavy_check_mark: )](https://github.com/advancedtelematic/quickcheck-state-machine) | [( :heavy_check_mark: )](https://github.com/advancedtelematic/quickcheck-state-machine) | |\n| [SmallCheck](https://github.com/Bodigrim/smallcheck) | Haskell | :heavy_check_mark: | \u003csup\u003e1\u003c/sup\u003e | \u003csup\u003e1\u003c/sup\u003e | | | |\n| [Hedgehog](https://github.com/hedgehogqa/haskell-hedgehog) | Haskell | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |\n| [Americium](https://github.com/sageserpent-open/americium) | Java / Scala | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | |\n| [junit-quickcheck](https://github.com/pholser/junit-quickcheck) | Java | :heavy_check_mark: | :heavy_check_mark: | | | | [( :heavy_check_mark: )](https://github.com/rohanpadhye/JQF)\u003csup\u003e2\u003c/sup\u003e |\n| [QuickTheories](https://github.com/quicktheories/QuickTheories) | Java | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | [( :heavy_check_mark: )](https://github.com/quicktheories/QuickTheories/issues/42) | [( :heavy_check_mark: )](https://github.com/quicktheories/QuickTheories/issues/42) | [:heavy_check_mark:](https://github.com/quicktheories/QuickTheories#coverage-guidance) |\n| [jqwik](https://jqwik.net/) | Java | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | |\n| [fast-check](https://github.com/dubzzz/fast-check) | JS / TS | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | [( :heavy_check_mark: )](https://fast-check.dev/docs/tutorials/detect-race-conditions)\u003csup\u003e3\u003c/sup\u003e | |\n| [propCheck](https://github.com/1Jajen1/propCheck) | Kotlin | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |\n| [Lua-QuickCheck](https://github.com/luc-tielen/lua-quickcheck) | Lua | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | |\n| [Fox](https://github.com/jeffh/Fox) | Obj.C / Swift | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | [( :heavy_check_mark: )](https://github.com/jeffh/Fox/pull/28) | |\n| [QCheck](https://github.com/c-cube/qcheck) | OCaml | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:\u003csup\u003e4\u003c/sup\u003e | [:heavy_check_mark:](https://github.com/ocaml-multicore/multicoretests) | [:heavy_check_mark:](https://github.com/ocaml-multicore/multicoretests) | |\n| [Crowbar](https://github.com/stedolan/crowbar) | OCaml | :heavy_check_mark: | ( :heavy_check_mark: )\u003csup\u003e5\u003c/sup\u003e | | | | :heavy_check_mark: |\n| [Monolith](https://gitlab.inria.fr/fpottier/monolith) | OCaml | :heavy_check_mark: | ( :heavy_check_mark: )\u003csup\u003e5\u003c/sup\u003e | | | | :heavy_check_mark: |\n| [Base_quickcheck](https://opensource.janestreet.com/base_quickcheck) | OCaml | :heavy_check_mark: | :heavy_check_mark: | | | | |\n| [Popper](https://github.com/jobjo/popper) | OCaml | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | |\n| [quickcheck](https://github.com/nicoabie/quickcheck) | Prolog | :heavy_check_mark: | :heavy_check_mark: | | | | |\n| [Hypothesis](https://github.com/HypothesisWorks/hypothesis) | Python | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | [( :heavy_check_mark: )](https://pypi.org/project/hypothesis-trio/)\u003csup\u003e3\u003c/sup\u003e | [( :heavy_check_mark: )](https://hypofuzz.com), [( :heavy_check_mark: )](https://hypothesis.readthedocs.io/en/latest/details.html#use-with-external-fuzzers) |\n| [TSTL](https://github.com/agroce/tstl) | Python | \u003csup\u003e6\u003c/sup\u003e | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: |\n| [R-Hedgehog](https://github.com/hedgehogqa/r-hedgehog) | R | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | |\n| [racket-quickcheck](https://github.com/ifigueroap/racket-quickcheck) | Racket | :heavy_check_mark: | | | | | |\n| [PropCheck](https://github.com/Qqwy/ruby-prop_check) | Ruby | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | |\n| [PBT](https://github.com/ohbarye/pbt) | Ruby | :heavy_check_mark: | :heavy_check_mark: | | | | |\n| [Rantly](https://github.com/rantly-rb/rantly) | Ruby | :heavy_check_mark: | :heavy_check_mark: | | | | |\n| [quickcheck](https://github.com/burntsushi/quickcheck) | Rust | :heavy_check_mark: | :heavy_check_mark: | | | | |\n| [proptest](https://github.com/proptest-rs/proptest) | Rust | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | |\n| [Scala-Hedgehog](https://github.com/hedgehogqa/scala-hedgehog) | Scala / JVM | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |\n| [ScalaCheck](https://github.com/typelevel/scalacheck) | Scala / JVM | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | :heavy_check_mark: | |\n| [scalaprops](https://github.com/scalaprops/scalaprops) | Scala / JVM | :heavy_check_mark: | :heavy_check_mark: | | | | |\n| [Echidna](https://github.com/crytic/echidna) | Solidity / EVM | | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: |\n| [qcheck](https://github.com/league/qcheck) | Standard ML | :heavy_check_mark: | :heavy_check_mark: | | | | |\n| [SwiftCheck](https://github.com/typelift/SwiftCheck) | Swift | :heavy_check_mark: | :heavy_check_mark: | | | | |\n| ...\n\n**Legend:**\n - **Language** denotes the frameworks's language or platform\n - **Generator EDSL** denotes whether the framework has an expressive, embedded domain-specific language for programming custom generators (`int`, `list`, `map`, ...)\n - **Shrinking** denotes whether the framework has built-in support for reducing counterexamples.\n - **Integrated shrinking** denotes whether a shrinker automatically preserves any invariants of an EDSL-based custom generator (sorted lists, non-empty array, valid JSON, ...)\n - **State machine** denotes whether the framework has a state-machine library for model-based testing.\n - **Parallel state machine** denotes whether the framework supports parallel state-machine testing for race conditons, etc.\n - **Coverage guidance** denotes whether the framework's generators (and shrinkers) can be guided by code coverage information obtained via instrumentation.\n\n **Footnotes**\n - \u003csup\u003e1\u003c/sup\u003e Contrary to QuickCheck and its descendants, [SmallCheck's generators enumerate values starting from the smallest ones (up to some bound)](https://github.com/Bodigrim/smallcheck/wiki/Comparison-with-QuickCheck). SmallCheck will therefore encounter a minimal counterexample first and hence doesn't require shrinking.\n - \u003csup\u003e2\u003c/sup\u003e JQF and the underlying Zest [supports multiple forms feedback guidance beyond coverage](https://github.com/rohanpadhye/jqf/wiki/The-Guidance-interface).\n - \u003csup\u003e3\u003c/sup\u003e [Hypothesis](https://github.com/HypothesisWorks/hypothesis) and [fast-check](https://github.com/dubzzz/fast-check) support asynchronous state machine testing, which can find race conditions (although it is strictly speaking not using parallel testing).\n - \u003csup\u003e4\u003c/sup\u003e QCheck supports integrated shrinking in the generators from the [QCheck2](https://c-cube.github.io/qcheck/0.21/qcheck-core/QCheck2/) module, whereas the original [QCheck](https://c-cube.github.io/qcheck/0.21/qcheck-core/QCheck/) module does not.\n - \u003csup\u003e5\u003c/sup\u003e Crowbar and Monolith both use AFL which [trims each test input as part of its core genetic algorithm](https://lcamtuf.coredump.cx/afl/README.txt). In addition they support test case reduction via `afl-tmin` [which is unaware of (and hence may break) OCaml typing](https://tarides.com/blog/2020-08-03-fuzzing-ocamlformat-with-afl-and-crowbar).\n - \u003csup\u003e6\u003c/sup\u003e [TSTL](https://github.com/agroce/tstl) instead uses an *external DSL*.\n\nBackground:\n-----------\n\nThe term *property-based testing* seems to originate from [*'Towards a Property-based Testing Environment with Applications to Security-Critical\nSoftware'* by Fink, Ko, Archer, and Levitt (Irvine Software Symposium, 1994)](https://citeseerx.ist.psu.edu/document?repid=rep1\u0026type=pdf\u0026doi=0c9dbe7b54d1a2b2db5103e707e8f6b23e6818b2).\nIt was later the topic of Fink's 1995 UC Davis PhD dissertation *'Discovering security and safety flaws using property-based testing'* and the paper\n[*'Property-Based Testing; A New Approach to Testing for Assurance'* by Fink and Bishop (SE Notes 1997)](http://nob.cs.ucdavis.edu/~bishop/papers/1997-sen/pbt.pdf).\nThe approach was popularized as an embedded domain-specific language in [*'QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs'* by Claessen and Hughes (ICFP 2000)](http://www.eecs.northwestern.edu/%7Erobby/courses/395-495-2009-fall/quick.pdf) which inspired ports to many other languages.\n\n**Integrated shrinking** is explained in more detail in\n - [an early design thread on the Haskell mailing list](https://mail.haskell.org/pipermail/libraries/2013-November/021674.html)\n - [a Reddit thread announcing Hedgehog](https://www.reddit.com/r/haskell/comments/646k3d/ann_hedgehog_property_testing/)\n - [a talk by Jacob Stanley on Hedgehog's design](https://www.youtube.com/watch?v=AIv_9T0xKEo)\n - [a blog post by Edsko de Vries with pro/cons](https://www.well-typed.com/blog/2019/05/integrated-shrinking/)\n - [McIver's Hypothesis article advocating the approach](https://hypothesis.works/articles/integrated-shrinking/)\n - [*'Test-Case Reduction via Test-Case Generation: Insights From the Hypothesis Reducer'* by MacIver and Donaldson (ECOOP 2020)](https://www.doc.ic.ac.uk/~afd/homepages/papers/pdfs/2020/ECOOP_Hypothesis.pdf)\n - [Everything You Ever Wanted to Know About Test-Case Reduction, But Didn’t Know to Ask](https://blog.trailofbits.com/2019/11/11/test-case-reduction/) - describing DeepState's reducer/shrinker\n\n\n**State machines** to test protocols and systems with state are described in\n - [*'Testing reactive systems with GAST'* by Koopman and Plasmeijer (TFP 2003, revised 2005)](https://repository.ubn.ru.nl/bitstream/handle/2066/60573/60573.pdf?sequence=1) - which describes a model-based framework for Clean's Gast library\n - [*'Testing Telecoms Software with Quviq QuickCheck'* Arts, Hughes, Johansson, and Wiger (Erlang 2006)](https://www.quviq.com/wp-content/uploads/erlang001-arts.pdf) - which describes Erlang's first `eqc_commands` module\n - [*'QuickCheck Testing for Fun and Profit'* by Hughes (PADL 2007)](https://citeseerx.ist.psu.edu/document?repid=rep1\u0026type=pdf\u0026doi=5ae25681ff881430797268c5787d7d9ee6cf542c) describes a later revision\n - [the API documentation of Quviq's latest (commercial) version](http://quviq.com/documentation/eqc/)\n - [*'A Note on State-Machine Frameworks for Property-Based Testing'*](https://janmidtgaard.dk/quickcheck/stmnote.pdf) - a tutorial which reconstructs `qcstm` for OCaml\n\n\n**Parallel state-machine tests** for race conditions were later introduced in\n - [*'QuickCheck Testing for Fun and Profit'* by Hughes (PADL 2007)](https://citeseerx.ist.psu.edu/document?repid=rep1\u0026type=pdf\u0026doi=5ae25681ff881430797268c5787d7d9ee6cf542c) - Sec.8 mentions running 2 parallel command sequences and a property searching for an interleaving\n - [*'Finding Race Conditions in Erlang with QuickCheck and PULSE'* by Claessen et al. (ICFP 2009)](https://smallbone.se/papers/finding-race-conditions.pdf) -- along with a scheduler and a process visualizer\n - *'Testing a Database for Race Conditions with QuickCheck'* by Hughes and Bolinder (Erlang 2011)\n\n\nAdditional resources:\n---------------------\n - [A shrinking challenge for comparing frameworks](https://github.com/jlink/shrinking-challenge)\n - [*'QuickCheck Advice'* by Jesper L. Andersen](https://medium.com/@jlouis666/quickcheck-advice-c357efb4e7e6)\n - [A recent course by John Hughes](http://www.cse.chalmers.se/~rjmh/MGS2019/)\n - [My own course material](https://janmidtgaard.dk/quickcheck/)\n - ...\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmid%2Fpbt-frameworks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjmid%2Fpbt-frameworks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmid%2Fpbt-frameworks/lists"}