{"id":17862644,"url":"https://github.com/stevana/coverage-guided-pbt","last_synced_at":"2025-04-02T21:15:09.128Z","repository":{"id":256716494,"uuid":"856180427","full_name":"stevana/coverage-guided-pbt","owner":"stevana","description":null,"archived":false,"fork":false,"pushed_at":"2024-10-15T10:04:49.000Z","size":435,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-08T11:34:44.269Z","etag":null,"topics":["coverage-guided","property-based-testing"],"latest_commit_sha":null,"homepage":"","language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stevana.png","metadata":{"files":{"readme":"README-unprocessed.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-12T06:10:26.000Z","updated_at":"2024-10-15T10:04:52.000Z","dependencies_parsed_at":"2024-10-28T09:24:47.590Z","dependency_job_id":null,"html_url":"https://github.com/stevana/coverage-guided-pbt","commit_stats":null,"previous_names":["stevana/coverage-driven-pbt"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stevana%2Fcoverage-guided-pbt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stevana%2Fcoverage-guided-pbt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stevana%2Fcoverage-guided-pbt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stevana%2Fcoverage-guided-pbt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stevana","download_url":"https://codeload.github.com/stevana/coverage-guided-pbt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246892848,"owners_count":20850850,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["coverage-guided","property-based-testing"],"created_at":"2024-10-28T08:54:30.330Z","updated_at":"2025-04-02T21:15:09.105Z","avatar_url":"https://github.com/stevana.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Coverage-guided property-based testing\n\n*Work in progress, please don't share, but do feel free to get involved!*\n\nAlmost ten years ago, back in 2015, Dan Luu wrote a\n[post](https://danluu.com/testing/) asking why coverage-guided property-based\ntesting wasn't a thing. \n\nIn this post I'll survey the coverage-guided landscape, looking at what was\nthere before Dan's post and what has happened since. I'll also show how to add\nbasic coverage-guidance to the first version of the\noriginal property-based testing tool, QuickCheck, in about 35 lines of code.\n\nUnlike many previous implementations of this idea, the technique used to\nimplement coverage-guidance is programming language agnostic and doesn't rely\non any language-specific instrumentation of the software under test.\n\n## Motivation\n\nBefore we start, let me try to motivate why one would want to combine\ncoverage-guided fuzzing and property-based testing to begin with.\n\nConsider the following example[^1], where an error is triggered if some input\nbyte array starts with the bytes `\"bad!\"`:\n\n```\nfunc sut(input []byte) {\n    if input[0] == 'b' {\n        if input[1] == 'a' {\n            if input[2] == 'd' {\n                if input[3] == '!' {\n                    panic(\"input must not be bad!\")\n                }\n            }\n        }\n    }\n}\n```\n\nWhat are the odds that a property-based testing tool (without\ncoverage-guidance) would be able to find the error?\n\nTo make the calculation easier, let's say that we always generate arrays of\nlength $4$. A byte consists of eight bits, so it has $2^8$ possible values.\nThat means that the probability is $\\frac{1}{2^8} \\cdot \\frac{1}{2^8} \\cdot\n\\frac{1}{2^8} \\cdot \\frac{1}{2^8} = (\\frac{1}{2^8})^4 = \\frac{1}{2^{32}}$ which\nis approximately $1$ in $4$ billion. In a realistic test suite, we wouldn't\nrestrict the length of the array to be $4$, and hence the probability will be\neven worse.\n\nWith coverage-guidance we keep track of inputs that resulted in increased\ncoverage. So, for example, if we generate the array `[]byte{'b'}` we get\nfurther into the nested ifs, and so we take note of that and start generating\nlonger arrays that start with `'b'` and see if we get even further, etc. By\nbuilding on previous successes in getting more coverage, we can effectively\nreduce the problem to only need $\\frac{1}{2^8} + \\frac{1}{2^8} + \\frac{1}{2^8} + \n\\frac{1}{2^8} = \\frac{1}{2^8} \\cdot 4 = \\frac{1}{2^{10}} = \\frac{1}{1024}$.\n\nIn other words coverage-guidance turns an exponential problem into a polynomial\nproblem!\n\n## Background and prior work\n\nThere's a lot to cover here, so I'll split it up in before and after\nDan's post.\n\n### Before 2015\n\nFuzzing has an interesting origin. It started as a class\n[project](http://pages.cs.wisc.edu/~bart/fuzz/CS736-Projects-f1988.pdf) in an\nadvanced operating systems course taught by Barton Miller at the University of\nWisconsin in 1988.\n\nThe project was inspired by the observation that back then, if you logged into\nyour workstation via a dail-up modem from home and it rained, then frequently\nrandom characters would appear in the terminal. The line noise wasn't the\nsurprising thing, but rather that the extra characters would sometimes crash\nthe program that they tried to invoke. Among these programs were basic\nutilities such as `vi`, `mail`, `cc`, `make`, `sed`, `awk`, `sort`, etc, and it\nwas reasonable to expect that these would give an error message rather than\ncrash and core dump if fed with some extra characters caused by the rain.\n\nSo the project set out to basically recreate what the rain did, but more\neffectively, but essentially generating random noise (stream of bytes) and\nfeeding that to different utilities and see if they crashed. A couple of years\nlater Barton et al published [*An empirical study of the reliability of UNIX\nutilities*](https://dl.acm.org/doi/10.1145/96267.96279) (1990) which documents\ntheir findings.\n\nInserting random characters was effective in finding corner cases where the\nprogrammers forgot to properly validate the input from the user. However it\nwouldn't trigger bugs hiding deeper under the surface, such as the `\"bad!\"`\nexample from the previous section. \n\nThis changed around 2007 when people [started\nthinking](https://lcamtuf.coredump.cx/afl/historical_notes.txt) about how\nfuzzing can be combined with [evolutionary\nalgorithms](https://en.wikipedia.org/wiki/Evolutionary_algorithm). The idea\nbeing that instead of generating random bytes all the time as with classical\nfuzzing, we can use coverage information from one test to mutate the input for\nthe next test. Or to use the evolution metaphor: seeds that lead to better\ncoverage are mutated with the hope that they will lead to even better coverage. \n\nOne of the first, and perhaps still most widely known, such *coverage-guided*\nfuzzers is Michał Zalewski's [AFL](https://lcamtuf.coredump.cx/afl/) (2013). To\nget a feel for how effective AFL-style coverage-guidance is, check out the list\nof [bugs](https://lcamtuf.coredump.cx/afl/#bugs) that it found and this post\nabout how it manages to figure out the [jpeg\nformat](https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thin-air.html)\non its own[^2].\n\nSince AFL is the tool that Dan explicitly mentions in his post, let's stop at\nthis point and go back to his point, before looking at what happened since\nwith coverage-guided fuzzers. \n\nRecall that Dan was asking why this idea of coverage-guidance wasn't present in\nproperty-based testing tools. I've written about the\n[history](https://stevana.github.io/the_sad_state_of_property-based_testing_libraries.html#the-history-of-property-based-testing)\nof property-based testing and explained how it\n[works](https://stevana.github.io/the_sad_state_of_property-based_testing_libraries.html#pure-property-based-testing-recap)\nalready, so I won't take up space by repeating myself here. Let's just note\nthat the [original\npaper](https://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf) on\nproperty-based testing was published in 2000. So when Dan wrote asking about\nthis question property-based testing would have been fifteen and AFL two years\nold.\n\nThe main difference between property-based testing and fuzzing is that\nfuzzing requires less work by the user. Simply hook up the byte generator to\nthe function that expects bytes as input and off it goes looking for crashes.\nProperty-based testing on the other hand can test functions that take arbitrary\ndata structures as input (not just bytes), but you have to describe how to\ngenerate such inputs. \n\nFuzzing only looks for crashes, while property-based testing lets you specify\narbitrary relations that should hold between the input and output of the system\nunder test. For example, we can generate binary search trees and check that\nafter we insert something into an arbitrary binary search tree then it will\nremain sorted (when we do an inorder traversal). Fuzzing can't check such\nproperties, and furthermore because they generate random bytes it's unlikely\nthat they'll even generate a valid binary search tree to begin with (without\nlots of coverage-driven testing).\n\nOn the other hand, the coverage of property-based tests is only as good as\nthe user provided generators. Corner cases where slightly modified data leads\nto e.g. exception handling is not explored automatically, and coverage\ninformation is not used to guide the input generation process.\n\nBy now we should have enough background to see that the idea of combining\ncoverage-guidance and property-based testing makes sense. Basically what we'd\nlike is to:\n\n1. Use user provided generators to kick start the exploration in the right\n   direction;\n2. Mutate the generated data, while preserving its type, to surface bugs that\n   user provided generators alone wouldn't have found;\n3. Use coverage information to iteratively get deeper into the state space of\n   the system under test, like in the \"bad!\" example from the motivation.\n\n### After 2015\n\nHaving covered what had happened before Dan's post, let's have a look at what\nhas happened in the ten years since his post. First off, it's worth noting that\nat some point Dan added an update to his post:\n\n\u003e \"Update: Dmitry Vyukov's Go-fuzz, which looks like it was started a month\n\u003e after this post was written, uses the approach from the proof of concept in\n\u003e this post of combining the sort of logic seen in AFL with a QuickCheck-like\n\u003e framework, and has been shown to be quite effective. I believe David R.\n\u003e MacIver is also planning to use this approach in the next version of\n\u003e hypothesis.\"\n\nSo let's start there, with Go-fuzz. At a first glance, all functions that are\ntested with Go-fuzz need to take an array of bytes as input. My initial thought\nwas, how can I write properties which involve generating more interesting data\nstructures? But it turns out it's\n[possible](https://news.ycombinator.com/item?id=40876822), as somebody pointed\nout in the comments of an old post of mine. Furthermore there are also\n[ways](https://adalogics.com/blog/structure-aware-go-fuzzing-complex-types) of\nmaking the fuzzer aware of more complex data structures. I haven't played\naround enough with this to be able to compare how well shrinking works yet\nthough, but overall I'd say this ticks the box of being a property-based\ntesting tool that is also coverage-guided.\n\nNext up Dan mentions Python's Hypothesis. I was searching through the\ndocumentation trying to find out how coverage-guidance works, but I couldn't\nfind anything. Searching through the repository I found the following [release\nnote](https://github.com/HypothesisWorks/hypothesis/pull/1564/commits/dcbea9148be3446392bc3af8892d49f3cc74fbe3)\n(2018):\n\n\u003e \"This release deprecates the coverage-guided testing functionality,\n\u003e as it has proven brittle and does not really pull its weight.\n\u003e \n\u003e We intend to replace it with something more useful in the future, but the\n\u003e feature in its current form does not seem to be worth the cost of using, and\n\u003e whatever replaces it will likely look very different.\"\n\nAs far as I can tell, it hasn't been reintroduced since. However it's possible\nto hook Hypothesis up to [use external\nfuzzers](https://hypothesis.readthedocs.io/en/latest/details.html#use-with-external-fuzzers).\nHypothesis already uses random bytes as basis for its generators, unlike\nQuickCheck which uses an integer seed, so I suppose that the external fuzzers\nessentially fuzz the random input bytes that in turn are used to generate more\nstructured input. \n\nTraditional fuzzers are usually designed to target a single binary, where as\nthe test suite which uses property-based testing typically has many properties.\nThe [HypoFuzz](https://hypofuzz.com/docs/features.html#fuzzer-details) tool\nworks around this mismatch by scheduling fuzzing time among the many Hypothesis\nproperties in your test suite.\n\nWhat else has happened since Dan's post?\n\nOne of the first things I noticed is that AFL is no longer\n[maintained](https://lcamtuf.coredump.cx/afl/):\n\n\u003e \"Note: AFL hasn't been updated for a couple of years; while it should still\n\u003e work fine, a more complex fork with a variety of improvements and additional\n\u003e features, known as AFL++, is available from other members of the community\n\u003e and is worth checking out.\" \n\nWhereas AFL is based on a single idea of how the fuzzer does its exploration\nwith very few knobs,\n[AFL++](https://www.usenix.org/system/files/woot20-paper-fioraldi.pdf) (2020)\nkeeps the basic AFL evolutionary algorithm structure, but incorporates a lot of\nnew research on other ways to explore the state space. For example, which seed\ngets scheduled and how many times it gets mutated per round are [two new\nparameters](https://mboehme.github.io/paper/CCS16.pdf) that can be tweaked to\nachieve different paths of exploration throughout the system under test.\n\nThe next thing I did was to search for \"coverage-guided property-based\ntesting\" in the academic literature.\n\nOne of the first papers I found was [*Coverage guided, property based\ntesting*](https://dl.acm.org/doi/10.1145/3360607) by Leonidas Lampropoulos,\nMichael Hicks, Benjamin C. Pierce (2019). In this paper FuzzChick, Coq/Rocq\nlibrary, that adds AFL-style coverage instrumentation to QuickChick (a Rocq\nQuickCheck clone) is presented. Unfortunately the only source code I could find\nlives in an [unmaintained\nbranch](https://github.com/QuickChick/QuickChick/compare/master...FuzzChick)\nthat [doesn't compile](https://github.com/QuickChick/QuickChick/issues/277).\n\nThe related works section of the paper has a couple of interesting\nreferences though.\n\nThe main inspiration for FuzzChick seems to have been Stephen Dolan et al's\nOCaml library called\n[Crowbar](https://github.com/ocaml/ocaml.org-media/blob/086fc25105cbccb188c28ec74126d72962921ff8/meetings/ocaml/2017/extended-abstract__2017__stephen-dolan_mindy-preston__testing-with-crowbar.pdf)\n(2017). Crowbar uses a stream of bytes to drive its generators, similar to\nHypothesis, and it's this stream that AFL is hooked up to. This indirection is\nCrowbar's (and by extension, I guess, also HypoFuzz's) biggest weakness. \n\nAFL is good at manipulating this byte stream, but because the bytes are\nnot used directly to test the system under test, but rather to generate\ndata which in turn is used for testing, some of its effectiveness is\nlost. This becomes particularly obvious when data structures with sparse\npre-conditions, e.g. sorted list or a binary search tree. That's what the\nauthors of FuzzChick say at least, while claiming that they addressed this\nweakness by doing type-aware mutations.\n\nThe other libraries that the paper mentions are from the imperative\nlanguage community. \n\nFor example [*JQF + Zest: Coverage-guided semantic fuzzing for\nJava*](https://github.com/rohanpadhye/jqf), \n[libfuzzer](https://llvm.org/docs/LibFuzzer.html) and it's successor\n[FuzzTest](https://github.com/google/fuzztest) (2022?) for C++.\n\nRust's `cargo fuzz` seems to build upon libfuzzer, see the chapter on\n[*Structure-aware fuzzing using libfuzzer-sys in\nRust*](https://rust-fuzz.github.io/book/cargo-fuzz/structure-aware-fuzzing.html)\nin the Rust Fuzz Book.\n\nThe FuzzTest README claims \"It is a first-of-its-kind tool that bridges\nthe gap between fuzzing and property-based testing\". I can't tell why\nthey would claim that, given that it appears to have been released in\n2022 and many of the tools we looked at above seem to have successfully\ncombined the two approaches before that. For example, how is it\ndifferent from Go-fuzz?\n\nIn my search I also found the paper [*MUTAGEN: Reliable Coverage-Guided,\nProperty-Based Testing using Exhaustive\nMutations*](https://www.mista.me/assets/pdf/icst23-preprint.pdf) by Agustín\nMista and Alejandro Russo (2023). This paper seems to build upon the FuzzChick\npaper, however it swaps out the AFL-style coverage instrumentation for the use\nof a GHC\n[plugin](https://github.com/OctopiChalmers/mutagen/blob/main/src/Test/Mutagen/Tracer/Plugin.hs)\nto annotate source code with coverage information of: function clauses, case\nstatements, and each branch of if-then-else expressions.\n\nImperative languages such as Go, Python, C++, Rust, and Java seem ahead of\nfunctional languages when it comes to combining coverage-guided fuzzing and\nproperty-based testing.\n\nLet's try to change that by implementing a small functional programming\nversion, based on the original property-based testing implementation.\n\n## Prototype implementation\n\n### Getting the coverage information\n\nOne key question we need to answer in order to be able to implement anything\nthat's coverage-guided is: where do we get the coverage information from?\n\nWhen I've been thinking about how to implement coverage-guided property-based\ntesting in the past, I always got stuck thinking that parsing the coverage\noutput from the compiler in between test case generation rounds would be\nannoying and slow.\n\nI thought that in the best case scenario the compiler might provide a\nlibrary which exposes the coverage information[^3].\n\nIt wasn't until I started researching this post that I realised that\nAFL, and most coverage-guided fuzzers since, actually inject custom\ncoverage capturing code into compiled programs at every branch point or\n[basic block](https://en.wikipedia.org/wiki/Basic_block). It's explained\nin more detail in the\n[whitepaper](https://lcamtuf.coredump.cx/afl/technical_details.txt). \n\nThe main reason they do it is because of performance, not because it's\nnecessarily easier, in fact I still don't understand exactly how it\nworks.\n\nIt wasn't until I read about Antithesis' [\"sometimes\nassertions\"](https://antithesis.com/docs/best_practices/sometimes_assertions.html)\nthat I started seeing a simple solution to the problem of collecting coverage\ninformation.\n\nTo understand how \"sometimes assertions\" work let's first recall how regular\nassertions, or \"always assertions\", work. If we add `assert b \"message\"`\nsomewhere in the code base and the boolean `b` evaluates to false at run-time\nthen the program will fail with `\"message\"`. \n\n\"Sometimes assertions\" are different in that they don't need to always hold.\nConsider the example:\n\n```\nfor (1..10000) {\n  c := flipCoin()\n  sometimesAssert (c == Heads) \"probably an unfair coin\"\n}\n```\n\nAbove the \"sometimes assertion\" will only fail if we flip 10000 tails.\n\nHow is this related to coverage though? If we sprinkle \"sometimes assertions\"\nat every branch point:\n\n```\nif b {\n  sometimesAssert True \"true branch\"\n} else {\n  sometimesAssert True \"false branch\"\n}\n```\n\nThen we'll get a failure when some branch hasn't been covered! The neat thing\nabout \"sometimes assertions\" is that we don't need to annotate every single\nbranch, we can annotate *interesting* points in our program, that's why we can\nthink of \"sometimes assertions\" as generalised coverage.\n\nThe final piece of the puzzle, and I think this is the only original idea that\nthis post adds[^4], is that property-based testing already has functionality for\nimplementing \"sometimes assertions\": the machinery for gathering run-time\nstatistics of the generated data!\n\nThis machinery is [crucial](https://www.youtube.com/watch?v=NcJOiQlzlXQ) for\nwriting good tests and has been part of the QuickCheck implementation since the\nvery first version[^5].\n\nSo the question is: can we implement coverage-guided property-based testing\nusing the internal notion of coverage that property-based testing already has?\n\n### The first version of QuickCheck\n\nBefore we answer the above question, let's remind ourselves of how a\nproperty-based testing library is implemented. For the sake of\nself-containment, let's reproduce the essential parts of QuickCheck as defined\nin the appendix of the original\n[paper](https://dl.acm.org/doi/10.1145/351240.351266) that first introduced\nproperty-based testing (ICFP, 2000).\n\n#### Generating input data\n\nLet's start with the generator[^6], which is used to generate random inputs to\nthe software under test:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=Gen}\n```\n\nSo a `Gen a` is basically a function from a size and a pseudo-random number\ngenerator into `a`. The pseudo-random number generator and size can be\naccessed using the following two functions:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=rand}\n```\n\nUsing these together with the `Functor`, `Applicative` and `Monad`\ninstances of `Gen` we can derive other useful combinators for generating\ndata:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=derivedCombinators}\n```\n\nInstead of defining generators directly for different datatypes,\nQuickCheck first wraps generators in a type class called\n`Arbitrary`[^7]:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=Arbitrary}\n```\n\n#### Specifying properties\n\nNext up, let's look at how properties are expressed. The `Property` type\nis a wrapper around `Gen Result`:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=Property}\n```\n\nWhere `Result` is defined as follows:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=Result}\n```\n\nThe idea being that `ok :: Maybe Bool` is `Nothing` if the input gets\ndiscarded and otherwise the boolean indicates whether the property\npassed or not. The `stamp` field is used to collect statistics about the\ngenerated test cases, while `arguments` contains all the generated\ninputs (or arguments) to the property.\n\nIn order to allow the user to write properties of varying arity another\ntype class is introduced:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=Testable}\n```\n\nThe key ingredient in the function instance of `Testable` is `forAll`:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=forAll}\n```\n\nWhich in turn depends on:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=evaluate}\n```\n\nOne last construct for writing properties that we need is the ability to\nadd assumptions or pre-conditions about the input:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=assuming}\n```\n\nNotice how if the input doesn't pass this test, then it will be\ndiscarded.\n\n#### Collecting statistics\n\nThe way we collect statistics about the generated data is through these\ntwo functions:\n\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=classify}\n```\n\n#### Running the tests\n\nFinally we have all the pieces we need to be able to actually run the\ntests. The testing can be configured:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=Config}\n```\n\nWhere `maxTest` is the amount of passing test cases that will be run,\n`maxFail` is the amount of tests that are allowed to be discarded,\n`size` is how the size parameter to the generator changes between tests,\nand `every` is used to print something (or not) between each test.\n\nThe tests themselves can now be run as follows:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=quickCheck}\n```\n\n### Example: traditional use of coverage\n\nOkay, so the above is the first version of the original property-based testing\ntool, QuickCheck. \n\nBefore we extend it with coverage-guidance, let's have a look at an example\nproperty and how collecting statistics is typically used to ensure good\ncoverage.\n\nThe example we'll have a look at is `insert`ing into an already sorted\nlist (from which insertion sort can be implemented):\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=insert}\n```\n\nIf we do so, then we resulting list should remain sorted:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=prop_insert1}\n```\n\nThis test passes:\n\n```\n\u003e\u003e\u003e quickCheck prop_insert\nOK, passed 100 tests.\n```\n\nWhat do the test cases that are generated look like? This is where\n`classify` comes in:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=prop_insert2}\n```\n\nRunning this property, we get some statistics about the generated data:\n\n```\n\u003e\u003e\u003e quickCheck prop_insert'\nOK, passed 100 tests.\n54% empty.\n27% singleton.\n19% short.\n```\n\nAs we can see, all of the lists that get generated are less than 3\nelements long! This is perhaps not what we expected. However if we\nconsider that precondition says that the list must be sorted, then it\nshould become clear that it's unlikely to generate such longer such\nlists completely by random[^9].\n\n### The extension to add coverage-guidance\n\nNow let's add coverage-guidance to it using the machinery for\ncollecting statistics about the generated data.\n\nThe function that checks a property with coverage-guidance slight different\nfrom `quickCheck`[^8]:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=coverCheck}\n```\n\nIn particular notice that instead of `Testable a` we use an explicit predicate\non a list of `a`, `[a] -\u003e Property`. The reason for using a list in the\npredicate is so that we can iteratively make progress, using the coverage\ninformation. We see this more clearly if we look at the coverage-guided\nanalogue of the `tests` function, in particular the `xs` parameter:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=testsC1}\n```\n\nThe other important difference is the `cov`erage parameter, which keeps track\nof how many things have been `classify`ed (the `stamps` parameter). Notice how\nwe only add the newly generated input, `x`, if the `cov`erage increases.\n\n### Example: using coverage to guide generation\n\nWe now have all the pieces to test the example from the\n[motivation](#motivation) section:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=bad}\n```\n\nThis property basically says that there's no string that's equal to `\"bad!\"`,\nwhich is obviously false. If we try to test this property using the unmodified\nfirst version of QuickCheck:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=testBad1}\n```\n\nWe'll see spin away, but not actually find the bad string:\n\n```\n\u003e\u003e\u003e testBad\n32301\n^CInterrupted.\n```\n\nI stopped it after about 32k tries, in theory we'd need more than 4 billion\nattempts to find the bad string using this approach.\n\nWhereas if we use coverage-guided generation:\n\n``` {.haskell include=src/QuickCheckV1.hs snippet=testBad2}\n```\n\nWe find the bad string pretty quickly. I'm using verbose output here so you can\nsee how it first find the `\"b\"`, then `\"ba\"`, etc[^10]:\n\n```\n\u003e\u003e\u003e testBad'\n0: \"n\"          23: \"#\"         46: \"bai\"       69: \"badb\"\n1: \"T\"          24: \"T\"         47: \"ba}\"       70: \"bad~\"\n2: \"L\"          25: \"T\"         48: \"bay\"       71: \"bada\"\n3: \"|\"          26: \"_\"         49: \"bac\"       72: \"bad%\"\n4: \"X\"          27: \"@\"         50: \"bak\"       73: \"bad9\"\n5: \"\\\"\"         28: \"}\"         51: \"ba`\"       74: \"badE\"\n6: \"e\"          29: \"y\"         52: \"bad\"       75: \"bad8\"\n7: \"G\"          30: \"-\"         53: \"bad8\"      76: \"bad{\"\n8: \"R\"          31: \"s\"         54: \"bade\"      77: \"badS\"\n9: \"}\"          32: \"b\"         55: \"bad0\"      78: \"badn\"\n10: \"C\"         33: \"bx\"        56: \"badA\"      79: \"bad?\"\n11: \"3\"         34: \"b9\"        57: \"badP\"      80: \"badn\"\n12: \"\u003e\"         35: \"bf\"        58: \"badQ\"      81: \"badq\"\n13: \"C\"         36: \"bl\"        59: \"bad0\"      82: \"bady\"\n14: \"J\"         37: \"ba\"        60: \"bade\"      83: \"badA\"\n15: \"9\"         38: \"baC\"       61: \"bad)\"      84: \"bad4\"\n16: \"=\"         39: \"baX\"       62: \"bado\"      85: \"bad;\"\n17: \"9\"         40: \"baA\"       63: \"badE\"      86: \"bad9\"\n18: \"L\"         41: \"baE\"       64: \"bad\\\"\"     87: \"badU\"\n19: \")\"         42: \"bay\"       65: \"bad@\"      88: \"bad!\"\n20: \"5\"         43: \"baX\"       66: \"bad{\"\n21: \"6\"         44: \"ba6\"       67: \"badX\"\n22: \"x\"         45: \"ba@\"       68: \"bado\"\nFalsifiable, after 88 tests:\n\"bad!\"\n```\n\nThe full source code is available\n[here](https://github.com/stevana/coverage-guided-pbt/blob/main/src/QuickCheckV1.hs).\n\n## Conclusion and further work\n\nWe've seen how to add converage-guidance to the first version of the first\nproperty-based testing tool, QuickCheck, in about 35 lines of code.\n\nCoverage-guidance effectively reduced a exponential problem into a polynomial\none, by building on previous test runs' successes in increasing the coverage.\n\nThe solution does change the QuickCheck API slightly by requring a property on\na list of `a`, rather than merely `a`, so it's not suitable for all properties.\n\nI think this limitation isn't so important, because going further I'd like to\napply coverage-guidance to testing stateful systems. When testing stateful\nsystems, which I've written about\n[here](https://stevana.github.io/the_sad_state_of_property-based_testing_libraries.html),\none always generates a list of commands anyway, so the limitation doesn't matter.\n\nA more serious limitation with the current approach is that it's too greedy and\nwill seek to maximise coverage, without ever backtracking. This means that it\ncan easily get stuck in local maxima. Consider the example:\n\n```\nif input[0] == 'o'\n  if input[1] == 'k'\n    return\nif input[0] == 'b'\n  if input[1] == 'a'\n    if input[2] == 'd'\n      error\n```\n\nIf we generate an input that starts with 'o' (rather than 'b'), then we'll get\nstuck never finding the error.\n\nReal coverage-guided tools, like AFL, will not get stuck like that. While I\nhave a variant of the code that can cope with this, I chose to present the\nabove greedy version because it's simpler. \n\nI might write another post with a more AFL-like solution at some later point,\nbut I'd also like to encourge others to port these ideas to your favorite\nlanguage and experiment!\n\n\n[^1]: This example is due to Dmitry Vyukov, the main author of\n    [go-fuzz](https://github.com/dvyukov/go-fuzz), but it's basically an easier\n    to understand version of the example from Dan Luu's post. For comparison,\n    here's Dan's example in full:\n\n    ```\n    // Checks that a number has its bottom bits set\n    func some_filter(x int) bool {\n    \tfor i := 0; i \u003c 16; i = i + 1 {\n    \t\tif !(x\u00261 == 1) {\n    \t\t\treturn false\n    \t\t}\n    \t\tx \u003e\u003e= 1\n    \t}\n    \treturn true\n    }\n    \n    // Takes an array and returns a non-zero int\n    func dut(a []int) int {\n    \tif len(a) != 4 {\n    \t\treturn 1\n    \t}\n    \n    \tif some_filter(a[0]) {\n    \t\tif some_filter(a[1]) {\n    \t\t\tif some_filter(a[2]) {\n    \t\t\t\tif some_filter(a[3]) {\n    \t\t\t\t\treturn 0 // A bug! We failed to return non-zero!\n    \t\t\t\t}\n    \t\t\t\treturn 2\n    \t\t\t}\n    \t\t\treturn 3\n    \t\t}\n    \t\treturn 4\n    \t}\n    \treturn 5\n    }\n    ```\n\n[^2]: For more details about how it works, see the [AFL\n    \"whitepaper\"](https://lcamtuf.coredump.cx/afl/technical_details.txt) and\n    its [mutation\n    heuristics](https://lcamtuf.blogspot.com/2014/08/binary-fuzzing-strategies-what-works.html).\n\n[^3]: In the case of Haskell, I didn't know that there was such a\n    library until read Shae \"shapr\" Erisson's post [*Run property tests\n    until coverage stops\n    increasing*](https://shapr.github.io/posts/2023-07-30-goldilocks-property-tests.html)\n    (2023). Note that Shae's post only uses coverage as a stopping\n    condition, not to actually drive the generation of test cases.\n\n[^4]: As I was writing up, I stumbled across the paper [*Ijon: Exploring Deep\n    State Spaces via Fuzzing*](https://ieeexplore.ieee.org/document/9152719)\n    (2020) which lets the user to add custom coverage annotations. HypoFuzz\n    also has this\n    [functionality](https://hypofuzz.com/docs/configuration.html#custom-coverage-events).\n\n[^5]: See the appendix of the original\n    [paper](https://dl.acm.org/doi/10.1145/351240.351266) that first introduced\n    property-based testing. It's interesting to note that the collecting\n    statistics functionality is older than shrinking.\n\n[^6]: We'll not talk about the `coarbitrary` method of the `Arbitrary` type\n    class, which is used to generate functions, in this post. \n\n[^7]: The reason for wrapping `Gen` in the `Arbitrary` type class is so\n    that generators don't have to be passed explicitly. Not everyone\n    agrees that this is a good idea, as type class instances cannot be\n    managed by the module system.\n\n[^8]: It might be interesting to note that we can implement this signature\n    using the original combinators:\n    ``` {.haskell include=src/QuickCheckV1.hs snippet=testsC2}\n    ```\n\n[^9]: The standard workaround here is to introduce a wrapper type for\n    which we write a custom generator which generates a random list and\n    then sorts it before returning. That way no pre-condition is\n    needed, as the input will be sorted by construction so to say.\n\n[^10]: To save vertical space I've also arranged the output in columns rather\n    than one per line.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstevana%2Fcoverage-guided-pbt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstevana%2Fcoverage-guided-pbt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstevana%2Fcoverage-guided-pbt/lists"}