{"id":22116307,"url":"https://github.com/jmid/wasm-prop-tester","last_synced_at":"2025-08-16T11:06:13.801Z","repository":{"id":82291668,"uuid":"290888221","full_name":"jmid/wasm-prop-tester","owner":"jmid","description":"A stack-driven generator of arbitrary WebAssembly programs","archived":false,"fork":false,"pushed_at":"2021-09-05T20:56:37.000Z","size":330,"stargazers_count":19,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-07-15T01:29:03.267Z","etag":null,"topics":["property-based-testing","property-testing","quickcheck","wasm","webassembly"],"latest_commit_sha":null,"homepage":"","language":"OCaml","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jmid.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-27T21:43:47.000Z","updated_at":"2024-11-05T00:37:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"8b808c85-0b23-4cd2-9913-b55e9090c9e8","html_url":"https://github.com/jmid/wasm-prop-tester","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jmid/wasm-prop-tester","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fwasm-prop-tester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fwasm-prop-tester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fwasm-prop-tester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fwasm-prop-tester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jmid","download_url":"https://codeload.github.com/jmid/wasm-prop-tester/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fwasm-prop-tester/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270702562,"owners_count":24630877,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-16T02:00:11.002Z","response_time":91,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["property-based-testing","property-testing","quickcheck","wasm","webassembly"],"created_at":"2024-12-01T12:26:37.361Z","updated_at":"2025-08-16T11:06:13.776Z","avatar_url":"https://github.com/jmid.png","language":"OCaml","funding_links":[],"categories":[],"sub_categories":[],"readme":"Property-Based Testing of WebAssembly [![Build Status](https://travis-ci.com/jmid/wasm-prop-tester.svg?branch=master)](https://travis-ci.com/jmid/wasm-prop-tester)\n========================================================\n\nThis project implements a stack-driven generator of arbitrary WebAssembly programs\ndescribed in the paper\n\n\u003e Árpád Perényi and Jan Midtgaard  \n\u003e Stack-Driven Program Generation of WebAssembly  \n\u003e APLAS'2020  \n\u003e https://janmidtgaard.dk/papers/Perenyi-Midtgaard%3aAPLAS20.pdf\n\nWe run each generated WebAssembly (Wasm) program on the engines underlying Chrome, Firefox, Safari, and Edge\nand on the [WebAssembly reference interpreter](https://github.com/WebAssembly/spec) \nand compare their output to help ensure consistent behaviour. The generator is stack-directed\nto produce programs that will pass validation (\"are type correct\"). A custom stack-directed shrinker\nreduces the often large random counterexample programs, without breaking validation.\n\nSurprisingly this black-box generator approach found 2 crashing bugs,\ndespite browser vendor efforts to fuzz test the engines with\ncoverage-aware (gray-box) fuzzers.\n\nThe code piggybacks on the [WebAssembly reference interpreter's](https://github.com/WebAssembly/spec)\nabstract syntax tree (in OCaml) and uses the [QCheck library](https://github.com/c-cube/qcheck) for property-based testing (QuickCheck).\n\n\nExternal Dependencies\n---------------------\n\n* OCaml and the [QCheck](https://github.com/c-cube/qcheck) package:\n  ```\n  opam install qcheck\n  ```\n\n* The WebAssembly reference interpreter from the [WebAssembly specification](https://github.com/WebAssembly/spec):\n  ```\n  opam install wasm\n  ```\n\n* A `bash` shell with a `cmp` command for diffing log files of observed outputs\n\n* A `timeout` command to break infinite loops\n  (on Mac OSX: install coreutils, then, e.g., `ln -s /opt/local/bin/gtimeout ~/bin/timeout`\n\n* [Node.js](https://nodejs.org/en/) to help transform a generated `.wasm` file into an independent JavaScript-file\n  suitable for running on a barebones JavaScript engine:\n  ```\n  sudo npm install -g npm\n  ```\n\n* JavaScript (engine) Version Updater: [jsvu](https://github.com/GoogleChromeLabs/jsvu)\n  ```\n  sudo npm install -g jsvu\n  jsvu --engines=chakra,javascriptcore,spidermonkey,v8\n  export PATH=\"$PATH:$HOME/.jsvu\"\n  ```\n\n\nAn Optional Dependency\n----------------------\n\nThe generator uses the reference interpreter for emitting the Wasm binary format (`.wasm`).\nHowever we have also used the [WebAssembly binary toolkit (wabt)](https://github.com/WebAssembly/wabt)\nto convert `.wat` to `.wasm`. This is a dependency to run our full internal testsuite.\n\nAssuming wabt is installed in the `wabt` sub-directory:\n```\n export PATH=\"$PATH:$PWD/wabt/bin\"\n```\n\n\nRunning\n-------\n\nWe have tested the generator under both Linux and Mac OSX.\nWith a recent OCaml installed, compiling should be as simple as:\n```\n make\n```\n\nOnce compiled and the `PATH` setup, you can generate 100 programs as follows:\n```\n ./main.native -v\n```\n\nFor fun, you can run a loop that continues restarting the test runner\nuntil it finds a counterexample:\n```\n while ./main.native -v; do :; done\n```\n\nIf `jsc` complains about an unsupported locale (and thus producing a different output)\n```\n export LC_ALL=\"C\"\n```\nshould make it behave.\n\n\nExamples\n--------\n\nHere is first an example of successful test run:\n```\n $ ./main.native -v\n random seed: 169405920\n generated error fail pass / total     time test name\n [✓]  100    0    0  100 /  100    56.8s compare engines\n ================================================================================\n success (ran 1 tests)\n```\nIn this case in less than 1 minute 100 modules were generated and had their output\ncompared across the reference interpreter and the four JavaScript engines\n(V8, JavaScriptCore, SpiderMonkey, Chakra) without finding any disagreements.\n\nWe can find the latest generated module in `tmp/tmp_module.{wat,wasm}`,\nits embeddings in `tmp/tmp_{v8,sm,jsc,ch}.js`, and the (normalized)\noutputs in `tmp/tmp_{v8,sm,jsc,ch,spec}`.\n\n\n\nHere's another example finding a disagreement:\n```\n $ ./main.native -v\n random seed: 252860053\n generated error fail pass / total     time test name\n [✗]   27    0    1   26 /  100    72.8s ref interpret vs. js-engines\n\n --- Failure --------------------------------------------------------------------\n\n Test ref interpret vs. js-engines failed (27 shrink steps):\n\n (module\n   (memory $0 4)\n   (data\n     0\n     (offset (i32.const 261_218))\n     \"\\69\\02\\96\\22\\d8\\19\\8c\\be\\82\\d1\\61\\67\\ba\\09\\86\\4a\"\n     \"\\c4\\53\\dc\\9f\\7e\\98\\df\\20\\78\\c2\\11\\dc\\e4\\8d\\f6\\f9\"\n     \"\\d4\\4b\\3c\\4b\\c3\\c5\\03\\c8\\c1\\16\\55\\03\\a8\\9f\\78\\52\"\n\n     ...\n\n     \"\\bd\\27\\a7\\dd\\5b\\39\\b4\\20\\42\\32\\b0\\c9\\6a\\ef\\23\\fe\"\n     \"\\0a\\ae\\86\\cd\\66\\f4\\ed\\02\\11\\43\\7e\\d2\\47\\95\\0c\\2d\"\n     \"\\38\\16\\29\\35\\25\\87\\86\\09\\cb\\96\\a1\\12\\06\\d7\\5c\"\n   )\n )\n\n ================================================================================\n failure (1 tests failed, 0 tests errored, ran 1 tests)\n```\nHere, on the 27th generated module a disagreement was found. Afterwards it took\n27 shrinking steps (reductions or simplications) to cut the counterexample down.\nIn total it took a little more than one minute.\n\nSince the last (unsuccessful) shrinking step overwrote the previous\noutput files, we save the previous disagreeing run in separate files.\nThe latest generated module illustrating disagreement is stored in `tmp/prev.{wat,wasm}`,\nits embeddings in `tmp/prev_{v8,sm,jsc,ch}.js`, and the (normalized)\noutputs in `tmp/prev_{v8,sm,jsc,ch,spec}`.\n\nWe can thus observe the output disagreement:\n```\n $ cat tmp/prev_{ch,jsc,sm,v8,spec}\n LinkError data segment does not fit memory\n LinkError data segment does not fit memory\n RuntimeError out of bounds memory access\n RuntimeError data segment does not fit memory\n LinkError data segment does not fit memory\n```\nHere we see that the counterexample triggers a `RuntimeError` exception\non SpiderMonkey and V8 but a `LinkError` exception on Chakra, JavaScriptCore,\nand in the reference interpreter. Furthermore, we failed to normalize\nSpiderMonkey's exception message to something agreeing with the rest.\n\nPrinting the unnormalized messages carried by each exception shows a\nrange of detail, revealing how it may be tricky to separate this\nparticular SpiderMoney error from others (from shortest to longest):\n\n- SM: `index out of bounds`\n- CH: `Data segment is out of range`\n- V8: `WebAssembly.Instance(): data segment is out of bounds`\n- JSC: `Invalid data segment initialization: segment of 927 bytes memory of 262144 bytes, at offset 261218, segment writes outside of memory (evaluating 'new WebAssembly.Instance(new WebAssembly.Module(buffer), importObject)')`\n\n\nIssues Found\n------------\n\n* SpiderMonkey: [Crash when start function is added to table](https://bugzilla.mozilla.org/show_bug.cgi?id=1545086)  (new, fixed)\n* JavaScriptCore: [Wasm engine segmentation fault](https://bugs.webkit.org/show_bug.cgi?id=202786)  (never confirmed, fixed)\n* JavaScriptCore: [Register allocation crash? FATAL: No color for %ftmp0](https://bugs.webkit.org/show_bug.cgi?id=209294)  (known, already fixed)\n* JavaScriptCore: [br_table behavior](https://bugs.webkit.org/show_bug.cgi?id=209333)  (new, confirmed, fixed)\n* Chakra: [Compile-time rejection of unreachable tee_local](https://github.com/microsoft/ChakraCore/issues/6185)  (known)\n* Chakra: [Stack overflow crash](https://github.com/microsoft/ChakraCore/issues/6524)  (new)\n\n* Wasmer: [Type mismatch on validation](https://github.com/wasmerio/wasmer/issues/1740)  (known, fixed)\n\n### Other inconsistencies:\n\n* JavaScriptCore: [Different exception name properties](https://bugs.webkit.org/show_bug.cgi?id=204054)\n* Different stack overflow exceptions (described in the paper, not reported)\n* Different data segment exceptions (described in the paper and above, not reported)\n\n\nIf you find more errors using the generator please let us know.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmid%2Fwasm-prop-tester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjmid%2Fwasm-prop-tester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmid%2Fwasm-prop-tester/lists"}