{"id":22116317,"url":"https://github.com/jmid/efftester","last_synced_at":"2025-10-16T21:29:53.800Z","repository":{"id":82291516,"uuid":"90962094","full_name":"jmid/efftester","owner":"jmid","description":"Effect-Driven Compiler Tester for OCaml","archived":false,"fork":false,"pushed_at":"2019-09-25T20:03:55.000Z","size":30,"stargazers_count":38,"open_issues_count":0,"forks_count":5,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-07-25T13:41:17.916Z","etag":null,"topics":["compiler-testing","ocaml","property-based-testing","quickcheck"],"latest_commit_sha":null,"homepage":null,"language":"OCaml","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jmid.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-11T09:29:57.000Z","updated_at":"2023-10-12T07:48:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"4258cb8d-ef49-4568-a69d-a29dd91c4fa0","html_url":"https://github.com/jmid/efftester","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jmid/efftester","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fefftester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fefftester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fefftester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fefftester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jmid","download_url":"https://codeload.github.com/jmid/efftester/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fefftester/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268199866,"owners_count":24211826,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-01T02:00:08.611Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compiler-testing","ocaml","property-based-testing","quickcheck"],"created_at":"2024-12-01T12:26:44.316Z","updated_at":"2025-10-16T21:29:53.699Z","avatar_url":"https://github.com/jmid.png","language":"OCaml","readme":"Effect-Driven Compiler Tester:\n==============================\n\nThis is a prototype implementation of a compiler testing approach\ndescribed in the paper:\n\n\u003e  \"Effect-Driven QuickChecking of Compilers\",\n\u003e  Jan Midtgaard, Mathias Nygaard Justesen, Patrick Kasting, Flemming Nielson, Hanne Riis Nielson,\n\u003e  ICFP'17,\n\u003e  [http://janmidtgaard.dk/papers/Midtgaard-al:ICFP17-full.pdf](http://janmidtgaard.dk/papers/Midtgaard-al%3aICFP17-full.pdf) (full version)\n\n\nWe suggest to generate programs following a dedicated type and effect\nsystem that determines whether a program has evaluation-order\ndependence. By reading the type-and-effect-system relation bottom up,\nin a goal-directed manner we can generate programs that are free of\nevaluation-order dependence. We can thus run such programs with both\nthe bytecode and native code backends of OCaml, log their observable\nbehaviour to a file, and `diff` the two behaviours.\n\n\nBuilding and running:\n---------------------\n\nThe tester requires OCaml, the QCheck library, and a *nix-like platform with `diff` and `/dev/null`.\nThe below output is obtained with QCheck version 0.5.2. If compiled as is with a later QCheck version\n(known to produce more output) the resulting program may produce a somewhat garbled output.\n\n\nTo build simply run `make`.\n\nNow run `effmain.native -v` (the native code build is recommended for speed reasons).\nThis should produce output along these lines:\n\n```\n$ ./effmain.native -v\nrandom seed: 228021772\n...........................................................................x...x...x...x...x...x...x...x...x...x....x...x..x....x....x.....x..x...x..x......x.....x.....x.....x.....x.....x.....x.....x.....x.....x.....x.....x.....x.....x.....\nlaw bytecode/native backends agree: 76 relevant cases (76 total)\n\n  test `bytecode/native backends agree`\n  failed on ≥ 1 cases:\n  Some (let o = (let j = List.hd [] in fun w -\u003e int_of_string) \"\" in 0) (after 32 shrink steps)\n  \nfailure (1 tests failed, ran 1 tests)\n```\n\nInterestingly, the produced counterexamples such as\n`let o = (let j = List.hd [] in fun w -\u003e int_of_string) \"\" in 0`\nabove illustrate observable differences in the executables produced by\nthe two backend, aka, bugs. This particular one is a variant of MPR#7531\nlisted below.\n\nYou can provide a known seed with the flag `-s`: `./effmain.native -v -s 228021772`\nto reproduce an earlier run.\n\nWhen invoked without a command-line seed QCheck is seeded differently\nfor each run and will therefore (most likely) produce different\nresults for each of them:\n- each run may reveal different errors or\n- no bugs are found in the given run.\n\nThe reproducability with respect to a given seed should be taken with\na large grain of salt. A seed may still yield different results\ndepending on the underlying architecture, OS, bit-width,\nOCaml-version, and QCheck-version. Furthermore it is (all too) easy to\ninvalidate previous seeds by any adjustments to\n- the generator -- as the adjusted version may utilize/interpret the\n  old seed differently,\n- the generator's initial environment -- as choices herein may come\n  out differently for a smaller/bigger/different environment,\n- the shrinker -- as the adjusted version may shrink a bigger\n  problematic counterexample to a different minimal example.\n\nThe generated programs are written to `testdir/test.ml` and the\ncompiled versions are named `testdir/native` and `testdir/byte`.\nThese are overwritten on each iteration. As such, the last version\nremaining rarely represents a counterexample. Instead it represents\nthe shrinker's last (failed) attempt to shrink the minimal\ncounterexample one step further.\n\nThe current version tests OCaml version 4.04.0's native-code compiler\nwith flag `-O3` against the bytecode compiler. It was originally\ntested with version 4.02.3 which doesn't support the `-O3` flag. So\nfar the tester has found the same bugs (below) related to erroneous\noptimization in 4.02.3 as in 4.04.0 with the `-O3` flag.\n\nIf you find more errors using this approach please let me know.\n\n\nBugs found:\n-----------\n\nMPR#7531  https://caml.inria.fr/mantis/view.php?id=7531  Delayed effects in partially applied functions\n\nMPR#7533  https://caml.inria.fr/mantis/view.php?id=7533  Wrong code generation of `e / e'` and `e mod e'` with effectful `e`\n\nMPR#7201  https://caml.inria.fr/mantis/view.php?id=7201  Wrong optimization of `0 / e`\n\t  \t\t\t\t\t\t (reported by way of Jean-Christophe Filliatre)\n\n   #8864  https://github.com/ocaml/ocaml/issues/8864     Left-shift of 0 yields negative number\n                                                         (found by [Murilo Giacometti Rocha](https://github.com/murilogiacometti))\n\nA quick port to [js_of_ocaml](http://ocsigen.org/js_of_ocaml/) revealed the following 2 bugs (reported together):\n \nhttps://github.com/ocsigen/js_of_ocaml/issues/584  function difference+equality exception removed\n\nA quick port to [BuckleScript](https://bucklescript.github.io/) revealed the following 8 bugs:\n\nhttps://github.com/bloomberg/bucklescript/issues/1667 failure: internal compiler Failure\n\nhttps://github.com/bloomberg/bucklescript/issues/1692 BuckleScript produces incorrect JavaScript\n\nhttps://github.com/BuckleScript/bucklescript/issues/1759 comparison+difference of units yields different results\n\nhttps://github.com/BuckleScript/bucklescript/issues/1760 div+mod: Division_by_zero removed\n\nhttps://github.com/BuckleScript/bucklescript/issues/1761 comp: function comparison succeeds\n      \nhttps://github.com/BuckleScript/bucklescript/issues/1762 order: bool_of_string erroneously optimized\n\n\nKnown bugs recreated:\n---------------------\n\nGPR#954  https://github.com/ocaml/ocaml/pull/954  Wrong optimisation of `0 mod e`\n                                                  (fixes both the division and mod cases)\n\nGPR#956  https://github.com/ocaml/ocaml/pull/956  Keep possibly-effectful expressions when optimizing multiplication by zero\n\n\n\nObserving the generated programs:\n---------------------------------\n\nTo invoke the generator directly, first `make eff` and then start\n`ocaml` from within this directory (this will load modules suitably\nvia `.ocamlinit`).\n\nTo generate an arbitrary program you can now invoke `Gen.generate1\nterm_gen.gen` (wrapped with a suitable string coersion to make the\ngenerator's output more understandable).  The output changes per each\ninvocation:\n\n```\n$ ocaml\n        OCaml version 4.04.0\n\n[some lines about loaded libs omitted]\n\n# Print.option (toOCaml ~typeannot:false) (Gen.generate1 term_gen.gen);;\n- : string = \"Some (exit ((fun n -\u003e (-98051756132636271)) string_of_bool))\"\n\n# Print.option (toOCaml ~typeannot:false) (Gen.generate1 term_gen.gen);;\n- : string =\n\"Some ((mod) (pred (let h = if true then exit ((/) (if false then 1830787755246062127\n else (-2895157864674163253)) (lnot ((asr) (-3089269914618536456) 93))) else \n let w = print_endline (let i = lnot 996022529208063915 in string_of_int 25) in \n string_of_bool (exit (abs (-427726557501168681))) in ( * ) (pred (lnot (List.hd \n (List.hd (exit 991))))) (lnot 5))) ((lsr) 225 755))\"\n```\n\n\nTo observe the distribution of the effect-driven program generator you\ncan build a different target with `make stat` which results in a\n`effstat.native` executable. This program runs a constant true test\nover generated terms while logging the size of each term.\n\nA bashscript `runstat.sh` will then run `./effstat.native -v`, log the\noutput, process it with `sed`, and pass the output to the program\n'ministat'.\n\n`runstat.sh` itself requires a *nix platform with bash and 'ministat'\ninstalled.\n\nWhen run, `runstat.sh` should produce output along these lines (you\ncan adjust the 50 generated terms in `src/effstat.ml`):\n```\n$ ./runstat.sh \nx stat_size.txt\n+-------------------------------------------------------------------------------------------------------------+\n|             x                                                                                               |\n|            xx                                                                                               |\n|            xx                                                                                               |\n|           xxx                                                                                               |\n|          xxxx                                                                                               |\n|          xxxx                                                                                               |\n|          xxxx                                                                                               |\n|          xxxx                                                                                               |\n|          xxxx    x x              xx                                                                        |\n|          xxxxx   xxx    x x      xxx                 x                            x x     x                x|\n||____________M_________A_____________________|                                                               |\n+-------------------------------------------------------------------------------------------------------------+\n    N           Min           Max        Median           Avg        Stddev\nx  50             1           639            16         80.82     148.41691\n```\n\nThe very latest version of the QCheck library can provide similar\noutput and thereby removes the need for the above bashscript.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmid%2Fefftester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjmid%2Fefftester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmid%2Fefftester/lists"}