{"id":22116318,"url":"https://github.com/jmid/mutaml","last_synced_at":"2025-04-09T18:23:40.536Z","repository":{"id":45590375,"uuid":"420277722","full_name":"jmid/mutaml","owner":"jmid","description":"An OCaml mutation tester","archived":false,"fork":false,"pushed_at":"2025-03-19T11:27:03.000Z","size":274,"stargazers_count":67,"open_issues_count":10,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-02T17:04:59.017Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"OCaml","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jmid.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-23T00:36:07.000Z","updated_at":"2025-03-19T11:27:07.000Z","dependencies_parsed_at":"2025-03-19T12:35:56.145Z","dependency_job_id":null,"html_url":"https://github.com/jmid/mutaml","commit_stats":{"total_commits":129,"total_committers":4,"mean_commits":32.25,"dds":"0.11627906976744184","last_synced_commit":"cb731549abbdd67f15a130df3a850d4445781127"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fmutaml","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fmutaml/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fmutaml/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmid%2Fmutaml/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jmid","download_url":"https://codeload.github.com/jmid/mutaml/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248086099,"owners_count":21045275,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-01T12:26:46.859Z","updated_at":"2025-04-09T18:23:40.510Z","avatar_url":"https://github.com/jmid.png","language":"OCaml","funding_links":[],"categories":[],"sub_categories":[],"readme":"Mutaml: A Mutation Tester for OCaml [![Main CI workflow](https://github.com/jmid/mutaml/actions/workflows/ci.yml/badge.svg)](https://github.com/jmid/mutaml/actions/workflows/ci.yml)\n===============================================\n\nMutaml is a mutation testing tool for OCaml.  \nBriefly, that means Mutaml tries to change your code randomly to see\nif the changes are caught by your tests.\n\n![](demo.gif)\n\nIn more detail: [Mutation testing](https://en.wikipedia.org/wiki/Mutation_testing) is\na form of fault injection used to assess the quality of a program's\ntest suite. Mutation testing works by repeatedly making small, breaking\nchanges to a program's text, such as turning a `+` into `-`, negating\nthe condition of an `if-then-else`, ..., and subsequently rerunning\nthe test suite to see if each such 'mutant program' is 'killed'\n(caught) by one or more tests in the test suite. By finding examples of\nuncaught wrong behaviour, mutation testing can thereby reveal\nlimitations of an existing test suite and indirectly suggest\nimprovements.\n\nSince OCaml already prevents many potential programming errors at compile\ntime due to its strong type system, pattern-match compiler warnings, etc. Mutaml\nfavors mutations that\n- preserve typing and\n- would not be caught statically, e.g., changes in the values computed.\n\nMutaml consists of:\n\n - a [`ppxlib`](https://github.com/ocaml-ppx/ppxlib)-preprocessor that\n   first transforms the program under test.\n - `mutaml-runner` that loops through a range of possible program mutations,\n   and saves the output from running the test suite on each of the mutants\n - `mutaml-report` that prints a test report to the console.\n\n\nInstallation:\n-------------\n\nYou can install `mutaml` with a single `opam` command:\n\n```\n$ opam install mutaml\n\n```\n\nAlternatively, you can also install it from a clone of the repository:\n\n```\n$ git clone https://github.com/jmid/mutaml.git\n$ cd mutaml\n$ opam install .\n```\n\n\nInstructions:\n-------------\n\nHow you can use `mutaml` depends on your project's build setup.\nFor now it has only been tested with `dune`, but it should work\nwith other build systems supporting an explicit two-staged build\nprocess.\n\n\n### Using Mutaml with `dune`\n\n1. Mark the target code for instrumentation in your `dune` file(s):\n   ```\n   (library\n     (public_name your_library)\n     (instrumentation (backend mutaml)))\n   ```\n   Using `dune`'s [`instrumentation` stanza](https://dune.readthedocs.io/en/stable/instrumentation.html), your project's code is\n   only instrumented when you pass the `--instrument-with mutaml`\n   option.\n\n\n2. Compile your test code with `mutaml` instrumentation enabled:\n   ```\n   $ dune build test --instrument-with mutaml\n   ```\n   assuming you have a `test/mytests.ml` test driver.\n   This creates/overwrites an individual `lib.muts` file for each\n   instrumented `lib.ml` file and an overview file\n   `mutaml-mut-files.txt` listing them.\n   These files are written to `dune`'s current build context.\n\n\n3. Start `mutaml-runner`, passing the name of the test executable to run:\n   ```\n   $ mutaml-runner _build/default/test/mytests.exe\n   ```\n   This reads from the files written in step 2. Running the command also\n   creates/overwrites the file `mutaml-report.json`.\n   You can also pass a command that runs the executable through `dune`\n   if you prefer:\n   ```\n   $ mutaml-runner \"dune exec --no-build test/mytests.exe\"\n   ```\n\n4. Generate a report, optionally passing the json-file\n   (`mutaml-report.json`) created above:\n   ```\n   $ mutaml-report\n   ```\n   By default this prints `diff`s for each mutation that flew under\n   the radar of your test suite. The `diff` output can be suppressed by\n   passing `--no-diff`.\n\n\nSteps 3 and 4 output a number of additional files.\nThese are all written to a dedicated directory named `_mutations`.\n\n\n\nInstrumentation Options and Environment Variables\n-------------------------------------------------\n\nThe preprocessor's behaviour can be configured through either\nenvironment variables or instrumentation options in the `dune` file:\n\n- `MUTAML_SEED` - an integer value to seed `mutaml-ppx`'s randomized\n  mutations (overridden by instrumentation option `-seed`)\n- `MUTAML_MUT_RATE` - a integer between 0 and 100 to specify the\n  mutation frequency (0 means never and 100 means always - overridden\n  by instrumentation option `-mut-rate`)\n- `MUTAML_GADT` - allow only pattern mutations compatible with GADTs\n  (`true` or `false`, overridden by instrumentation option `-gadt`)\n\n\nFor example, the following `dune` file sets all three instrumentation\noptions:\n```\n (executable\n  (name test)\n  (instrumentation (backend mutaml -seed 42 -mut-rate 75 -gadt false))\n )\n```\nWe could achieve the same behaviour by setting three environment\nvariables:\n```bash\n  $ export MUTAML_SEED=42\n  $ export MUTAML_MUT_RATE=75\n  $ export MUTAML_GADT=false\n```\nIf you do both, the values passed as instrumentation options in the\n`dune` file take precedence.\n\n\nRunner Options and Environment Variables\n----------------------------------------\n\nBy default, `mutaml-runner` expects to find the preprocessor's output\nfiles in the default build context `_build/default`. This can be\nconfigured via an environment variable or a command-line option, e.g.,\nif [instrumentation is enabled via another `dune-workspace` build context](https://dune.readthedocs.io/en/stable/instrumentation.html#enabling-disabling-instrumentation):\n\n- `MUTAML_BUILD_CONTEXT` - a path prefix string (overridden by\n  command-line option `--build-context`)\n\n`mutaml-runner` also repeats test suite runs for all instrumented\n`lib.ml` files by default. An option `--muts muts-file` is available\nto enable more targeted mutation testing. Running, e.g.,\n```\nmutaml-runner --muts lib/lib2.muts _build/default/test/mytests.exe\n```\nwill only consider mutations of the corresponding library\n`lib/lib2.ml`, which the runner searches for in the build context.\n\n\nReport Options and Environment Variables\n----------------------------------------\n\nCurrently `mutaml-report` uses `diff --color -u` as its default\ncommand to print `diff`s. It falls back to `diff -u` when the\nenvironment variable `CI` is `true`. The used command can also be\nconfigured with an environment variable:\n\n- `MUTAML_DIFF_COMMAND` - the command and options to use instead,\n  e.g. `MUTAML_DIFF_COMMAND=\"diff -U 5\"` will disable colored outputs\n  and add 5 lines of unified context. Mutaml expects the specified\n  command to support `--label` options.\n\nPassing the option `--no-diff` to `mutaml-report` prevents any\nmutation `diff`s from being printed.\n\n\n\nStatus\n------\n\nThis is an *alpha* release. There are therefore rough edges:\n\n- Mutaml is designed to avoid repeated recompilation for each\n  mutation. It does so by writing files during preprocessing which are\n  later read during the `mutaml-runner` testing loop. As a consequence,\n  if you attempt to merge steps 2 and 3 above into one step this will not work:\n  ```\n  $ mutaml-runner \"dune test --force --instrument-with mutaml\"\n  ```\n  The preprocessor in this case only writes the relevant files when\n  `mutaml-runner` first calls the command, and thus *after* it needs the\n  information contained in the files...\n\n- There are [issues to force `dune` to\nrebuild](https://github.com/ocaml/dune/issues/4390). This can affect\n  Mutaml, e.g., in case just an environment variable changed. `dune\n  clean` is a crude but effective work-around to this issue.\n\n- The output files to `_build/default` are not registered with `dune`.\n  This means rerunning steps 2,3,4 above will fail, as the additional\n  output files in `_build/default` are not cached by `dune` and hence\n  deleted. Again `dune clean` is a crude but effective work-around.\n\n- ...\n\n\nMutations should not introduce compiler errors, be it type errors or\nfrom the pattern-match compiler. If you encounter a situation where\nthis happens please report it in an issue.\n\n\nAcknowledgements\n----------------\n\nMutaml was developed with support from the [OCaml Software Foundation](https://ocaml-sf.org/).\nWhile developing it, I also benefitted from studying the source code\nof [bisect_ppx](https://github.com/aantron/bisect_ppx).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmid%2Fmutaml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjmid%2Fmutaml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmid%2Fmutaml/lists"}