https://github.com/ropensci-review-tools/autotest
Automatic testing of R packages
https://github.com/ropensci-review-tools/autotest
Last synced: 5 months ago
JSON representation
Automatic testing of R packages
- Host: GitHub
- URL: https://github.com/ropensci-review-tools/autotest
- Owner: ropensci-review-tools
- Created: 2020-07-03T07:32:44.000Z (almost 5 years ago)
- Default Branch: main
- Last Pushed: 2024-07-10T11:54:51.000Z (10 months ago)
- Last Synced: 2024-08-13T07:14:02.956Z (8 months ago)
- Language: R
- Homepage: https://docs.ropensci.org/autotest
- Size: 1.74 MB
- Stars: 54
- Watchers: 3
- Forks: 5
- Open Issues: 18
-
Metadata Files:
- Readme: README.Rmd
- Codemeta: codemeta.json
Awesome Lists containing this project
- jimsghstars - ropensci-review-tools/autotest - Automatic testing of R packages (R)
README
---
title: "autotest"
output:
md_document:
variant: gfmrmarkdown::html_vignette:
self_contained: no
---# autotest
```{r setup, include=FALSE}
knitr::opts_chunk$set (
collapse = TRUE,
warning = TRUE,
message = TRUE,
width = 120,
comment = "#>",
fig.retina = 2,
fig.path = "README-"
)
```[](https://github.com/ropensci-review-tools/autotest/actions?query=workflow%3AR-CMD-check)
[](https://codecov.io/gh/ropensci-review-tools/autotest)
[](https://www.repostatus.org/#concept)Automatic mutation testing of R packages. Mutation in the sense of mutating
inputs (parameters) to function calls. `autotest` primarily works by scraping
documented examples for all functions, and mutating the parameters input to
those functions.## Installation
The easiest way to install this package is via the associated
[`r-universe`](https://ropensci-review-tools.r-universe.dev/ui#builds). As
shown there, simply enable the universe with```{r options, eval = FALSE}
options (repos = c (
ropenscireviewtools = "https://ropensci-review-tools.r-universe.dev",
CRAN = "https://cloud.r-project.org"
))
```And then install the usual way with,
```{r install, eval = FALSE}
install.packages ("autotest")
```Alternatively, the package can be installed by running one of the following
lines:```{r gh-installation, eval = FALSE}
# install.packages("remotes")
remotes::install_git ("https://git.sr.ht/~mpadge/autotest")
remotes::install_bitbucket ("mpadge/autotest")
remotes::install_gitlab ("mpadge/autotest")
remotes::install_github ("ropensci-review-tools/autotest")
```The package can then be loaded the usual way:
```{r load-fakey, eval = FALSE}
library (autotest)
```
```{r load, echo = FALSE, message = FALSE}
devtools::load_all (".", export_all = FALSE)
```## Usage
The simply way to use the package is
```{r autotest-example, eval = FALSE}
x <- autotest_package ("")
```The main argument to the [`autotest_package()`
function](https://docs.ropensci.org/autotest/reference/autotest_package.html)
can either be the name of an installed package, or a path to a local directory
containing the source for a package. The result is a `data.frame` of errors,
warnings, and other diagnostic messages issued during package `auotest`-ing.
The function has an additional parameter, `functions`, to restrict tests to
specified functions only.By default,
[`autotest_package()`](https://docs.ropensci.org/autotest/reference/autotest_package.html)
returns a list of all tests applied to a package without actually running them.
To implement those tests, set the parameter `test` to `TRUE`. Results are only returned for tests
in which functions do not behave as expected, whether through triggering
errors, warnings, or other behaviour as described below. The ideal behaviour of
`autotest_package()` is to return nothing (or strictly, `NULL`), indicating
that all tests passed successfully. See the [main package
vignette](https://docs.ropensci.org/autotest/articles/autotest.html) for an
introductory tour of the package.## What is tested?
The package includes a function which lists all tests currently implemented.
```{r autotest_types}
autotest_types ()
```That functions returns a [`tibble`](https://tibble.tidyverse.org) describing
`r nrow(autotest_types())` unique tests. The default behaviour of
[`autotest_package()`](https://docs.ropensci.org/autotest/reference/autotest_package.html)
with `test = FALSE` uses these test types to identify which tests will be
applied to each parameter and function. The table returned from
[`autotest_types()`](https://docs.ropensci.org/autotest/reference/autotest_types.html)
can be used to selectively switch tests off by setting values in the `test`
column to `FALSE`, as demonstrated below.## How Does It Work?
The package works by scraping documented examples from all `.Rd` help files,
and using those to identify the types of all parameters to all functions. Usage
therefore first requires that the usage of all parameters be demonstrated in
example code.As described above, tests can also be selectively applied to particular
functions through the parameters `functions`, used to nominate functions to
include in tests, or `exclude`, used to nominate functions to exclude from
tests. The following code illustrates.```{r stats-var-no-test, fig.show = "hide"}
x <- autotest_package (package = "stats", functions = "var", test = FALSE)
print (x)
```Testing the `var` function also tests `cor` and `cov`, because these are all
documented within a single `.Rd` help file. Typing `?var` shows that the help
topic is `cor`, and that the examples include the three functions, `var`,
`cor`, and `cov`. That result details the `r nrow (x)` tests which would be
applied to the `var` function from the `stats` package. These `r nrow (x)`
tests yield the following results when actually applied:```{r stats-var-test}
y <- autotest_package (package = "stats", functions = "var", test = TRUE)
print (y)
```And only `r nrow (y)` of the original `r nrow (x)` tests produced unexpected
behaviour. There were in fact only `r length (unique (y$operation))` kinds of
tests which produced these `r nrow (y)` results:```{r unique-operations}
unique (y$operation)
```One of these involves conversion of a vector to a list-column representation
(via `I(as.list())`). Relatively few packages accept this kind of input,
even though doing so is relatively straightforward. The following lines
demonstrate how these tests can be switched off when `autotest`-ing a package.
The `autotest_types()` function, used above to extract information on all types
of tests, also accepts a single argument listing the `test_name` entries of any
tests which are to be switched off.```{r stats-var-test-switch}
types <- autotest_types (notest = "vector_to_list_col")
y <- autotest_package (
package = "stats", functions = "var",
test = TRUE, test_data = types
)
print (y)
```Those tests are still returned from `autotest_package()`, but with `test =
FALSE` to indicate they were not run, and a `type` of "no_test" rather than the
previous "diagnostic".## Can `autotest` automatically create tests in my `tests` directory?
Not yet, but that should be possible soon. In the meantime, there are
[`testthat`](https://testthat.r-lib.org) expectations, listed in the [main
package
functions](https://docs.ropensci.org/autotest/reference/index.html),
which enable `autotest` to be used in a package's test suite.## Prior work
1. The
[`great-expectations`](https://github.com/great-expectations/great_expectations)
framework for python, described in [this medium
article](https://medium.com/@expectgreatdata/down-with-pipeline-debt-introducing-great-expectations-862ddc46782a).
2. [`QuickCheck`](https://hackage.haskell.org/package/QuickCheck) for Haskell
3. [`mutate`](https://github.com/mbj/mutant) for ruby
4. [`mutant`](https://github.com/ropensci/mutant) for mutation of R code itself## Code of Conduct
Please note that this package is released with a [Contributor Code of
Conduct](https://ropensci.org/code-of-conduct/). By contributing to this
project, you agree to abide by its terms.## Contributors
All contributions to this project are gratefully acknowledged using the [`allcontributors` package](https://github.com/ropenscilabs/allcontributors) following the [all-contributors](https://allcontributors.org) specification. Contributions of any kind are welcome!
### Code
### Issue Authors
### Issue Contributors
