Automatic mutation testing of R packages. Mutation in the sense of mutating inputs (parameters) to function calls, rather than mutation of underlying code (see, for example, mutant for that). autotest primarily works by scraping documented examples for all functions, and mutating the parameters input to those functions.
This package is very unstable and subject to ongoing development (Feb, 2021)
Installation
Not yet on CRAN, so must be installed from remote repository host systems using any one of the following options:
# install.packages("remotes")
remotes::install_git("https://git.sr.ht/~mpadge/autotest")
remotes::install_bitbucket("mpadge/autotest")
remotes::install_gitlab("mpadge/autotest")
remotes::install_github("ropenscilabs/autotest")The package can then be loaded the usual way:
Usage
The simply way to use the package is
x <- autotest_package ("<package>")The argument to autotest_package() can either be the name of an installed package, or a path to a local directory containing the source for a package. The result is a data.frame of errors, warnings, and other diagnostic messages issued during package auotest-ing. See the main package vignette for an introductory tour of the package.
What is tested?
The package includes a function which lists all tests currently implemented.
autotest_types ()
#> # A tibble: 25 x 8
#> type test_name fn_name parameter parameter_type operation content test
#> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <lgl>
#> 1 dummy rect_as_o… <NA> <NA> rectangular Convert on… "check f… TRUE
#> 2 dummy rect_comp… <NA> <NA> rectangular Convert on… "expect … TRUE
#> 3 dummy rect_comp… <NA> <NA> rectangular Convert on… "expect … TRUE
#> 4 dummy rect_comp… <NA> <NA> rectangular Convert on… "expect … TRUE
#> 5 dummy extend_re… <NA> <NA> rectangular Extend exi… "(Should… TRUE
#> 6 dummy replace_r… <NA> <NA> rectangular Replace cl… "(Should… TRUE
#> 7 dummy vector_to… <NA> <NA> vector Convert ve… "(Should… TRUE
#> 8 dummy vector_cu… <NA> <NA> vector Custom cla… "(Should… TRUE
#> 9 dummy double_is… <NA> <NA> numeric Check whet… "int par… TRUE
#> 10 dummy trivial_n… <NA> <NA> numeric Add trivia… "(Should… TRUE
#> # … with 15 more rowsThat functions returns a tibble describing 25 unique tests. All autotest functions return these same kinds of objects. The table returned from autotest_types() can be used to selectively switch tests off by setting values in the test column to FALSE, as demonstrated below.
Descriptions of each test can be readily extracted from the results of that function:
a <- autotest_types ()
print (a [, c ("parameter_type", "operation", "content")], n = Inf)
#> # A tibble: 25 x 3
#> parameter_type operation content
#> <chr> <chr> <chr>
#> 1 rectangular Convert one rectangular cla… "check for error/warning mes…
#> 2 rectangular Convert one rectangular cla… "expect dimensions are same "
#> 3 rectangular Convert one rectangular cla… "expect column names are ret…
#> 4 rectangular Convert one rectangular cla… "expect all columns retain i…
#> 5 rectangular Extend existent class with … "(Should yield same result)"
#> 6 rectangular Replace class with new class "(Should yield same result)"
#> 7 vector Convert vector input to lis… "(Should yield same result)"
#> 8 vector Custom class definitions fo… "(Should yield same result)"
#> 9 numeric Check whether double is onl… "int parameters should have …
#> 10 numeric Add trivial noise to numeri… "(Should yield same result)"
#> 11 single integer Integer value converted to … "(Should yield same result)"
#> 12 single logical Substitute integer values f… "(Function call should still…
#> 13 single character random character string as … "Should error"
#> 14 single character Change case "(Should yield same result)"
#> 15 single integer Ascertain permissible range "Should either be unrestrict…
#> 16 single integer Length 2 vector for length … "Should trigger message, war…
#> 17 single name or fo… Unquoted name/formula as qu… "Capture any warnings or err…
#> 18 single logical Substitute character values… "should trigger warning or e…
#> 19 single logical Negate default value of log… "(Function call should still…
#> 20 (return object) Check that function success… <NA>
#> 21 (return object) Check that description has … <NA>
#> 22 (return object) Check whether description o… <NA>
#> 23 (return object) Compare class of return val… <NA>
#> 24 <NA> Check that parameter usage … "Examples do not demonstrate…
#> 25 <NA> Identify functions without … <NA>How Does It Work?
The autotest_package() function returns by default a list of all tests which would be conducted on a package, without actually implementing those tests. The function has a parameter, test, with a default value of FALSE. Setting test = TRUE then implements all tests, and only returns results from tests which diverge from expected behaviour, whether unexpected errors, warnings, or other behaviour. An ideal result is that autotest_package(., test = TRUE) returns nothing (strictly, NULL), indicating that all tests passed successfully.
Tests can also be selectively applied to particular functions through the parameters functions, used to nominate functions to include in tests, or exclude, used to nominate functions to exclude from tests. The following code illustrates.
x <- autotest_package (package = "stats", functions = "var", test = FALSE)
#>
#> ── autotesting stats ──
#>
#> ✔ [1 / 4]: var
#> ✔ [2 / 4]: cor
#> ✔ [3 / 4]: cov
#> ✔ [4 / 4]: cov
print (x)
#> # A tibble: 112 x 9
#> type test_name fn_name parameter parameter_type operation content test
#> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <lgl>
#> 1 dummy int_as_n… var x integer vector Integer … (Shoul… TRUE
#> 2 dummy vector_c… var x vector Custom c… (Shoul… TRUE
#> 3 dummy vector_t… var x vector Convert … (Shoul… TRUE
#> 4 dummy negate_l… var na.rm single logical Negate d… (Funct… TRUE
#> 5 dummy subst_in… var na.rm single logical Substitu… (Funct… TRUE
#> 6 dummy subst_ch… var na.rm single logical Substitu… should… TRUE
#> 7 dummy single_p… var na.rm single logical Length 2… Should… TRUE
#> 8 dummy return_s… var (return … (return objec… Check th… <NA> TRUE
#> 9 dummy return_v… var (return … (return objec… Check th… <NA> TRUE
#> 10 dummy return_d… var (return … (return objec… Check wh… <NA> TRUE
#> # … with 102 more rows, and 1 more variable: yaml_hash <chr>Testing the var function also tests cor and cov, because the package works by scraping the documented examples from the associated .Rd help file, and ?var shows that the help topic is cor, and includes the three functions, var, cor, and cov. That result details the 112 tests which would be applied to the var function from the stats package. These 112 tests yield the following results when actually applied:
y <- autotest_package (package = "stats", functions = "var", test = TRUE)
#> ── autotesting stats ──
#>
#> ✔ [1 / 4]: var
#> ✔ [2 / 4]: cor
#> ✔ [3 / 4]: cov
#> ✔ [4 / 4]: cov
print (y)
#> # A tibble: 16 x 9
#> type test_name fn_name parameter parameter_type operation content test
#> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <lgl>
#> 1 diag… vector_t… var x vector Convert … Functi… TRUE
#> 2 diag… int_as_n… var x integer vector Integer … Functi… TRUE
#> 3 diag… int_as_n… var y integer vector Integer … Functi… TRUE
#> 4 diag… vector_t… var y vector Convert … Functi… TRUE
#> 5 diag… single_c… cor use single charac… upper-ca… is cas… TRUE
#> 6 diag… single_c… cor method single charac… upper-ca… is cas… TRUE
#> 7 diag… single_c… cor use single charac… upper-ca… is cas… TRUE
#> 8 diag… single_c… cor method single charac… upper-ca… is cas… TRUE
#> 9 diag… single_c… cov use single charac… upper-ca… is cas… TRUE
#> 10 diag… single_c… cov method single charac… upper-ca… is cas… TRUE
#> 11 diag… single_c… cov use single charac… upper-ca… is cas… TRUE
#> 12 diag… single_c… cov method single charac… upper-ca… is cas… TRUE
#> 13 diag… single_c… cov use single charac… upper-ca… is cas… TRUE
#> 14 diag… single_c… cov method single charac… upper-ca… is cas… TRUE
#> 15 warn… par_is_d… var use <NA> Check th… Exampl… TRUE
#> 16 warn… par_is_d… cov y <NA> Check th… Exampl… TRUE
#> # … with 1 more variable: yaml_hash <chr>And only 16 of the original 112 tests produced unexpected behaviour. There were in fact only three kinds of tests which produced these 16 results:
unique (y$operation)
#> [1] "Convert vector input to list-columns"
#> [2] "Integer value converted to numeric"
#> [3] "upper-case character parameter"
#> [4] "Check that parameter usage is demonstrated"The first involves conversion of a vector to a list-column representation (via I(as.list(<vec>))). Relatively few packages accept this kind of input, even though doing so is relatively straightforward. The following lines demonstrate how these tests can be switched off when autotest-ing a package. The autotest_types() function, used above to extract information on all types of tests, also accepts a single argument listing the test_name entries of any tests which are to be switched off.
types <- autotest_types (notest = "vector_to_list_col")
y <- autotest_package (package = "stats", functions = "var",
test = TRUE, test_data = types)
#> ── autotesting stats ──
#>
#> ✔ [1 / 4]: var
#> ✔ [2 / 4]: cor
#> ✔ [3 / 4]: cov
#> ✔ [4 / 4]: cov
print (y)
#> # A tibble: 18 x 9
#> type test_name fn_name parameter parameter_type operation content test
#> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <lgl>
#> 1 no_t… vector_t… var x vector Convert … (Shoul… FALSE
#> 2 diag… int_as_n… var x integer vector Integer … Functi… TRUE
#> 3 diag… int_as_n… var y integer vector Integer … Functi… TRUE
#> 4 no_t… vector_t… var y vector Convert … (Shoul… FALSE
#> 5 no_t… vector_t… cor x vector Convert … (Shoul… FALSE
#> 6 no_t… vector_t… cor y vector Convert … (Shoul… FALSE
#> 7 diag… single_c… cor use single charac… upper-ca… is cas… TRUE
#> 8 diag… single_c… cor method single charac… upper-ca… is cas… TRUE
#> 9 diag… single_c… cor use single charac… upper-ca… is cas… TRUE
#> 10 diag… single_c… cor method single charac… upper-ca… is cas… TRUE
#> 11 diag… single_c… cov use single charac… upper-ca… is cas… TRUE
#> 12 diag… single_c… cov method single charac… upper-ca… is cas… TRUE
#> 13 diag… single_c… cov use single charac… upper-ca… is cas… TRUE
#> 14 diag… single_c… cov method single charac… upper-ca… is cas… TRUE
#> 15 diag… single_c… cov use single charac… upper-ca… is cas… TRUE
#> 16 diag… single_c… cov method single charac… upper-ca… is cas… TRUE
#> 17 warn… par_is_d… var use <NA> Check th… Exampl… TRUE
#> 18 warn… par_is_d… cov y <NA> Check th… Exampl… TRUE
#> # … with 1 more variable: yaml_hash <chr>Those tests are still returned from autotest_package(), but with test = FALSE to indicate they were not run, and a type of “no_test” rather than the previous “diagnostic”.
Can autotest automatically create tests in my tests directory?
Not yet, but that should be possible soon. In the meantime, there are testthat expectations, listed in the main package functions, which enable autotest to be used in a package’s test suite.
Prior work
- The
great-expectationsframework for python, described in this medium article. -
QuickCheckfor Haskell -
mutatefor ruby.
