Control of tests
Mark Padgham
2024-10-23
Source:vignettes/autotest-control.Rmd
autotest-control.RmdThe first
vignette demonstrates the process of applying autotest
at all stages of package development. This vignette provides additional
information for those applying autotest to already
developed packages, in particular through describing how tests can be
selectively applied to a package. By default the autotest_package()
function tests an entire package, but testing can also be restricted
to specified functions only. This vignette will demonstrate application
to a few functions from the stats
package, starting by loading the package.
1. .Rd files, example code, and the autotest
workflow
To understand what autotest does, it is first necessary
to understand a bit about the structure of documentation files for R
package, which are contained in files called ".Rd" files.
Tests are constructed by parsing individual .Rd
documentation files to extract the example code, identifying parameters
passed to the specified functions, and mutating those parameters.
The general procedure can be illustrated by examining a specific
function, for which we now choose the cor
function, because of its relative simplicity. The following lines
extract the documentation for the cor function, a
.html version of which can be seen by clicking on the link
above. Note that that web page reveals the name of the .Rd
file to be “cor” (in the upper left corner), meaning that the name of
the .Rd file is "cor.Rd". The following lines
extract that content, first by loading the entire .Rd
database for the stats
package.
The database itself is a list, with each entry holding the contents
of one .Rd file in an object of class Rd, which is
essentially a large nested list of components corresponding to the
various .Rd tags such as \arguments,
\details, and \value. An internal function
from the tools
package can be used to extract individual components (using the
::: notation to access internal functions). For example, a
single .Rd file often describes the functionality of
several functions, each of which is identified by specifying the
function name as an "alias". The aliases for the
"cor.Rd" file are:
tools:::.Rd_get_metadata (cor_rd, "alias")
#> [1] "var" "cov" "cor" "cov2cor"This one file thus contains documentation for those four functions.
Example code can be extracted with a dedicated function from the tools
package:
tools::Rd2ex (cor_rd)#> ### Name: cor
#> ### Title: Correlation, Variance and Covariance (Matrices)
#> ### Aliases: var cov cor cov2cor
#> ### Keywords: univar multivariate array
#>
#> ### ** Examples
#>
#> var(1:10) # 9.166667
#>
#> var(1:5, 1:5) # 2.5
#>
#> ## Two simple vectors
#> cor(1:10, 2:11) # == 1
#>
#> ## Correlation Matrix of Multivariate sample:
#> (Cl <- cor(longley))
#> ## Graphical Correlation Matrix:
#> symnum(Cl) # highly correlated
#>
#> ## Spearman's rho and Kendall's tau
#> symnum(clS <- cor(longley, method = "spearman"))
#> symnum(clK <- cor(longley, method = "kendall"))
#> ## How much do they differ?
#> i <- lower.tri(Cl)
#> cor(cbind(P = Cl[i], S = clS[i], K = clK[i]))
#>
#>
#> ## cov2cor() scales a covariance matrix by its diagonal
#> ## to become the correlation matrix.
#> cov2cor # see the function definition {and learn ..}
#> stopifnot(all.equal(Cl, cov2cor(cov(longley))),
#> all.equal(cor(longley, method = "kendall"),
#> cov2cor(cov(longley, method = "kendall"))))
#>
#> ##--- Missing value treatment:
#> C1 <- cov(swiss)
#> range(eigen(C1, only.values = TRUE)$values) # 6.19 1921
#>
#> ## swM := "swiss" with 3 "missing"s :
#> swM <- swiss
#> colnames(swM) <- abbreviate(colnames(swiss), minlength=6)
#> swM[1,2] <- swM[7,3] <- swM[25,5] <- NA # create 3 "missing"
#>
#> ## Consider all 5 "use" cases :
#> (C. <- cov(swM)) # use="everything" quite a few NA's in cov.matrix
#> try(cov(swM, use = "all")) # Error: missing obs...
#> C2 <- cov(swM, use = "complete")
#> stopifnot(identical(C2, cov(swM, use = "na.or.complete")))
#> range(eigen(C2, only.values = TRUE)$values) # 6.46 1930
#> C3 <- cov(swM, use = "pairwise")
#> range(eigen(C3, only.values = TRUE)$values) # 6.19 1938
#>
#> ## Kendall's tau doesn't change much:
#> symnum(Rc <- cor(swM, method = "kendall", use = "complete"))
#> symnum(Rp <- cor(swM, method = "kendall", use = "pairwise"))
#> symnum(R. <- cor(swiss, method = "kendall"))
#>
#> ## "pairwise" is closer componentwise,
#> summary(abs(c(1 - Rp/R.)))
#> summary(abs(c(1 - Rc/R.)))
#>
#> ## but "complete" is closer in Eigen space:
#> EV <- function(m) eigen(m, only.values=TRUE)$values
#> summary(abs(1 - EV(Rp)/EV(R.)) / abs(1 - EV(Rc)/EV(R.)))This is the entire content of the \examples portion of
"cor.Rd", as can be confirmed by comparing with the online
version.
2. Internal structure of the autotest workflow
For each .Rd file in a package, autotest
tests the code given in the example section according to the following
general steps:
- Extract example lines from the
.Rdfile, as demonstrated above; - Identify all function
aliasesdescribed by that file; - Identify all points at which those functions are called;
- Identify all objects passed to those values, including values, classes, attributes, and other properties.
- Identify any other parameters not explicitly passed in example code, but defined via default value;
- Mutate the values of all parameters according to the kinds of test
described in
autotest_types().
Calling autotest_package(..., test = FALSE) implements
the first 5 of those 6 steps, and returns data on all possible mutations
of each parameter, while setting test = TRUE actually
passes the mutated parameters to the specified functions, and returns
reports on any unexpected behaviour.
3. autotest-ing the stats::cov
function
The preceding sections describe how autotest actually
works, while the present section demonstrates how the package is
typically used in practice. As demonstrated in the README,
information on all tests implemented within the package can be obtained
by calling the autotest_types()
function. The main function for testing package is autotest_package().
The nominated package can be either an installed package, or the full
path to a local directory containing a package’s source code. By default
all .Rd files of a package are tested, with restriction to
specified functions possible either by nominating functions to exclude
from testing (via the exclude parameter), or functions to
include (via the functions parameter). The
functions parameter is intended to enable testing only of
specified functions, while the exclude parameter is
intended to enable testing of all functions except those specified with
this parameter. Specifying values for both of these parameters is not
generally recommended.
3.1 Listing tests without conducting them
The following demonstrates the results of autotest-ing
the cor
function of the stats package, noting that the default
call uses test = FALSE, and so returns details of all tests
without actually implementing them (and for this reason we name the
object xf for “false”):
xf <- autotest_package (package = "stats", functions = "cor")
print (xf)#> # A tibble: 15 × 8
#> type test_name fn_name parameter parameter_type operation content test
#> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <lgl>
#> 1 dummy single_char_c… cor use single_charac… lower-ca… (Shoul… TRUE
#> 2 dummy single_char_c… cor use single_charac… upper-ca… (Shoul… TRUE
#> 3 dummy single_par_as… cor use single_charac… Length 2… Should… TRUE
#> 4 dummy return_succes… cor (return … (return objec… Check th… NA TRUE
#> 5 dummy return_val_de… cor (return … (return objec… Check th… NA TRUE
#> 6 dummy return_desc_i… cor (return … (return objec… Check wh… NA TRUE
#> 7 dummy return_class_… cor (return … (return objec… Compare … NA TRUE
#> 8 dummy par_is_docume… cor x NA Check th… NA TRUE
#> 9 dummy par_matches_d… cor x NA Check th… NA TRUE
#> 10 dummy par_is_docume… cor use NA Check th… NA TRUE
#> 11 dummy par_matches_d… cor use NA Check th… NA TRUE
#> 12 dummy par_is_docume… cor method NA Check th… NA TRUE
#> 13 dummy par_matches_d… cor method NA Check th… NA TRUE
#> 14 dummy par_is_docume… cor y NA Check th… NA TRUE
#> 15 dummy par_matches_d… cor y NA Check th… NA TRUEThe object returned from autotest_package() is a simple
tibble, with
each row detailing one test which would be applied to the each of the
listed functions and parameters. Because no tests were conducted, all
tests will generally have a type of "dummy".
In this case, however, we see the following:
table (xf$type)
#>
#> dummy
#> 15In addition to the 15 dummy tests, the function also returns 0 warnings, the corresponding rows of which are:
xf [xf$type != "dummy", c ("fn_name", "parameter", "operation", "content")]
#> # A tibble: 0 × 4
#> # ℹ 4 variables: fn_name <chr>, parameter <chr>, operation <chr>, content <chr>Although the auotest package is primarily intended to
apply mutation tests to all parameters of all functions of a package,
doing so requires identifying parameter types and classes through
parsing example code. Any parameters of a function which are neither
demonstrated within example code, nor given default values can not be
tested, because it is not possible to determine their expected types.
The above result reveals that neither the use parameter of
the var function, nor the y parameter of
cov, are demonstrated in example code, triggering a warning
that these parameter are unable to be tested.
3.2 Conducting tests
The 15 tests listed above with type == "dummy" can then
be applied to all nominated functions and parameters by calling the same
function with test = TRUE. Doing so yields the following
results (as an object names xt for “true”):
xt <- autotest_package (package = "stats",
functions = "cor",
test = TRUE)
print (xt)And the 15 tests yielded 5 unexpected responses. The best way to
understand these results is to examine the object in detail, typically
through edit(xt), or equivalently in RStudio, clicking on
the listed object. The different types of tests which produced
unexpected responses were:
table (xt1$operation)
#>
#> Check that documentation matches class of input parameter
#> 4
#> upper-case character parameter
#> 1Two of those reflect the previous results regarding parameters unable
to be tested, while the remainder come from only two types of tests.
Information on the precise results is contained in the
content column, although in this case it is fairly
straightforward to see that the operation “upper case character
parameter” arises because the use and method
parameters of the cor and cov functions are
case-dependent, and are only accepted in lower case form. The other
operation is the conversion of vectors to list-column format, as
described in the first
vignette.
3.4 Controlling which tests are conducted
The test parameter of the autotest_package()
function can be used to control whether all tests are conducted or
not. Finer-level control over tests can be achieved by specifying the
test_data parameter. This parameter must be an object of
class autotest_package, as returned by either the autotest_types()]
or autotest_package()
functions. The former of these is the function which specifies all
unique tests, and so returns a relatively small tibble of
27 rows. The following lines demonstrate how to switch off the
list-column test for all functions and parameters:
types <- autotest_types()
types$test [grep ("list_col", types$test_name)] <- FALSE
xt2 <- autotest_package (package = "stats",
functions = "cor",
test = TRUE,
test_data = types)
print (xt2)#> # A tibble: 5 × 8
#> type test_name fn_name parameter parameter_type operation content test
#> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <lgl>
#> 1 warning par_match… cor x NA Check th… Parame… TRUE
#> 2 warning par_match… cor y NA Check th… Parame… TRUE
#> 3 warning par_match… cor x NA Check th… Parame… TRUE
#> 4 warning par_match… cor y NA Check th… Parame… TRUE
#> 5 diagnostic single_ch… cor use single charac… upper-ca… is cas… TRUEThe result now has four rows with test == FALSE, and
type == "no_test", indicating that these tests were not
actually conducted. That also makes apparent the role of these
test flags. When initially calling autotest_package()
with default test = FALSE, the result contains a
test column in which all values are TRUE.
Although potentially perplexing at first, this value must be understood
in relation to the type column. A type of
"dummy" indicates that a test has not been conducted, in
which case test = TRUE is a control flag used to determine
what would be conducted if these data were submitted as the
test_data parameter. For all type values other
than "dummy", the test column specifies
whether or not each test was actually conducted.
The preceding example showed how the results of autotest_types()
can be used to control which tests are implemented for an entire
package. Finer-scale control can be achieved by modifying individual
rows of the full table returned by autotest_package().
The following code demonstrates by showing how list-column tests can be
switched off only for particular functions, starting again with the
xf data of dummy tests generated above.
xf <- autotest_package (package = "stats",
functions = "cor")
xf$test [grepl ("list_col", xf$test_name) & xf$fn_name == "var"] <- FALSE
xt3 <- autotest_package (package = "stats",
functions = "cor",
test = TRUE,
test_data = xf)
print (xt3)#> # A tibble: 5 × 8
#> type test_name fn_name parameter parameter_type operation content test
#> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <lgl>
#> 1 warning par_match… cor x NA Check th… Parame… TRUE
#> 2 warning par_match… cor y NA Check th… Parame… TRUE
#> 3 warning par_match… cor x NA Check th… Parame… TRUE
#> 4 warning par_match… cor y NA Check th… Parame… TRUE
#> 5 diagnostic single_ch… cor use single charac… upper-ca… is cas… TRUEThese procedures illustrate the three successively finer levels of control over tests, by switching them off for:
- Entire packages;
- Specified functions only; or
- Specific parameters of particular functions only.
4. autotest-ing your package
autotest can be very easily incorporated in your
package’s tests/ directory via to simple testthat
expectations:
-
expect_autotest_no_testdata, which will expectautotest_packageto work on your package with default values including no additionaltest_dataspecifying tests which are not to be run; or -
expect_autotest_testdata, to be used when specific tests are switched off.
Using these requires adding autotest to the
Suggests list of your package’s DESCRIPTION
file, along with testthat. Note that the use of testing
frameworks other than testthat is possible
through writing custom expectations for the output of autotest_package(),
but that is not considered here.
To use these expectations, you must first decide which, if any, tests
you judge to be not applicable to your package, and switch them off
following the procedure described above (that is, at the package level
through modifying the test flag of the object returned from
autotest_types(),
or at finer function- or parameter-levels by modifying equivalent values
in the object returned from autotest_package(..., test = FALSE).
These objects must then be passed as the test_data
parameter to autotest_package().
If you consider all tests to be applicable, then autotest_package()
can be called without specifying this parameter.
If you switch tests off via a test_data parameter, then
the expect_autotest expectation requires you to append an
additional column to the test_data object called
"note" (case-insensitive), and include a note for each row
which has test = FALSE explaining why those tests have been
switched off. Lines in your test directory should look something like
this:
library (testthat) # as called in your test suite
# For example, to switch off vector-to-list-column tests:
test_data <- autotest_types (notest = "vector_to_list_col")
test_data$note <- ""
test_data$note [test_data$test == "vector_to_list_col"] <-
"These tests are not applicable because ..."
expect_success (expect_autotest_testdata (test_data))This procedure of requiring an additional "note" column
ensures that your own test suite will explicitly include all
explanations of why you deem particular tests not to be applicable to
your package.
In contrast, the following expectation should be used when
autotest_package() passes with all tests are implemented,
in which case no parameters need be passed to the expectation, and tests
will confirm that no warnings or errors are generated.
4.2 Finer control over testing expectations
The two expectations shown above call the autotest_package()
function internally, and assert that the results follow the expected
pattern. There are also three additional testthat expectations
which can be applied to pre-generated autotest objects, to
allow for finer control over testing expectations. These are:
-
expect_autotest_no_errto expect no errors in results fromautotest_package(); -
expect_autotest_no_warnto expect no warnings; and -
expect_autotest_notesto expect tests which have been switched off to have an additional"note"column explaining why.
These tests are demonstrated in one of the testing files used in this
package, which the following lines recreate to demonstrate the general
process. The first two expectations are that an object be free from both
warnings and errors. The tests implemented here are applied to the stats::cov()
function, which actually triggers warnings because two parameters do
not have their usage demonstrated in the example code. The tests
therefore expect_failure(), when they generally should
expect_success() throughout.
library (testthat) # as called in your test suite
# For example, to switch off vector-to-list-column tests:
test_data <- autotest_types (notest = "vector_to_list_col")
x <- autotest_package (package = "stats",
functions = "cov",
test = TRUE,
test_data = test_data)
expect_success (expect_autotest_no_err (x))
expect_failure (expect_autotest_no_warn (x)) # should expect_success!!The test files then affirms that simply passing the object,
x, which has tests flagged as
type == "no_test", yet without explaining why in an
additional "note" column, should cause
expect_autotest() to fail. The following line, removing the
logical testthat expectation, demonstrates:
As demonstrated above, these expect_autotest_... calls
should always be wrapped in a direct testhat expectation
of expect_success().
To achieve success in that case, we need to append an additional
"note" column containing explanations of why each test has
been switched off:
x$note <- ""
x [grep ("vector_to_list", x$test_name), "note"] <-
"these tests have been switched off because ..."
expect_success (expect_autotest_notes (x))In general, using autotest in a package’s test suite
should be as simple as adding autotest to
Suggests, and wrapping either
expect_autotest_no_testdata or
expect_autotest_testdata in an expect_success
call.