Perform model selection using leave one out cross validation of model objects

This function performs leave one out cross validation of a list of model fits using functions in the loo package, as described in Vehtari, Gelman, and Gabry (2017) doi:10.1007/s11222-016-9696-4. Compare models fit using jointModel() or models fits using traditionalModel(). See more examples in the Package Vignette.

Usage

jointSelect(modelfits)

Arguments

modelfits: A list containing model fits of class stanfit.

Value

A matrix of delta elpd (expected log pointwise predictive density) between model fits. Function is performed using the loo package.

Note

Before model selection, this function makes the following check:

Input is a list of model fits of class 'stanfit'.
All models compared were fit wither either jointModel() or all with traditionalModel().

If any of these checks fail, the function returns an error message.

Examples

# \donttest{
data(greencrabData)

# Fit a model without estimating a catchability coefficient for traditional
# survey gear types.
# This model assumes all traditional survey methods have the same
# catchability.
# Count data is modeled using a poisson distribution.
fit.no.q <- jointModel(data = greencrabData, family = "poisson",
                       p10priors = c(1,20), q = FALSE, multicore = FALSE)
#> 
#> SAMPLING FOR MODEL 'joint_binary_pois' NOW (CHAIN 1).
#> Chain 1: 
#> Chain 1: Gradient evaluation took 6.6e-05 seconds
#> Chain 1: 1000 transitions using 10 leapfrog steps per transition would take 0.66 seconds.
#> Chain 1: Adjust your expectations accordingly!
#> Chain 1: 
#> Chain 1: 
#> Chain 1: Iteration:    1 / 3000 [  0%]  (Warmup)
#> Chain 1: Iteration:  500 / 3000 [ 16%]  (Warmup)
#> Chain 1: Iteration:  501 / 3000 [ 16%]  (Sampling)
#> Chain 1: Iteration: 1000 / 3000 [ 33%]  (Sampling)
#> Chain 1: Iteration: 1500 / 3000 [ 50%]  (Sampling)
#> Chain 1: Iteration: 2000 / 3000 [ 66%]  (Sampling)
#> Chain 1: Iteration: 2500 / 3000 [ 83%]  (Sampling)
#> Chain 1: Iteration: 3000 / 3000 [100%]  (Sampling)
#> Chain 1: 
#> Chain 1:  Elapsed Time: 0.926 seconds (Warm-up)
#> Chain 1:                2.203 seconds (Sampling)
#> Chain 1:                3.129 seconds (Total)
#> Chain 1: 
#> 
#> SAMPLING FOR MODEL 'joint_binary_pois' NOW (CHAIN 2).
#> Chain 2: 
#> Chain 2: Gradient evaluation took 6.3e-05 seconds
#> Chain 2: 1000 transitions using 10 leapfrog steps per transition would take 0.63 seconds.
#> Chain 2: Adjust your expectations accordingly!
#> Chain 2: 
#> Chain 2: 
#> Chain 2: Iteration:    1 / 3000 [  0%]  (Warmup)
#> Chain 2: Iteration:  500 / 3000 [ 16%]  (Warmup)
#> Chain 2: Iteration:  501 / 3000 [ 16%]  (Sampling)
#> Chain 2: Iteration: 1000 / 3000 [ 33%]  (Sampling)
#> Chain 2: Iteration: 1500 / 3000 [ 50%]  (Sampling)
#> Chain 2: Iteration: 2000 / 3000 [ 66%]  (Sampling)
#> Chain 2: Iteration: 2500 / 3000 [ 83%]  (Sampling)
#> Chain 2: Iteration: 3000 / 3000 [100%]  (Sampling)
#> Chain 2: 
#> Chain 2:  Elapsed Time: 0.869 seconds (Warm-up)
#> Chain 2:                2.142 seconds (Sampling)
#> Chain 2:                3.011 seconds (Total)
#> Chain 2: 
#> 
#> SAMPLING FOR MODEL 'joint_binary_pois' NOW (CHAIN 3).
#> Chain 3: 
#> Chain 3: Gradient evaluation took 6.2e-05 seconds
#> Chain 3: 1000 transitions using 10 leapfrog steps per transition would take 0.62 seconds.
#> Chain 3: Adjust your expectations accordingly!
#> Chain 3: 
#> Chain 3: 
#> Chain 3: Iteration:    1 / 3000 [  0%]  (Warmup)
#> Chain 3: Iteration:  500 / 3000 [ 16%]  (Warmup)
#> Chain 3: Iteration:  501 / 3000 [ 16%]  (Sampling)
#> Chain 3: Iteration: 1000 / 3000 [ 33%]  (Sampling)
#> Chain 3: Iteration: 1500 / 3000 [ 50%]  (Sampling)
#> Chain 3: Iteration: 2000 / 3000 [ 66%]  (Sampling)
#> Chain 3: Iteration: 2500 / 3000 [ 83%]  (Sampling)
#> Chain 3: Iteration: 3000 / 3000 [100%]  (Sampling)
#> Chain 3: 
#> Chain 3:  Elapsed Time: 0.956 seconds (Warm-up)
#> Chain 3:                1.502 seconds (Sampling)
#> Chain 3:                2.458 seconds (Total)
#> Chain 3: 
#> 
#> SAMPLING FOR MODEL 'joint_binary_pois' NOW (CHAIN 4).
#> Chain 4: 
#> Chain 4: Gradient evaluation took 6.1e-05 seconds
#> Chain 4: 1000 transitions using 10 leapfrog steps per transition would take 0.61 seconds.
#> Chain 4: Adjust your expectations accordingly!
#> Chain 4: 
#> Chain 4: 
#> Chain 4: Iteration:    1 / 3000 [  0%]  (Warmup)
#> Chain 4: Iteration:  500 / 3000 [ 16%]  (Warmup)
#> Chain 4: Iteration:  501 / 3000 [ 16%]  (Sampling)
#> Chain 4: Iteration: 1000 / 3000 [ 33%]  (Sampling)
#> Chain 4: Iteration: 1500 / 3000 [ 50%]  (Sampling)
#> Chain 4: Iteration: 2000 / 3000 [ 66%]  (Sampling)
#> Chain 4: Iteration: 2500 / 3000 [ 83%]  (Sampling)
#> Chain 4: Iteration: 3000 / 3000 [100%]  (Sampling)
#> Chain 4: 
#> Chain 4:  Elapsed Time: 0.904 seconds (Warm-up)
#> Chain 4:                2.124 seconds (Sampling)
#> Chain 4:                3.028 seconds (Total)
#> Chain 4: 
#> Refer to the eDNAjoint guide for visualization tips:  https://bookdown.org/abigailkeller/eDNAjoint_vignette/tips.html#visualization-tips 


# Fit a model estimating a catchability coefficient for traditional
# survey gear types.
# This model does not assume all traditional survey methods have the
# same catchability.
# Gear type 1 is used as the reference gear type.
# Count data is modeled using a negative binomial distribution.
fit.q <- jointModel(data = greencrabData, family = "negbin",
                    p10priors = c(1,20), q = TRUE, multicore = FALSE)
#> 
#> SAMPLING FOR MODEL 'joint_binary_catchability_negbin' NOW (CHAIN 1).
#> Chain 1: 
#> Chain 1: Gradient evaluation took 0.000282 seconds
#> Chain 1: 1000 transitions using 10 leapfrog steps per transition would take 2.82 seconds.
#> Chain 1: Adjust your expectations accordingly!
#> Chain 1: 
#> Chain 1: 
#> Chain 1: Iteration:    1 / 3000 [  0%]  (Warmup)
#> Chain 1: Iteration:  500 / 3000 [ 16%]  (Warmup)
#> Chain 1: Iteration:  501 / 3000 [ 16%]  (Sampling)
#> Chain 1: Iteration: 1000 / 3000 [ 33%]  (Sampling)
#> Chain 1: Iteration: 1500 / 3000 [ 50%]  (Sampling)
#> Chain 1: Iteration: 2000 / 3000 [ 66%]  (Sampling)
#> Chain 1: Iteration: 2500 / 3000 [ 83%]  (Sampling)
#> Chain 1: Iteration: 3000 / 3000 [100%]  (Sampling)
#> Chain 1: 
#> Chain 1:  Elapsed Time: 3.234 seconds (Warm-up)
#> Chain 1:                8.156 seconds (Sampling)
#> Chain 1:                11.39 seconds (Total)
#> Chain 1: 
#> 
#> SAMPLING FOR MODEL 'joint_binary_catchability_negbin' NOW (CHAIN 2).
#> Chain 2: 
#> Chain 2: Gradient evaluation took 0.000323 seconds
#> Chain 2: 1000 transitions using 10 leapfrog steps per transition would take 3.23 seconds.
#> Chain 2: Adjust your expectations accordingly!
#> Chain 2: 
#> Chain 2: 
#> Chain 2: Iteration:    1 / 3000 [  0%]  (Warmup)
#> Chain 2: Iteration:  500 / 3000 [ 16%]  (Warmup)
#> Chain 2: Iteration:  501 / 3000 [ 16%]  (Sampling)
#> Chain 2: Iteration: 1000 / 3000 [ 33%]  (Sampling)
#> Chain 2: Iteration: 1500 / 3000 [ 50%]  (Sampling)
#> Chain 2: Iteration: 2000 / 3000 [ 66%]  (Sampling)
#> Chain 2: Iteration: 2500 / 3000 [ 83%]  (Sampling)
#> Chain 2: Iteration: 3000 / 3000 [100%]  (Sampling)
#> Chain 2: 
#> Chain 2:  Elapsed Time: 3.375 seconds (Warm-up)
#> Chain 2:                10.724 seconds (Sampling)
#> Chain 2:                14.099 seconds (Total)
#> Chain 2: 
#> 
#> SAMPLING FOR MODEL 'joint_binary_catchability_negbin' NOW (CHAIN 3).
#> Chain 3: 
#> Chain 3: Gradient evaluation took 0.000386 seconds
#> Chain 3: 1000 transitions using 10 leapfrog steps per transition would take 3.86 seconds.
#> Chain 3: Adjust your expectations accordingly!
#> Chain 3: 
#> Chain 3: 
#> Chain 3: Iteration:    1 / 3000 [  0%]  (Warmup)
#> Chain 3: Iteration:  500 / 3000 [ 16%]  (Warmup)
#> Chain 3: Iteration:  501 / 3000 [ 16%]  (Sampling)
#> Chain 3: Iteration: 1000 / 3000 [ 33%]  (Sampling)
#> Chain 3: Iteration: 1500 / 3000 [ 50%]  (Sampling)
#> Chain 3: Iteration: 2000 / 3000 [ 66%]  (Sampling)
#> Chain 3: Iteration: 2500 / 3000 [ 83%]  (Sampling)
#> Chain 3: Iteration: 3000 / 3000 [100%]  (Sampling)
#> Chain 3: 
#> Chain 3:  Elapsed Time: 3.506 seconds (Warm-up)
#> Chain 3:                8.797 seconds (Sampling)
#> Chain 3:                12.303 seconds (Total)
#> Chain 3: 
#> 
#> SAMPLING FOR MODEL 'joint_binary_catchability_negbin' NOW (CHAIN 4).
#> Chain 4: 
#> Chain 4: Gradient evaluation took 0.000288 seconds
#> Chain 4: 1000 transitions using 10 leapfrog steps per transition would take 2.88 seconds.
#> Chain 4: Adjust your expectations accordingly!
#> Chain 4: 
#> Chain 4: 
#> Chain 4: Iteration:    1 / 3000 [  0%]  (Warmup)
#> Chain 4: Iteration:  500 / 3000 [ 16%]  (Warmup)
#> Chain 4: Iteration:  501 / 3000 [ 16%]  (Sampling)
#> Chain 4: Iteration: 1000 / 3000 [ 33%]  (Sampling)
#> Chain 4: Iteration: 1500 / 3000 [ 50%]  (Sampling)
#> Chain 4: Iteration: 2000 / 3000 [ 66%]  (Sampling)
#> Chain 4: Iteration: 2500 / 3000 [ 83%]  (Sampling)
#> Chain 4: Iteration: 3000 / 3000 [100%]  (Sampling)
#> Chain 4: 
#> Chain 4:  Elapsed Time: 3.566 seconds (Warm-up)
#> Chain 4:                9.214 seconds (Sampling)
#> Chain 4:                12.78 seconds (Total)
#> Chain 4: 
#> Refer to the eDNAjoint guide for visualization tips:  https://bookdown.org/abigailkeller/eDNAjoint_vignette/tips.html#visualization-tips 

# Perform model selection
jointSelect(modelfits = list(fit.no.q$model, fit.q$model))
#>        elpd_diff se_diff
#> model2    0.0       0.0 
#> model1 -164.8      39.0 
# }

Perform model selection using leave one out cross validation of model objects

Usage

Arguments

Value

Note

Examples

About

Community

Resources