# Perform model selection using leave one out cross validation of model objects

Source:`R/jointSelect.R`

`jointSelect.Rd`

This function performs leave one out cross validation of a list of model
fits using functions in the `loo`

package, as described in Vehtari, Gelman,
and Gabry (2017) doi:10.1007/s11222-016-9696-4. Compare models fit using
`jointModel()`

or models fits using `traditionalModel()`

. See more examples
in the Package
Vignette.

## Value

A matrix of delta elpd (expected log pointwise predictive density)
between model fits. Function is performed using the `loo`

package.

## Note

Before model selection, this function makes the following check:

Input is a list of model fits of class 'stanfit'.

All models compared were fit wither either

`jointModel()`

or all with`traditionalModel().`

If any of these checks fail, the function returns an error message.

## Examples

```
# \donttest{
data(greencrabData)
# Fit a model without estimating a gear scaling coefficient for traditional
# survey gear types.
# This model assumes all traditional survey methods have the same
# catchability.
# Count data is modeled using a poisson distribution.
fit.no.q <- jointModel(data = greencrabData, family = "poisson",
p10priors = c(1,20), q = FALSE, multicore = FALSE)
#>
#> SAMPLING FOR MODEL 'joint_binary_pois' NOW (CHAIN 1).
#> Chain 1:
#> Chain 1: Gradient evaluation took 6.4e-05 seconds
#> Chain 1: 1000 transitions using 10 leapfrog steps per transition would take 0.64 seconds.
#> Chain 1: Adjust your expectations accordingly!
#> Chain 1:
#> Chain 1:
#> Chain 1: Iteration: 1 / 3000 [ 0%] (Warmup)
#> Chain 1: Iteration: 500 / 3000 [ 16%] (Warmup)
#> Chain 1: Iteration: 501 / 3000 [ 16%] (Sampling)
#> Chain 1: Iteration: 1000 / 3000 [ 33%] (Sampling)
#> Chain 1: Iteration: 1500 / 3000 [ 50%] (Sampling)
#> Chain 1: Iteration: 2000 / 3000 [ 66%] (Sampling)
#> Chain 1: Iteration: 2500 / 3000 [ 83%] (Sampling)
#> Chain 1: Iteration: 3000 / 3000 [100%] (Sampling)
#> Chain 1:
#> Chain 1: Elapsed Time: 0.908 seconds (Warm-up)
#> Chain 1: 2.159 seconds (Sampling)
#> Chain 1: 3.067 seconds (Total)
#> Chain 1:
#>
#> SAMPLING FOR MODEL 'joint_binary_pois' NOW (CHAIN 2).
#> Chain 2:
#> Chain 2: Gradient evaluation took 6.2e-05 seconds
#> Chain 2: 1000 transitions using 10 leapfrog steps per transition would take 0.62 seconds.
#> Chain 2: Adjust your expectations accordingly!
#> Chain 2:
#> Chain 2:
#> Chain 2: Iteration: 1 / 3000 [ 0%] (Warmup)
#> Chain 2: Iteration: 500 / 3000 [ 16%] (Warmup)
#> Chain 2: Iteration: 501 / 3000 [ 16%] (Sampling)
#> Chain 2: Iteration: 1000 / 3000 [ 33%] (Sampling)
#> Chain 2: Iteration: 1500 / 3000 [ 50%] (Sampling)
#> Chain 2: Iteration: 2000 / 3000 [ 66%] (Sampling)
#> Chain 2: Iteration: 2500 / 3000 [ 83%] (Sampling)
#> Chain 2: Iteration: 3000 / 3000 [100%] (Sampling)
#> Chain 2:
#> Chain 2: Elapsed Time: 0.851 seconds (Warm-up)
#> Chain 2: 2.093 seconds (Sampling)
#> Chain 2: 2.944 seconds (Total)
#> Chain 2:
#>
#> SAMPLING FOR MODEL 'joint_binary_pois' NOW (CHAIN 3).
#> Chain 3:
#> Chain 3: Gradient evaluation took 6e-05 seconds
#> Chain 3: 1000 transitions using 10 leapfrog steps per transition would take 0.6 seconds.
#> Chain 3: Adjust your expectations accordingly!
#> Chain 3:
#> Chain 3:
#> Chain 3: Iteration: 1 / 3000 [ 0%] (Warmup)
#> Chain 3: Iteration: 500 / 3000 [ 16%] (Warmup)
#> Chain 3: Iteration: 501 / 3000 [ 16%] (Sampling)
#> Chain 3: Iteration: 1000 / 3000 [ 33%] (Sampling)
#> Chain 3: Iteration: 1500 / 3000 [ 50%] (Sampling)
#> Chain 3: Iteration: 2000 / 3000 [ 66%] (Sampling)
#> Chain 3: Iteration: 2500 / 3000 [ 83%] (Sampling)
#> Chain 3: Iteration: 3000 / 3000 [100%] (Sampling)
#> Chain 3:
#> Chain 3: Elapsed Time: 0.931 seconds (Warm-up)
#> Chain 3: 1.468 seconds (Sampling)
#> Chain 3: 2.399 seconds (Total)
#> Chain 3:
#>
#> SAMPLING FOR MODEL 'joint_binary_pois' NOW (CHAIN 4).
#> Chain 4:
#> Chain 4: Gradient evaluation took 6.2e-05 seconds
#> Chain 4: 1000 transitions using 10 leapfrog steps per transition would take 0.62 seconds.
#> Chain 4: Adjust your expectations accordingly!
#> Chain 4:
#> Chain 4:
#> Chain 4: Iteration: 1 / 3000 [ 0%] (Warmup)
#> Chain 4: Iteration: 500 / 3000 [ 16%] (Warmup)
#> Chain 4: Iteration: 501 / 3000 [ 16%] (Sampling)
#> Chain 4: Iteration: 1000 / 3000 [ 33%] (Sampling)
#> Chain 4: Iteration: 1500 / 3000 [ 50%] (Sampling)
#> Chain 4: Iteration: 2000 / 3000 [ 66%] (Sampling)
#> Chain 4: Iteration: 2500 / 3000 [ 83%] (Sampling)
#> Chain 4: Iteration: 3000 / 3000 [100%] (Sampling)
#> Chain 4:
#> Chain 4: Elapsed Time: 0.881 seconds (Warm-up)
#> Chain 4: 2.069 seconds (Sampling)
#> Chain 4: 2.95 seconds (Total)
#> Chain 4:
#> Refer to the eDNAjoint guide for visualization tips: https://bookdown.org/abigailkeller/eDNAjoint_vignette/tips.html#visualization-tips
# Fit a model estimating a gear scaling coefficient for traditional
# survey gear types.
# This model does not assume all traditional survey methods have the
# same catchability.
# Gear type 1 is used as the reference gear type.
# Count data is modeled using a negative binomial distribution.
fit.q <- jointModel(data = greencrabData, family = "negbin",
p10priors = c(1,20), q = TRUE, multicore = FALSE)
#>
#> SAMPLING FOR MODEL 'joint_binary_catchability_negbin' NOW (CHAIN 1).
#> Chain 1:
#> Chain 1: Gradient evaluation took 0.000278 seconds
#> Chain 1: 1000 transitions using 10 leapfrog steps per transition would take 2.78 seconds.
#> Chain 1: Adjust your expectations accordingly!
#> Chain 1:
#> Chain 1:
#> Chain 1: Iteration: 1 / 3000 [ 0%] (Warmup)
#> Chain 1: Iteration: 500 / 3000 [ 16%] (Warmup)
#> Chain 1: Iteration: 501 / 3000 [ 16%] (Sampling)
#> Chain 1: Iteration: 1000 / 3000 [ 33%] (Sampling)
#> Chain 1: Iteration: 1500 / 3000 [ 50%] (Sampling)
#> Chain 1: Iteration: 2000 / 3000 [ 66%] (Sampling)
#> Chain 1: Iteration: 2500 / 3000 [ 83%] (Sampling)
#> Chain 1: Iteration: 3000 / 3000 [100%] (Sampling)
#> Chain 1:
#> Chain 1: Elapsed Time: 3.215 seconds (Warm-up)
#> Chain 1: 8.095 seconds (Sampling)
#> Chain 1: 11.31 seconds (Total)
#> Chain 1:
#>
#> SAMPLING FOR MODEL 'joint_binary_catchability_negbin' NOW (CHAIN 2).
#> Chain 2:
#> Chain 2: Gradient evaluation took 0.000321 seconds
#> Chain 2: 1000 transitions using 10 leapfrog steps per transition would take 3.21 seconds.
#> Chain 2: Adjust your expectations accordingly!
#> Chain 2:
#> Chain 2:
#> Chain 2: Iteration: 1 / 3000 [ 0%] (Warmup)
#> Chain 2: Iteration: 500 / 3000 [ 16%] (Warmup)
#> Chain 2: Iteration: 501 / 3000 [ 16%] (Sampling)
#> Chain 2: Iteration: 1000 / 3000 [ 33%] (Sampling)
#> Chain 2: Iteration: 1500 / 3000 [ 50%] (Sampling)
#> Chain 2: Iteration: 2000 / 3000 [ 66%] (Sampling)
#> Chain 2: Iteration: 2500 / 3000 [ 83%] (Sampling)
#> Chain 2: Iteration: 3000 / 3000 [100%] (Sampling)
#> Chain 2:
#> Chain 2: Elapsed Time: 3.382 seconds (Warm-up)
#> Chain 2: 10.66 seconds (Sampling)
#> Chain 2: 14.042 seconds (Total)
#> Chain 2:
#>
#> SAMPLING FOR MODEL 'joint_binary_catchability_negbin' NOW (CHAIN 3).
#> Chain 3:
#> Chain 3: Gradient evaluation took 0.000276 seconds
#> Chain 3: 1000 transitions using 10 leapfrog steps per transition would take 2.76 seconds.
#> Chain 3: Adjust your expectations accordingly!
#> Chain 3:
#> Chain 3:
#> Chain 3: Iteration: 1 / 3000 [ 0%] (Warmup)
#> Chain 3: Iteration: 500 / 3000 [ 16%] (Warmup)
#> Chain 3: Iteration: 501 / 3000 [ 16%] (Sampling)
#> Chain 3: Iteration: 1000 / 3000 [ 33%] (Sampling)
#> Chain 3: Iteration: 1500 / 3000 [ 50%] (Sampling)
#> Chain 3: Iteration: 2000 / 3000 [ 66%] (Sampling)
#> Chain 3: Iteration: 2500 / 3000 [ 83%] (Sampling)
#> Chain 3: Iteration: 3000 / 3000 [100%] (Sampling)
#> Chain 3:
#> Chain 3: Elapsed Time: 3.482 seconds (Warm-up)
#> Chain 3: 8.738 seconds (Sampling)
#> Chain 3: 12.22 seconds (Total)
#> Chain 3:
#>
#> SAMPLING FOR MODEL 'joint_binary_catchability_negbin' NOW (CHAIN 4).
#> Chain 4:
#> Chain 4: Gradient evaluation took 0.000293 seconds
#> Chain 4: 1000 transitions using 10 leapfrog steps per transition would take 2.93 seconds.
#> Chain 4: Adjust your expectations accordingly!
#> Chain 4:
#> Chain 4:
#> Chain 4: Iteration: 1 / 3000 [ 0%] (Warmup)
#> Chain 4: Iteration: 500 / 3000 [ 16%] (Warmup)
#> Chain 4: Iteration: 501 / 3000 [ 16%] (Sampling)
#> Chain 4: Iteration: 1000 / 3000 [ 33%] (Sampling)
#> Chain 4: Iteration: 1500 / 3000 [ 50%] (Sampling)
#> Chain 4: Iteration: 2000 / 3000 [ 66%] (Sampling)
#> Chain 4: Iteration: 2500 / 3000 [ 83%] (Sampling)
#> Chain 4: Iteration: 3000 / 3000 [100%] (Sampling)
#> Chain 4:
#> Chain 4: Elapsed Time: 3.55 seconds (Warm-up)
#> Chain 4: 9.146 seconds (Sampling)
#> Chain 4: 12.696 seconds (Total)
#> Chain 4:
#> Refer to the eDNAjoint guide for visualization tips: https://bookdown.org/abigailkeller/eDNAjoint_vignette/tips.html#visualization-tips
# Perform model selection
jointSelect(modelfits = list(fit.no.q$model, fit.q$model))
#> elpd_diff se_diff
#> model2 0.0 0.0
#> model1 -164.8 39.0
# }
```