Errors when using loo_subsample()

Modeling
,
1

I’m currently trying to use loo_subsample() to compute k-fold CV on a hierarchical model.

On first attempt I used the likelihood function llfun_normal() defined below, with inputs of draws (list) and data (df). This passes all of the initial steps (calculating r_eff, loo_i) but fails when implementing loo_subsample with the error message printed below:

llfun_normal <- function(data_i, draws, log = TRUE) {
  pred <- (draws$bCons[,data_i$S] + draws$bAcc[,data_i$S] * data_i$Acc) * (1 + draws$bSymm[,data_i$S] * data_i$Dir)
  dnorm(x = data_i$UpdateS, mean = pred, sd = draws$sigma_ID[,data_i$S], log = log)
}

draws <- rstan::extract(sampledModel, pars = c("bCons", "bAcc", "bSymm", "sigma_ID"))
data <- as.data.frame(stan_data)

> Error in .ndraws.default(draws) : 
  .ndraws() has not been implemented for objects of class 'list'

I managed to fix this by doing some very ugly adjustments and turning the list of draws into a matrix and changing the likelihood function to the below:

llfun_normal3 <- function(data_i, draws, log = TRUE) {
  pred <- (draws[,c(1:154)][,data_i$S] + draws[,c(155:308)][,data_i$S] * data_i$Acc) * (1 + draws[,c(309:462)][,data_i$S] * data_i$Dir)
  dnorm(x = data_i$UpdateS, mean = pred, sd = draws[,c(463:616)][,data_i$S], log = log)
}

draws3 <- cbind(draws$bCons, draws$bAcc, draws$bSymm, draws$sigma_ID)

However, this now produces the below error message from loo_subsample(). This occurs when using the “plpd” loo_approximation method, but doesn’t occur when using “lpd”. I could therefore use lpd, but have read in the recent references that plpd is preferable.

Error in psis_loo_ss_object(x = loo_obj, idxs = idxs, elpd_loo_approx = elpd_loo_approx,  : 
  Assertion on 'elpd_loo_approx' failed: Must be of type 'numeric', not 'character'.
In addition: Warning message:
In mclapply(X = seq_len(N), mc.cores = cores, FUN = lpd_i, llfun,  :
  all scheduled cores encountered errors in user code

The example vignette from the link below runs fine when using plpd:
https://cran.r-project.org/web/packages/loo/vignettes/loo2-large-data.html

Any insight into this would be hugely helpful. Thanks for your time.

2

@avehtari

3

I missed this first time, was last week on vacation, and right now my R setup is not working, so I’m not able to check what should be the recommended code. I’ll get back to this later this week.

4

Now I have working R again. Can you try forming the matrix with

library(posterior)
draws3 <- subset_draws(as_draws_matrix(sampledModel), variable = c("bCons", "bAcc", "bSymm", "sigma_ID"))

The posterior package knows how to transform between different data types for draws. The cbind did mess the attributes which probably confused the later code.

1 Like
5

Thanks, Aki!