I’m attempting to use the loo package to use some model diagnostics, but getting the following error:

loo(extract_log_lik(model10))
Warning message:
Relative effective sample sizes ('r_eff' argument) not specified.
For models fit with MCMC, the reported PSIS effective sample sizes and
MCSE estimates will be over-optimistic.

Does this warning error apply to Stan models, which are fit with NUTS (rather than MCMC)? If so, then how should I specify r_eff using the output of the stan regression?

NB: model10 is fit using the following Stan code.

data {
int<lower=0> N; //number of obs
vector[N] QT;
vector[N] sqrtRR;
}
parameters {
real alpha;
real beta;
real<lower=0> sigma;
}
model {
alpha ~ normal(0, 10000);
beta ~ normal(0,10000);
sigma ~ cauchy(0, 5);
QT ~ normal(alpha + beta * sqrtRR, sigma);
}
generated quantities {
vector[N] log_lik;
vector[N] yrep;
for (i in 1:N) {
log_lik[i] = normal_lpdf(QT[i] | alpha + beta * sqrtRR[i], sigma);
yrep[i] = normal_rng(alpha + beta * sqrtRR[i], sigma);
}
}

It’s just a warning not an error. elpd_loo and SE for elpd_loo will be same without r_eff, but diagnostic are more accurate if we take into account that draws are not independent. PSIS-LOO computation itself is invariant for permutation of draws, but to estimate the efficiency of MCMC draws we need to know chains and have draws in order. See the updated vignette for instructions how to this.

As loo package can be used also with other software than Stan, MCMC here refers to any MCMC algorithm.

Also, in the next release of RStan there will be a loo() method for stanfit objects that does the r_eff computation automatically, so it will save you this step. You’ll still have to code up log-lik in the Stan program but you won’t have to worry about r_eff. In the meantime, take a look at the vignette Aki linked too as well as the relative_eff helper functions that we added in the latest release:

In brief, the current recommended usage with RStan is