Getting warning with `loo` package regarding r_eff

Hi,

I’m attempting to use the loo package to use some model diagnostics, but getting the following error:

loo(extract_log_lik(model10))

Warning message:
Relative effective sample sizes ('r_eff' argument) not specified.
For models fit with MCMC, the reported PSIS effective sample sizes and 
MCSE estimates will be over-optimistic. 

Does this warning error apply to Stan models, which are fit with NUTS (rather than MCMC)? If so, then how should I specify r_eff using the output of the stan regression?

NB: model10 is fit using the following Stan code.

data {
  int<lower=0> N; //number of obs

  vector[N] QT;
  vector[N] sqrtRR;
}

parameters {
  real alpha;
  real beta;
  real<lower=0> sigma;
}

model {
  alpha ~ normal(0, 10000);
  beta ~ normal(0,10000);
  sigma ~ cauchy(0, 5);
  
  QT ~ normal(alpha + beta * sqrtRR, sigma);
}

generated quantities {
  vector[N] log_lik;
  vector[N] yrep;
  for (i in 1:N) {
    log_lik[i] = normal_lpdf(QT[i] | alpha + beta * sqrtRR[i], sigma);
    yrep[i] = normal_rng(alpha + beta * sqrtRR[i], sigma); 
  }
}
1 Like

First, see r_eff instructions in the updated vignette at CRAN or http://mc-stan.org/loo/articles/loo2-with-rstan.html

It’s just a warning not an error. elpd_loo and SE for elpd_loo will be same without r_eff, but diagnostic are more accurate if we take into account that draws are not independent. PSIS-LOO computation itself is invariant for permutation of draws, but to estimate the efficiency of MCMC draws we need to know chains and have draws in order. See the updated vignette for instructions how to this.

As loo package can be used also with other software than Stan, MCMC here refers to any MCMC algorithm.

1 Like

Also, in the next release of RStan there will be a loo() method for stanfit objects that does the r_eff computation automatically, so it will save you this step. You’ll still have to code up log-lik in the Stan program but you won’t have to worry about r_eff. In the meantime, take a look at the vignette Aki linked too as well as the relative_eff helper functions that we added in the latest release:

In brief, the current recommended usage with RStan is

loglik <- extract_log_lik(stanfit, merge_chains = FALSE)
r_eff <- relative_eff(exp(loglik))
loo(loglik, r_eff = r_eff)

and in the next release you can use the shortcut

loo(stanfit)
1 Like

NUTS is MCMC, just a particular variant.

So, like Aki said, it applies to any MCMC algorithm, which mean NUTS too.

1 Like

thanks! i wrote a wrapper for it, which seems like is one way to do it:

get_loo <- function(model) {
  log_lik <- extract_log_lik(model, merge_chains=FALSE)
  r_eff <- relative_eff(exp(log_lik))
  loo(log_lik, r_eff=r_eff, cores=2)
}

the vignettes were extremely helpful.

2 Likes

Yeah that’s basically what the method in the next RStan does:

https://github.com/stan-dev/rstan/blob/3f7831bd07cd99bbc97ee77136aa4e33e65a92ff/rstan/rstan/R/loo.R

1 Like