Interpreting R-hat is NA warnings and unable to implement reloo

Interfaces brms
1

Hello all,

I am quite new to brms and I apologise in advance for any elementary mistakes. I have a multivariate model with four dependent variables (DVs), some factors and a bunch of human participants. The DVs are different rating scales (from 1-5) based on evaluating a set of images. I also have a factorial design, such that the images fall into different categories. Below is an example model (it takes hours to build hence why I have not included a reproducible version). I only include one factor below for simplicity. I have a varying intercept for ‘item’ = the image (or stimulus) and a varying intercept and slope for participant. The model is here:

brm(mvbind(DV1, DV2, DV3, DV4) ~ 1 + factor +
(1 + factor |a|participant) +
(1|item),
data = data, family = cumulative(“probit”),
prior = priors,
iter = 6000, warmup = 1000, cores = 4, chains = 4,
control = list(adapt_delta = 0.99, max_treedepth = 15),
init_r = 0.1)

After running the model, I get the following error (error 1). Although when I look at the model, the R-Hat values seem fine.
I can see that this has been posted before: https://github.com/paul-buerkner/brms/issues/865

But, as a Brms newbie, I am not sure how to proceed. Any help would be appreciated.

Error 1.

Warning messages:

1: The largest R-hat is NA, indicating chains have not mixed.

Running the chains for more iterations may help. See

http://mc-stan.org/misc/warnings.html#r-hat

2: Bulk Effective Samples Size (ESS) is too low, indicating posterior means and medians may be unreliable.

Running the chains for more iterations may help. See

http://mc-stan.org/misc/warnings.html#bulk-ess

3: Tail Effective Samples Size (ESS) is too low, indicating posterior variances and tail quantiles may be unreliable.

Running the chains for more iterations may help. See

http://mc-stan.org/misc/warnings.html#tail-ess

The second issues concerns using reloo=TRUE. After using ‘add_criterion(model_name, “loo”)’ to the model above, there are a few observations with high pareto_k values. If I follow the printed advice and use reloo=TRUE, it runs for a bit and then returns error 2:

Error 2.

[1] “Error in sampler$call_sampler(args_list[[i]]) : Initialization failed.”

The desired updates require recompiling the model

Compiling the C++ model

recompiling to avoid crashing R session

Start sampling

error occurred during calling the sampler; sampling not done

Error: The model does not contain posterior samples.

On other models that I’ve run in the past, reloo=TRUE has worked fine. Therefore, any thoughts or advice on how to proceed would be appreciated.

Regards,
Rich

A few further things to note.

  • With test data and simpler models, I can run models fine (without errors) and reloo=TRUE also works without error. So, it seems Brms is working ok on my machine.

  • I am running brms 2.13.1 on MacOS 10.14.6

2

Hi Rich,

from the github bugreport that you provided, I take it that this is actually not a problem with brms itself but is a problem in RStan. As @paul.buerkner says in the link you provided:

“the described warning pops up but this happens directly from within Rstan and cannot be controlled by brms”.

From personal experience, and because simpler models do not cause this problem for you, might this problem be caused by model misspecification? Are you sure that, for instance, all random effects in the model can be estimated or is the model maybe incorrectly or overspecified?

Maybe it helps if you try to first fit 4 univariate mixed-models and see whether you encounter the same problems, before slowly adding the variables 1-by-1 into a multivariate model. Alternatively, maybe try to remove all random slopes from the model and see whether a random-intercept only model works before adding random slopes.

Of course the problem could also be more technical on the site of RStan, in which case I also do not know how to fix it…

EDIT: I also see now that you specified init_r = 0.1. Was that necessary to get the model to sample to begin with? In that case might it be the case that initilization of the sampler only fails in some but not all cases?

3

Thanks for your response. It is much appreciated.

  • Intercepts only works just fine.
  • I’ll try the univariate models, as you suggest.
  • Regarding init_r = 0.1. Yes, I read online that if it has trouble starting, use inits = 0 or init_r = 0.1. Model with many (correlated) varying-slopes: Sampling not done. Although to be honest, I’m not 100% sure how this impacts things more generally. Hence, I cannot answer your question regarding initialization fails. Which cases are you referring to?

Thanks again.
Rich

4

As to your last point: inits and init_r provide starting values for model parameters in the MCMC sampling. See this post for a more elaborate explanation of this, but in short, the sampler will take the initial values and evaluate the log-posterior based on them. If this evaluation fails, i.e. it returns Inf / -Inf the sampler will try different initial values. However, it can happen that the initiation fails over and over again and does not even start. The init_r = 0.1 tells the sampler to try initial values between -0.1 and 0.1 (instead -2 and 2 which is the default). As these starting values are in fact sampled randomly, it might happen that in some cases the sampler will end up with starting values that work and beginn sampling, and in other situations it will not and show you the “initiation failed” message you describe above. It can also happen that some chains start sampling and others fail to initialize. My hinge is that this is what might have happend in your case: The model initializes fine when you first run it, but during reloo, when it is refit, it fails to initialize (for some chains) and returns you the error.

Have you tried the other option (specifying inits = "0") ? This often solves initiation problems and if it works it will work all the time because the starting values are always the same.

Another point with reloo is that I am not sure whether the specification of init_r = .10 or inits = "0" must be directly passed to reloo by specifying e,g, reloo_args = list(inits = "0") (@avehtari I hope it is ok if I tag you but is this the case? A quick google search did not help me to find out if this was necessary) .

I hope this helps!

5

Thanks so much. That’s very helpful indeed.

I ran the univariate models separately, as you suggested. reloo=TRUE works fine (and is only required for one DV - out of four DVs). What do you think makes the difference in the multivariate model context? Any ideas?

I still get the first error warning (Error 1 posted above) for all models (including the intercepts only models). As such, I am still unsure how to interpret this or deal with it. Any thoughts?

Thanks again.

Rich

6

I am not sure… Have you already tried fitting the multivariate model with specifying inits = "0" instead of init_r = .10? If that does not solve it, maybe sharing the output of summary() for the multivariate model would provide some insight…

It is also, of course, still a possibility that this warning is caused by something inside RStan which is not related to the model-fit itself, in which case I have no idea why this happens and how to solve it.

7

Just an update.

reloo = TRUE works for the multivariate model if you inlcude: reloo_args = list(inits = "0")

Thanks so much for your help.

1 Like