LOO error

Interfaces brms
1

Hello!

I recently updated my R version to the latest including all of the packages. After the update, the LOO function does not work anymore. I have saved a workspace with estimated models and previously I was able to estimate both LOO and WAIC for all of the models but now I get the error for one model if I try to run LOO on the fitted brms object. The error is the following: “Error: Can’t mix NA and not NA values in ‘r_eff’.”.

I fitted the model again, but the problem remains.

Please also provide the following information in addition to your question:

  • Operating System: Windows 10
  • brms Version: 2.9.0
2

Hi, this should not be happening. Can you provide a reproducible example for me to check what’s going on?

3

Hi! How can I provide a reproducible example? The non-working issue (for me) seems totally random as for some models it works well but for certain estimations it gives an error. WAIC I can estimate. The output for the same model:

waic(localrd_stud)

Computed from 4000 by 5044 log-likelihood matrix

      Estimate     SE

elpd_waic -33858.4 1852.1
p_waic 939.8 121.3
waic 67716.7 3704.1
Warning message:
234 (4.6%) p_waic estimates greater than 0.4. We recommend trying loo instead.

loo(localrd_stud)
Error: Can’t mix NA and not NA values in ‘r_eff’.

Probably I am doing something wrong but re-estimating did not solve the issue. In addition, with the older version, I was able to estimate the loo for all of the brmsfit objects.

Best,
H

4

I had an idea. Could you try it out with the latest github version of brms to be installed via

if (!requireNamespace("devtools")) {
  install.packages("devtools")
}
devtools::install_github("paul-buerkner/brms")
5

Hello! Unfortunately the problem persist, I try to run the regressions once more and then come back to the issue if the problem is not solved.

Best, H

6

Hello!

I ran the models again. For smaller sub sample (n=594) both Loo and Waic statistics are calculated as they should. However for the second, bigger (n=6327), sample I can only compute WAIC, but not loo. The error remainds the same:

loo(localrd_stud)
Error: Can’t mix NA and not NA values in ‘r_eff’.
waic(localrd_stud)

Computed from 4000 by 6327 log-likelihood matrix

      Estimate     SE

elpd_waic -42711.8 1931.8
p_waic 992.9 100.4
waic 85423.5 3863.5

I have cleaned the dataset properly, uninstalled and installed my R and Rtools plus loaded the brms package from the provided code above.

Best, H

7

That’s unfortunate, but I have no way to fix this without a reproducible example.

If you like, you can send me the model producing these problems via email (paul.buerkner@gmail.com) if you don’t want to share it on discourse.

8

Hi Paul, I am having the same issue when I try to call the loo function through brms on my glmms with 3864 observations. The waic function still works.

9

Thanks @HTas for providing an example via email! I was able to fix the problem and it should now run again after installing brms from github via:

if (!requireNamespace("devtools")) {
  install.packages("devtools")
}
devtools::install_github("paul-buerkner/brms")
3 Likes
10

I just had the same issue and the update fixed it.

1 Like
11

Actually, the update fixed the error when using add_criterion. But now the error popped up again when comparing two models using loo_model_weights.

Reproducible example (reloo should be on, but it makes things slow, and it doesn’t impact the error)

Data: https://www.dropbox.com/s/soaajqkh36uoa5n/data.csv?dl=0

Code: https://gist.github.com/fusaroli/16c0f33d14fb518fd1e1c6bd7e3fd84c

12

Thanks! Apparently I changed this in one place but not in another. It should now apply to all cases, but I didn’t check it with the code you provided. Would you mind testing it yourself with the latest github version of brms?

13

sorry, but as per the version I just downloaded, the problem with loo_model_weights persists.

14

Ok. I will test it with the data and code you provided and sort this out.

15

Hmm, your code works for me in the latest github version. In particular, when running loo_model_weights(m1,m2) I get the warning

 Ignoring relative efficiencies as some were NA. See argument 'r_eff' in ?loo::loo for more details.

which tells me that the problematic case was handled correcty.

16

Dear Paul,

I am also getting this error, with the github version. My regression was fitted with model weights, any case that that has not been fixed? Let me know if you need the data and code. Thanks!

17

I need to see a minimal reproducible example.

18

Sorry, I just re-updated and ran everything again, and it does work! Thanks a lot