PSIS-LOO+ warns about an "unreliable value"

Modeling
#1

It’s a categorical model with 4 outcome categories, 2 group-level effects and 46 x 3 = 138 population-level effects, fit using brms. When I calculate elpd_loo using loo(mymodel, reloo = TRUE), the model is re-fit once due to a problematic observation. This re-fitting finishes fine, and a loo object is returned. However, the following warning is printed:

Warning message:
UNRELIABLE VALUE: Future (‘’) unexpectedly generated random numbers without specifying argument ‘[future.]seed’. There is a risk that those random numbers are not statistically sound and the overall results might be invalid. To fix this, specify argument ‘[future.]seed’, e.g. ‘seed=TRUE’. This ensures that proper, parallel-safe random numbers are produced via the L’Ecuyer-CMRG method. To disable this check, use [future].seed=NULL, or set option ‘future.rng.onMisuse’ to “ignore”.

However, it seems to me that loo() doesn’t have an argument called seed. Nonetheless, just to make sure, I tried just now to run loo() on my model again, with reloo = TRUE and seed = TRUE. After another 90 minutes or re-fitting the model without the problematic observation, I get the same error again.

It might also be worth pointing out that the new reloo object is not identical to the first. Elpd_loo and the other diagnostics differ by a tiny amount (about 0.015 points). But I guess that is attributable to the fact that the refitting was done using different random seeds.

The question is: given that I’m still getting the warning, can I trust the reloo diagnostics or not?

#2

bump

#3

I haven’t seen this before but it looks like a warning coming from the future package that brms uses, presumably to parallelize model refitting during reloo. It’s definitely not coming from the loo package. For the model fitting done during any particular use of reloo(), as long as the seeds used are different from each other it should be fine.

@paul.buerkner Have you seen this before?

#4

Yeah the future package recently updated to throw that warning when an rng was used without explicitly setting a seed (all the new red warnings did make me panic a bit when I ran some code!). It’s an easy fix, just need to add seed = NULL to the future call if an rng isn’t needed, or set the seed to some value.

3 Likes
#5

Thanks! I will add this to brms.

#6

Does all this mean that I can use the loo object normally despite the warning?

#7

Yeah I think so.

1 Like
#8

The warning will no longer appear with the brms github version.

5 Likes
#9

Thanks guys.

1 Like
#10

Just ran into this myself using the future package with brms. @paul.buerkner Is this indicative that there is actually a problem with the random number generation when fitting a model? i.e. would installing the github version of brms fix an actual problem or would it just make the warning go away?

#11

I don’t know for sure to be honest but I don’t think so.

1 Like
#12

Turns out to be moot because in my own experimentation, future=TRUE turned out to be quite a bit slower than future=FALSE and just setting the number of cores normally.