Brms variational inference and importance_resampling

Interfaces brms
1

I am using brms and since mcmc was taking a long time, I tried using algorithm = "meanfield"

I am using the rstan backend, and I was surprised to get a warning about Pareto K diagnostics. Upon further investigation I found that rstan now will do PSIR to improve the meanfield posterior samples (cool!) but in my case it should be disabled as stated by the warning:

Warning: Pareto k diagnostic value is 5.31. Resampling is disabled. Decreasing tol_rel_obj may help if variational algorithm has terminated prematurely. Otherwise consider using sampling instead.

But is it really disabled? Looking at the code, I notice that the line in stanmodel-class.R where it should be then disabled is commented out? Is this working as intended? (this section starts on line 280 on develop )

              if (p$diagnostics$pareto_k > 1) {
                warning("Pareto k diagnostic value is ",
                        round(p$diagnostics$pareto_k,2),
                        ". Resampling is disabled.",
                        " Decreasing tol_rel_obj may help if variational algorithm has terminated prematurely.",
                        " Otherwise consider using sampling instead.", call. = FALSE, immediate. = TRUE)
                #importance_resampling <- FALSE

brms version: 2.22.0
rstan version: 2.35.0.9000

I also tried importance_resampling=FALSE but probably that is not how to do it. I didn’t detect any significant difference.

Edit: Oh , I suspect since rstan sets importance_resampling= FALSE by default this is probably not changed by brms. So perhaps it is not happening in any case

2

Doing a little source diving I was able to confirm that PSIR is not happening in any event (it must be explicitly enabled) and this message appears even when disabled, perhaps intended as a helpful diagnostic about the quality of the variational approximation.

I found the message a bit confusing, but perhaps that is just me. And after all there is a big “experimental” warning at the top !

PS: For my case I ended up sticking with MCMC which although slower, does not have to be run very often.