The recent paper by Yao et al. on Stan’s variational inference is pretty clear that lowering the threshold for change in ELBO will improve the fit of the approximated posterior and is more likely to pass the diagnostic checks they propose (see figure 2 in the paper). However, the current default in rstan, option tol_rel_obj, is still set at 0.01. Given the fact that rstan settings for full Bayesian inference are conservative (2000 iterations), it would seem a good practice to change the default as well for vb to use a more conservative standard for ELBO convergence, such as 10^{-4}.
I’m happy to put in a pull request to the rstan Github but I wanted to post on here first to get feedback on the idea.
Currently, the default setting is when running average or running medium of the objective function (elbo) falls below 1%. In many optimization problems, this is not a horrible condition.
A more conservative stopping rule help, but it is also not clear how conservative the threshold should be.
When k hat is outputted, we can also use k hat itself as a stopping rule.
Ok though K-hat can only be computed on a fitted model, right? It would seem that to “make robustness the first goal” @Bob_Carpenter, the initial run should have a strict threshold that more experienced users can adjust when wanting to do quicker optimization runs.
Also is there a plan to include k-hat calculations as part of the vb function (i.e. doesn’t that involve having the user program generated quantities first)?