So I want to build iterations of univariate and multi-response models with different numbers of variables and determine which model is the best? I’ll use variational inference to do this - but I actually don’t know what the steps are to complete a variation inference test. Setting the algorithm to “meanfield” or “fullrank” seems to be involved, but what comes next?
I can’t seem to find a vignette for brms that guides me through the process.
Can someone provide some guidance for variational inference?
Thanks!
Running the sampler is going to be a better way of doing this.
VI complicates things in two ways.
First the convergence diagnostics are not as good as the ones for Stan.
Second the approximation may work better or worse across different models, and so you might be picking between them partially based on the VI implementation in Stan.
If runtime is a problem, try to think up a smaller toy dataset for the problem and use the sampler. Once you have a model you can work on moving from the sampler to VI (seeing if the VI results are reliable and match the sampler results enough).
1 Like
Can you unpack this sentence for me? Where does the comparison between models come in? Can I simply use loo/bayes_R2/waic for model selection?
Model selection is a big thing I don’t know much about, but here’s a big page on it with a lot of Stan stuff: Model selection tutorials and talks
My comment about VI vs. the sampler is, in any case, your model selection is going to be based on some statistics which you are using VI or the sampler to compute.
The sampler has better diagnostics so you know when you’ve done the statistics you were trying to do better than VI. By trying to work all this out with VI, you might be making decisions based on the VI approximations rather than the statistics.
Edit: Added missing word
3 Likes
Wow, that is a good link. Thanks for sharing it!
1 Like