Variational Bayes runtime and memory usage


I am using vb() to fit a large model. The model fits just fine, however rstan takes a very long time between when the console states that it has drawn a sample from the approximate posterior and that it has “COMPLETED” until it actually finishes. During that time, which has been as long as an hour, there is intense CPU activity and very large spikes in memory usage (I noticed because my computer seized up for a bit when the memory pressure became severe). What is stan doing at this point? Why does it suddenly require far more memory than is required at any previous point in the vb() function? I am drawing only a tiny posterior sample of a small number of parameters, that is not what is using more than the 32GB of memory that I have in my computer.


I’m not sure if this is related, but I’ve noticed the same thing when fitting a large model using NUTS in pystan. Is it perhaps due to Stan computing some summary statistics on the posterior samples?


It reads off the disk, but I don’t think the I / O should be that noticeable.


I have encountered this before also.

After a bit of experimentation I concluded (perhaps incorrectly) that this occurs when you have large numbers of parameters.

The ADVI algorithm has computational complexity O(N*p) (I think), where N is the number of daya points and p is the number of parameters, whilst the generation of samples has complexity O(p^3) due to the Cholesky decomposition required.

If you have a large number of parameters the runtime for the second operation (what you describe) can be substantial.