I was looking more into ADVI (and taking a shot at breaking it up into more sensible pieces…) as part of the output refactor. It struck me that I can’t tell from the ADVI paper why they rely on stochastic gradient ascent (and as an aside, why we don’t just implement stochastic gradient ascent as a …

LBFGS requires accurate deterministic gradients, but in ADVI only stochastic gradients are available. Stochastic gradient ascent with decreasing step size is simple to implement. There are alternative step size adjustment algorithms some which have been used a lot in ML and some of of those might be…

In any case it sounds like the optimization step might be nice to abstract out of the advi implementation.

The gradients in ADVI are not gradients with respect to the log target density but rather the expected lower bound (ELBO) of the variational objective. The ELBO can be written as an integral which in ADVI is approximated with a Monte Carlo estimator. The gradient of that integral is then approxima…

This is helpful, thanks. I think this is also why these procedures are harder to break down into reasonable modules: there are just a lot of somewhat arbitrary pieces. Not that it can’t be cleaned up but I can see the gains are limited.

[image] sakrejda: In any case it sounds like the optimization step might be nice to abstract out of the advi implementation. From what I hear from talking to Matt Hoffman and Alp Kucukelbir (authors of NUTS and ADVI), there’s likely to be a lot of people working on the optimization angle of A…

[image] betanalpha: Regardless, stochastic gradient methods can be extremely fragile and require careful hand-tuning in order to yield reasonable results, especially for nontrivial objectives. This was also Breck’s and my experience building a lot of large-scale (millions of training data poi…

I have an ADVI re-write that I should push to the repo as an example of what I’m thinking, haven’t had the time to.

[image] Bob_Carpenter: The official Robbins-Monro learning rate schemes are way too slow, but more aggressive approaches require more tuning. It’s why I always take any SGD results with a grain of salt, as they’re often dependent on lots of grad-student hours tuning. Good thing about the new …

[image] avehtari: if optimization fails That’s good to hear. I need to understand the diagnostics better.

Why does ADVI use stochastic gradient ascent not LBFGS

Algorithms

sakrejda June 29, 2018, 6:12pm 3

Thanks that’s very helpful. I must’ve missed this in the algorithm just b/c it calls the same gradient calc as everything else. I’ll have to spend more time with it.

Topic		Replies	Views
How does ADVI select mini-batches for stochastic gradient ascent? Algorithms	3	616	April 7, 2021
ADVI / Stochastic quasi-Newton methods Algorithms variational-bayes	5	1398	September 22, 2017
Stan Variational Inference Deprecation and Documentation Developers fitting-issues	2	273	April 25, 2025
Does ADVI stepsize choice have the same problem with its stopping rule? Algorithms	5	669	December 21, 2020
Stochastic subsampling in ADVI Algorithms variational-bayes	1	706	June 18, 2019

Why does ADVI use stochastic gradient ascent not LBFGS

Related topics