Initialize variational approximation to speed-up sampling

Is it possible to pass an initialization to the variational approximation parameters when calling vb?

1 Like

Yeah, the init argument can be used to pass initial values for all or a subset of parameters.

1 Like

That’s initial values for parameters, but not for variational parameters? That is, I think it’s not possible to provide initial values, for exampe, for the variances of the meanfield normal approximate distribution.

1 Like

Yeah that’s for the model parameters. I think you’re right that you can’t provide inits for the parameters of the approximating distribution. Or maybe that’s possible in the C++ (I don’t know), but if it is I don’t think it is exposed to users.

So can I say that: when we use init params in VI, we actually set the init values for the model, and STAN VI will use these values to initialize the random values of the approximated distribution but not the parameters (mean and variance under Gaussian approximation) of the approximated distribution ?

The init in interfaces sets the mean of the approximate distribution, but you can’t currently set the variances.

Oh, I see. I got a misunderstanding here. Thank you very much. So in this way, initialization is indeed important for VI though the variances is out of control.

My current procedure is to estimate the initial values, and then use optimization to get a possible better initial values, and then send them to VI. Do you think this procedure is reasonable ?

Thanks for your patience!

Can I say initialization is not that import for HMC since HMC has the warmup stage and the initialization is only used during warmup stage? Or actually initialization in HMC are used in both warmup stage and sampling stage ?

Out of control in the current Stan interfaces. This could be changed.

Optimization as in optimization with L-BFGS? Tha works if the posterior is close to normal, but is bad if the posterior is very skewed that is common for hierarchical models,

See sections 5.6 and 11 in Bayesian workflow paper for example where initialization can be important for HMC, too. Also good initialization can made warm-up slightly faster, but the current warm-up in Stan is designed to be robust in many cases instead of being fast in easy cases.

1 Like

Thanks for your comments. They are usually helpful!
The paper seems interesting! I will read it.