Hello,
I have a conceptual question. I use variational inference to fit my model to the data. But for different runnings, with same values of ELBO, I obtain different posterior distributions for parameters (regarding mean, std,…). I guess it is due to local optimums. Is it the reason?! if so, how should I know which one is the correct answer from different tries! (because I have same elbo). When I use HMC, the posterior of parameters are very wide and flat!
Thank you very much in advance for your comment.
Yes there are local optima. Also, ADVI draws parameters in the unconstrained space from a multivariate normal distribution whose parameters are given at the optimum it lands on. So, these 1000 or whatever realizations are random and will introduce some variability into the summaries.
Thanks a lot for your comment.
So how should know which solution is the best solution (i.e, global min)?!
Do you have a comment why using HMC, I got very flat and fat posterior violin plot?!
Thanks.
Well, you can look at the values for the objective function that come up on the screen to see which is the best optimum that is found. But if there are many, it is hard to know whether you have found all of them. I don’t know anything about your model, but if you can get it to work with NUTS, then why worry about ADVI?
Thanks again. Yes there are many optima and I get same values for elbo.
About the HMC, my problem is getting such a uniform posterior in the physiological bounds of the parameters. (When I do a violin plot for the posterior of parameters, all of them give fat and flat distributions in the defined bound for parameters! :(