This topic is a follow-up of the discussion in today’s Stan weekly meeting. What I am interested in is a general shape of the credible interval (CI) of posterior samples in the unconstrained space. Is it reasonable to make an assumption that, in most cases, we can find a hyper-ellipse that covers and is mainly covered by the majority of posterior samples in the unconstrained space.
Let’s take dimension = 2 for example. In the following three plots, I use black circles to represent “posterior samples”, the red ellipse is estimated based on sample covariance. The “posterior samples” in those plots are artificial and the plots are just for illustration. According to the plots, the first two examples satisfy the assumption
While the third example, obviously, violates the assumption.
I am wondering what type of models might have a boomerang shape of the CI in unconstrained space? What might lead to a violation of the assumption? Is it common in Bayesian modeling?
Perhaps a more interesting question would be, how often a Variational Bayesian method using Gaussian approximation fails in practice. Of course a model that can be well approximated by Gaussian also has the 95% CI of posterior samples in an ellipse shape. So the assumption I am interested in should be more general than the condition for having reliable results through Variational Bayesian method like ADVI.
Any thoughts, ideas or interesting examples are welcome. Thanks!