I am working with a model where a location-dependent metric is required to efficiently sample the posterior distribution.
I have my own implementation of the simplified manifold MALA (smMALA) algorithm from Girolami and Calderhead (2011).
The covariance of the proposal distribution in my smMALA algorithm is proportional to the inverse Hessian of the posterior evaluated at the current parameters.
One issue I have come up against is that the Hessian is not always positive-definite, i.e., it sometimes has negative eigenvalues. As a result of this, the covariance of the proposal distribution is not valid.
A simple solution is to add a constant diagonal matrix to the Hessian, e.g. diag(-2*min(lambda) ), where lambda is the set of Hessian eigenvalues. This ensures that the covariance of the proposal distribution is positive-definite.
This solution generally has the effect of decreasing the overall variance of the proposal distribution, sometimes quite significantly.
Are there any better solutions to this problem? And does the Stan RHMC prototype address this issue at all?