A new parametrization transform a set of constrained parameters between a lower and upper bound, such that: p_unc = f§, where p_unc is an unconstrained set of parameters. The gradient of the log posterior must be multiplied by the Jacobian of the inverse transform to sample in the unconstrained space. The Jacobian terms could be very close to zero if the constrained parameters are close to their bounds. I am wondering, how does the algorithm handle this situation in the proposal distribution of the HMC algorithm?
It quite possibly can’t handle this situation numerically, even if you have a well-posed posterior distribution. The NUTS algorithm tries to adapt its tuning parameters during the warmup period but this may be largely unsuccessful when you have log-Jacobian terms that are very large in magnitude for some parts of the parameter space and relatively small in other parts. In that case, at the end you will get a warning about divergent transitions and you may well have to find an alternate transformation to draw from this posterior distribution.
Do you think that using a penalty function with the likelihood may solve the problem? It think it may prevent candidates suggested by the proposal distribution to stay stuck near the bound.
I doubt the Markov Chain will get stuck but NUTS may diverge when a proposal gets too close to a bound. Basically, there should be no posterior density at the bounds, which can be accomplished via a prior that places zero density at the bounds and almost zero density near the bounds.
Stan uses a NUTS-like algorithm by default, which is not a form of Metropolis in the sense of having a proposal distribution and Metropolis-Hastings correction.
If you have probably mass near boundaries, you’ll have statistical and computational problems and will need to take the steps @bgoodri suggests to remediate.
We generally do not recommend interval-bounded priors unless you have physical or logical constraints like requiring a correlation to be in [-1, 1] or a probability to be in [0, 1] or a component of an ordered vector to be between the previous and following member of the vector.