I know generally MCMC is poor to sample from multi-modal densities, but in the Metropolis Hastings algorithm, a heavy tailed proposal can be used so that there is more chance of moving between modes. For instance using a Cauchy proposal rather than a normal proposal can help mixing of the chain.
Is there anything in Stan that can be tweaked to help sample from multi-modal posteriors?
Or is the only option is to use alternative methods like parallel tempering?
I think the current consensus is that the best way is a tempering scheme like Adiabatic Monte-Carlo.
In theory you can have heavy-tailed momentum proposals in HMC, but in practice they’ll make the leapfrog steps go unstable. Stan currently doesn’t have that option probably for the aforementioned reason.
What kind of modes are we talking here? Do you know a-priori how many modes there are? You can always try having a different chain in each mode.
Thanks for the response! This is just in general so I don’t know how many modes there apriori.
If we did know where the modes were, wouldn’t starting the chain in each one not be that great either since we could over sample a mode cause we cannot escape them?
I think if you did know where the modes are then, like you said, you can start a chain there. Yes, it probably wouldn’t escape, but at least you’d be able to quantify the uncertainty for that specific mode.