I’m working with Pathfinder on a model that is pretty gnarly – lots of terribly geometry. We’ve been trying to reparameterize for quite a while now and it’s an ongoing project. It’s about a thousand lines of Stan code, so a bit difficult to debug without a lot of eyes. About 66k parameters, most of which are basically random effects I’m trying to marginalize.
We’ve been trying Pathfinder on it to see what happens, and we’re getting Pareto k-values on the order of 63-75. Significantly above reasonable thresholds of 0.5 or 0.7.
My question – does this tell us anything about the model? Aki has suggested that this indicates misspecification, but does anyone have an intuition about what kind of misspecification that might entail?
Could one cause be that there are significant “ridges” in the true posterior that are too steep to be captured by Hessian approximations?
I know that’s difficult to answer without the model, but any top-of-mind ideas would be appreciated.
That link is to discussion of high khats when using PSIS-LOO, and doesn’t apply here.
High khats when using Pathfinder indicate that the target is not well approximated by the normal distributions. That is as likely to happen with well-specficied and mis-specified models.
khat diagnostic is based on diagnosing whether the approximate distribution could be used as importance sampling distribution, and usually distribution of importance sampling ratios gets nasty when the number of dimensions increase (ee, e.g. Section 3.3 in [1507.02646] Pareto Smoothed Importance Sampling, and Challenges and Opportunities in High Dimensional Variational Inference). You have a very large number of dimensions so it is likely that khats are big, even if the Pathfinder would get means and variances close to true means and variances. The updated Birthdays example shows use of Pathfinder with posteriors that cause high khats (largest were >10).
3 Likes
Excellent, thank you for the overview!