Why not apply the monitor function to the samples from the comparison model?
My understanding is that any MCMC algorithm should be run as multiple chains to ensure there is no influence of the initialization context. Is this provably not a concern with your comparison method?
Which model from the repo is this? I don’t have time to look at it in any detail until the weekend, but two notes:
- your model implies a cauchy prior on
tau; you’re using the more efficient tan-uniform parameterization, but I believe that recent advice is to still avoid these unless they are strongly motivated by concrete theory. Previously folks used heavy-tailed priors to (I think) attempt to make things more “robust” in some sense by allowing more mass to extreme values. However, it turns out that this causes a posterior geometry that is very difficult to sample, causing Stan to degrade in performance dramatically. (It is possible that your alternate methods appear to sample more efficiently because they are not as sensitive to the true geometry and therefore seeming to perform better while being less accurate. Note I say “possible” because I can’t speak likelihood; just wanted to put this possibility on your radar) - usually find that in hierarchical models there is often a lot of redundant computation in the more obvious approaches to coding them. I can’t confirm this is the case with your data until I discern which dataset/model from the repo you’re talking about, but in the meantime you could check out this demo of what I mean by redundant computation and how to eliminate it. Arguably this might induce an unfair advantage when then comparing against a non-Stan sampler, so you’d probably want to implement the same tricks in whatever language those use for expressing the model too.