Split-Rhat diagnostic and relative effective sample size

Algorithms
1

New split-Rhat convergence diagnostic and relative effective sample size estimates using rank normalized values, quantiles, and posterior intervals. I learned new things about MCMC when making this [edit: updated link]:

https://avehtari.github.io/rhat_ess/rhat_ess.html

This has been waiting for some time as we are working also on a corresponding paper, but wanted now to get feedback for the different proposed approaches and warning levels. You can comment here or come discuss at StanCon next week.

6 Likes
2

The relative efficiency plots of the different quantiles are super cool

1 Like
3

Thanks for sharing this great work. I really like the ideas surrounding the error estimates for quantiles.

While trying to better understand what’s going on, I stumbled across a possible quirk in the functions monitor and monitor_extra of the updated code and thought I should pass along the info – I hope I’m not overstepping my bounds.

The first argument to both functions is sims. If sims was created by running only one chain, then the check for two dimensions won’t catch as sims has 3 dimensions, with the chains position equal to 1. Following this, the real issue comes up in the main for-loop of monitor and monitor_extra.

The line sims_i <- sims[, , i] will return a vector instead of a matrix since the chains dimension is 1. This means that the code prod(dim(sims_i)) returns 1 (since dim(sims_i) is NULL) instead of the intended iter * chains.

I understand that this code is not really intended for 1 chain runs, but thought that I should share this anyway.

5

I guess the link is broken again.

6

The correct link is
https://avehtari.github.io/rhat_ess/rhat_ess.html

I’m sorry I’ve changed the filename two times. Unfortunately I’m not allowed to edit that old post, and thus trhe wrong link stays there.

1 Like
7

Thank you.

8

I’ve deleted my post so as to reduce the risk of confusion.

9

I edited it the top post to include the new link. We could make you an editor, but I think Discourse limits the number of people we can give permissions to.

10

Here’s the link to the arxiv paper

@avehtari, I think equation 14 of the arxiv paper needs the same change as discussed

which should read

\rho_t = \frac{1}{\sigma^2} (\int_{\Theta} \theta^{(n)} \theta^{(n+t)} p(\theta) d\theta - \mu^2)

1 Like