Split-Rhat diagnostic and relative effective sample size


#1

New split-Rhat convergence diagnostic and relative effective sample size estimates using rank normalized values, quantiles, and posterior intervals. I learned new things about MCMC when making this
https://avehtari.github.io/rhat_neff/rhat_neff.html

This has been waiting for some time as we are working also on a corresponding paper, but wanted now to get feedback for the different proposed approaches and warning levels. You can comment here or come discuss at StanCon next week.


Analysis API
#2

The relative efficiency plots of the different quantiles are super cool


#3

Thanks for sharing this great work. I really like the ideas surrounding the error estimates for quantiles.

While trying to better understand what’s going on, I stumbled across a possible quirk in the functions monitor and monitor_extra of the updated code and thought I should pass along the info – I hope I’m not overstepping my bounds.

The first argument to both functions is sims. If sims was created by running only one chain, then the check for two dimensions won’t catch as sims has 3 dimensions, with the chains position equal to 1. Following this, the real issue comes up in the main for-loop of monitor and monitor_extra.

The line sims_i <- sims[, , i] will return a vector instead of a matrix since the chains dimension is 1. This means that the code prod(dim(sims_i)) returns 1 (since dim(sims_i) is NULL) instead of the intended iter * chains.

I understand that this code is not really intended for 1 chain runs, but thought that I should share this anyway.


#4

Link is now broken. Correct link https://avehtari.github.io/rhat_reff/rhat_reff.html