Writing a Program to Calculate ESS from Samples

General
1

Hey all,

I’m attempting to write a program to calculate ESS from the samples I get back from Stan because I run the program from the command line on the cluster, without R or python as an interface to generate a StanFit object.

I just want to make sure I understand the formula on 16.4 Effective sample size | Stan Reference Manual. I already know how to calculate W and var+_hat because I made an R_hat program using the Stan Reference Manual, but I’m still a little lost trying to figure out the rest of the term definitions for the ESS formula found in the Stan Reference Manual. Starting here…

phat

Does the circled area refer to the formula:

Screenshot 2023-12-15 at 4.47.23 PM

Where the covariance term is the normalized:

Screenshot 2023-12-15 at 4.50.43 PM?

And, with respect to the rest of the formula…

var

What does this circled area refer to the variance of? Is it the variance of samples for a particular parameter for a single chain?

Thanks!

Sincerely,
Corey

2

The circled \widehat{\rho}_{t, m} is the estimated lag-t correlation in chain m, whereas \rho_t is just the general definition of lag-t correlation for a Markov chain X. The expression \textrm{cov}_{x,y} is then an estimate given an observation of a subsequence of X. The x is an observed subsequence of X and the y is an observed sequence lagged by t.

s^2_m is defined where \widehat{R} is defined, back in section 16.3.1. It’s the estimated variance in chain m of the parameter whose ESS is being estimated in chain.

The easiest thing to do might be to read the Bayes-Kit code, which has a simpler version of the definitions that only work on a single chain at a time (and hence don’t adjust for lack of agreement among multiple chains). But it has the definitions of these key terms in a pretty simple form:

That does everything according to these definitions. On the other hand, we’ve updated posterior (R) and arviz (Python) to use newer definitions from Vehtari et al., and you can look at their code for more details on that.

2 Likes
3

if you run Stan from the command line using the CmdStan interface, then you can use the accompanying CmdStan utilities 20 stansummary: MCMC Output Analysis | CmdStan User’s Guide and 21 diagnose: Diagnosing Biased Hamiltonian Monte Carlo Inferences | CmdStan User’s Guide to get ESS and more.

you can also write your own utility, in which case, I suggest using the above utilities to check your work.

1 Like
5

Thank you! That is a very helpful utility to have handy. I will likely still write a utility for it, but this is very helpful to have on hand right now