N_eff BDA3 vs. Stan

Pull request has been merged.

Do RStan, ShinyStan and PyStan use also computation in chains.hpp?
Ping @bgoodri @jonah @ariddell

1 Like

It looks like RStan doesn’t: rstan/rstan/src/chains.cpp. I didn’t realize
it wasn’t using Stan’s structure. I think we should consolidate code if we
can.

Thanks for your work on chains.hpp, the changes look great.

Getting the interfaces to use the same code is or should be a top priority.

I tried at one point to have PyStan use chains.hpp but it turned out to
be really slow.

The main difficulty is exposing some sort function in the services
namespace which is easy and computationally convenient for the
interfaces to call. The function signature should only involve std data
structures, no Eigen data structures. You can, however, assume that any
interface interested in calling the chains functions will be able to
arrange the samples into contiguous memory. (The RStan people can
correct me if I’m wrong about this.)

Should I make pull requests for rstan/rstan/src/chains.cpp and pystan/pystan/_chains.pyx ?

Yeah, although it may require more hacking to get it to work.

Why there is effective_sample_size and effective_sample_size2 ? I can read the comment but I don’t understan it. Is effective_sample_size2 really used?

Hey @avehtari,

I am trying to implement this computation in PyMC3.

I have a question for you. In your tests, what exactly is dimensions? Is it the length of the chains or number of chains or something else?

Thanks in advance!
Sharan

I don’t know what dimensions you are referring. There is not such variable in the parts of the code I modified.

The more hacking seems to refer to the fact that it’s really difficult to build rstan. There are not much instructions and even @bgoodri keeps guessing what I should do to get it built…

PyStan build was easier (only couple hours to figure how to install and run in place), but I haven’t been able to figure out how to run tests, or compare results with CmdStan (same random seeds doesn’t seem to provide same answer). I’m novice in Python, so I’m now stuck, and would appreciate help how to load values of chains from csv-files (one csv-file per chain) and run ess function for those chains. @ariddell can you help?

If everything you need upstream has been merged into develop, try

source("https://raw.githubusercontent.com/stan-dev/rstan/develop/install_StanHeaders.R", echo = TRUE)

and then installing the modified rstan.

Where does this install those headers?

.libPaths()[1]

Now CmdStan, RStan and PyStan N_eff computations all use Geyer’s initial monotone sequence and results match at least with 6 digit accuracy when using exactly same chains (blocker.1.csv and blocker.2.csv).

Based on the experience it would be great if this kind of computational code would be shared…

Hi, cmdStan user on Matlab platform here. Sorry to get back to this after all this time. I’m trying to replicate the N_Eff and R_hat calculations of the stansummary with my own code. There are several practical reasons for wanting to do this, e.g. to easily display these stats on figures. R_hat is pretty straight forward. For N_Eff, it is difficult to exactly match the value from stansummary without knowing exactly how the truncation works. I know that an exact match is not necessarily all that important for inference, but for publication purposes, this kind of stuff matters. Is the code, or the formulas, to calculate the stats available somewhere? Perhaps I’m missing something, but did not have much luck with the user guide. I read through the above discussion but it is not clear to me what you all settled on. Thanks in advance.

Equations are in the paper Rank-Normalization, Folding, and Localization: An Improved Rˆ for Assessing Convergence of MCMC (with Discussion) (stansummary doesn’t use the rank normalization, but otherwise the details are the same).

You can also look at the code for the exact implementation

and

Or see R version ess_basic() function in posterior/convergence.R at master · stan-dev/posterior · GitHub

1 Like

Thank you.

Also, you can look at the following Matlab implementation by Beth Baribault : matstanlib/ess.m at master · baribault/matstanlib · GitHub

1 Like

I looked into this code and ran into problems:

You have to manually alter the method from within the code. The version on Git is set to ‘vehtari’ and it throws an error when ran that way. I tried with ‘BDA3’ and ‘BDA2’ methods and got back wildly different results. BDA3 returned a very small number= 7.2e-4. BDA2 returned an unexpectedly high one= 8.4e3.

The data has 2000 iterations on 4 chains. When I use stansummary, it calculates n_eff to be 4.1e3. My own code, in which I use the first four positive lags (after that autocorrelations look monotonic and alternate between positive and negative), I also get 4.1e3, although not an exact match to stansummary.

In short, I don’t understand the results from this Matlab code.

Maybe run against R/Python implementions side-by-side and compare results for each step

I want to do that, but do not have them installed on my computer right now and I’m not fully literate on either platform. For now, I’m sticking to comparison with cmdStan stansummary. Thanks