Hey all,
I wrote some code to generate some PDFs that are nice for visually comparing things; first scaling the number of bins linearly with n_eff and then with the square root of n_eff. All graphs are with 1e5 N_samp, but on each page I vary another parameter; first I show Dan’s RNG from above:
pnorm(rnorm(N_samp, sd=1 + runif(N_samp, -1/sqrt(n_eff), 1/sqrt(n_eff))))
Then on the next page with the same parameters and a discretized version:
round(dans.rng() * planck_length) / planck_length
Both of those with 26 n_eff and then the latter with planck_length of 1000. You can see the spike at 1 disappear with this discretization. The next 3 pages are with fixed n_eff and increasing planck_length; here I mostly noticed that increasing the planck_length gradually brought back the spike at 1 (and shortened the spike at 0) and that the histogram bin sizes vary with n_eff in the same way that they do in the other charts.
The next 3 pages vary n_eff; my synthesis here would be that sqrt(n_eff)*2 seems pretty much always safe (even if with some n_effs we could go higher).
w.r.t. squiggle + circle (KS test visualization) vs. plateau (uniform histogram) - As a novice I find the latter much easier to look for; I still don’t really have a good idea from the previous pages of tests how much of the line is fine to have outside the circle. On the other hand, the KS test number does seem pretty useful, especially for moving towards a more automated quantitative statistic. And I do think with all of my examples, the KS test had pretty low p-values for pretty off models (though maybe that deserves further study).
Excited to hear what you all think! Being able to reproduce this without using Stan at all has been immensely satisfying, even if I still haven’t figured out why the spike at 1 disappears and the spike at 0 gets larger with discretization.
code: https://gist.github.com/seantalts/c4a1de131f5dfc8f5b71c5c54b272617
<edit: Forgot the PDFs!>
linear_n_eff.pdf (501.2 KB)
sqrt_n_eff.pdf (50.5 KB)