How many iterations to run and how many digits to report (notebook)

General
1

I’ve written
“How many iterations to run and how many digits to report” notebook.

I gave last October a talk on this topic, but it seems the notebook was not public until now.

Summary of workflow for how many digits to report

  1. Run inference with some default number of iterations
  2. Check convergence diagnostics for all parameters
  3. Check that ESS is big enough for reliable convergence
    diagnostics for all quantities of interest
  4. Look at the posterior for quantities of interest and decide how
    many significant digits is reasonable taking into account the
    posterior uncertainty (using SD, MAD, or tail quantiles)
  5. Check that MCSE is small enough for the desired accuracy of
    reporting the posterior summaries for the quantities of
    interest.
  • If the accuracy is not sufficient, report less digits or run
    more iterations.
  • Halving MCSE requires quadrupling the number of iterations
    (if CLT holds).
  • Different quantities of interest have different MCSE and may
    require different number of iterations for the desired accuracy.
  • Some quantities of interest may have posterior distribution with
    infinite variance, and then the ESS and MCSE are not defined for
    the expectation. In such cases use, for example, median instead
    of mean and mean absolute deviation (MAD) instead of standard
    deviation. ESS and MCSE for (non-extreme) quantiles can be
    derived from the (non-extreme) cumulative probabilities that
    always have finite mean and variance.
9 Likes