Monte Carlo error for Pathfinder fit

Algorithms
1

When fitting a model using the Pathfinder algorithm (via cmdstanr, for example), is there a recommended way to judge the Monte Carlo error (i.e., to estimate the MCSE for a specific parameter, for example)? I’ve tried to find information about this in the paper by Zhang et al. (2022, Pathfinder: Parallel quasi-Newton variational inference), but the only related statements I have found are the following:

The accuracy and reliability of the Monte Carlo estimates can be diagnosed without regard to a reference distribution using a diagnostic based on the Pareto k statistic (Vehtari et al., 2019; Dhaka et al., 2021).

and

[…] and reduces the scale of Monte Carlo error by around (1 − \sqrt{5}/\sqrt{30}), or 60%.

but in the latter, it seems that the term “Monte Carlo error” refers to the (scaled) 1-Wasserstein distance.

1 Like
2

If you use cmdstan e.g. via cmdstanr, the default is to do PSIS-resampling in Stan, and if Pareto-k is small (say <0.5) then you can use the basic MCSE equation. If you want more accurate MCSE, use option psis_resample=FALSE and then get the weights with something like

pdraws <- pathfindr_fit$draws()
pdraws <- pdraws |>
  mutate_variables(lw=lp__-lp_approx__,
                   w=exp(lw-max(lw)),
                   ws=pareto_smooth(w, tail='right'))

Then ws are the Pareto-smoothed importance sampling weights and you can compute MCSE with Equation (5) in Pareto Smoothed Importance Sampling paper.

Sorry for very concise answer, I’m in a hurry, but ask more if this is not clear yet

3 Likes
3

Thanks, that helped a lot!