I am trying to develop some rough guidelines around how many samples after convergence are required for analyses at my work. Does anyone here have good references on how I can use the new mcse outputs from monitor() to help with this? We typically will report things like odds ratios out to a maximum of three decimal places, so for the tails would it make sense to suggest MCSE <= 0.0002? My thinking is that this would put 2sds at 0.0004 which would give me a relatively stable third decimal place? If I am way off base here it might just be easiest to point me towards a useful reference if you’re aware of one. The BUGS book suggests a general 0.5% of posterior SD rule but that doesn’t seem fit for purpose here.
param Q5 Q50 Q95 Mean SD MCSE_Q5 MCSE_Q50 MCSE_Q95 MCSE_Mean MCSE_SD Rhat Bulk_ESS Tail_ESS
OR[1,2] 0.7635 0.8073 0.8540 0.8078 0.0276 0.0003 0.0003 0.0004 0.0002 0.0002 1.0000 13267 23131
1 Like
You’ve got the logic down right (MCSE’s aren’t mysterious). This is the writeup for the new diagnostics: https://arxiv.org/pdf/1903.08008.pdf
The 0.5% rule is fine IFF you’re doing things like estimating the mean effect of daily temperature on ice melting speed. Nobody’s going to argue much that it exists, the only questions is how big and getting the SD roughly right is much more important than the edges. When you’re going to put thresholds on parameters and make decisions based on those thresholds like people in medical fields like to do, then it’s important to get the right digits in there.
Whether it’s worth digging more depends on how important these decimal places are to you. They could be off if some other model parameter is not-quite-mixing but it’s really rare that I see people examining that in detail. I usually try to understand if the well-behaved parameters are related to the questionable ones and fit an alternative model that might highlight the issue (e.g.-now mixing is good for all parameters did those Q95/MCSE95 values budge?).
That seems too much. I know it’s common, but it’s also as common complain that there are unnecessary many decimals reported. I don’t think there is any practical benefit to report more than two significant digits for odds ratios.
Totally agree but that is based on clinical obsession with intervals excluding unity and it will take some time to nudge people away from it. I’ve been asked for intervals down to 4 decimals and that I pushed back on, but it’s fighting against the inertia of a whole industry.
1 Like
The precision necessary of the MCMC estimators is determined by the comparisons you want to make between expectation values. If you want to compare differences of odds ratios to the third decimal place than your logic is sound. If you need only the first or second decimal place then you can get by with many fewer effective samples. If you know that the necessary precision is pretty small but people want many decimal places reported then you can go with fewer effective samples and report something like 0.583 +/- 0.022 – it will upset some people but it many be the path of least resistance given the inertia in the field.