Mixis and E_loo

General
21

thanks Aki

that is what I hoped - very glad to hear it.

thanks

Greg

22

hello Luca

yes, the reference is re PSIS estimates, i.e. elpd_psis-loo. I would also like to think that the improvement with MixIS makes this comparison more reliable, or at least plausible. The original reason I tried MixIS at Aki’s suggestion was the comment somewhere on the loo online pages that loo_compare relies on the absence of high pareto-k values. Given the outliers (not many) in my data, that was an obstacle.

anyway both of you are saying it is ok to approach the comparison in the same way, using the MixIS pointwise values for elpd.

thanks

Greg

23

Hello Greg,

Yes from a theoretical perspective, as Aki suggested, the comparison can be done irrespective of the elpd estimation method, and its more that logical to assume that more reliable elpd estimates lead to more reliable model comparison.

I just tried to stress that which elpd estimate is better depends on a couple of factors, so always trying to warn against thinking that a method can work in all situations :).

Best,

Luca

24

hello Luca

I take your point that in a particular situation PSIS may be better. However, the loo FAQ says “…if in the final stages we are comparing models that have similar performance, there are some high k^ values, and we want to be minimize probability of wrong conclusions, it’s best to fix the problems. There is no threshold for how many high k^ values would be acceptable.” With my actual data and model, I could find no way to fix the small number of high k^ values (moment match failed, an issue explored in an earlier post to this forum) and although it is conceivable that a longer run would resolve this, it didn’t seem to and the run time is already long, so MixIS was a better choice even though I can’t be sure that it is really better! The only real alternative I could see would be to delete the outliers and run the comparison with the rest of the data. But I am particularly interested in the outliers!

thanks again to you and Aki for comments, and for MixIS!

Greg

25

Hello Greg,

I do think your situation is where MixIS can shine, so did not want to give any push back on your methodology. Moreover, being its’ first authour, I for sure have all interest in MixIS being as wide spread as possible!

Just wanted to give as much as possible a complete and broad answer, that could work also for other situations. Thanks again for sharing this use case,

Best,

Luca