Understanding `loo` results for non-standard `Contaminated Controls` model

[jsocolar]

I don’t have time to really understand the model in depth right now :(

But I think I can help with a few of these questions:

@RobertoCerina

It’s hard for me to disentangle the misspecification / flexibility in this particular case - is my model miss-specified or too flexible ?

So the question here is how many parameters are in your model and how many data points you are fitting. By comparing these two numbers to p_loo and to each other, you should be able to figure out which regime you are in. If you’re using rstan, then rstan::get_num_upars will help you find how many parameters are in the model.

@RobertoCerina

And moreover - is loo even an appropriate metric to compare model specification and accuracy in this contaminated controls example ?
One can envision a model which is “wrong” by design - made to fit the actual DGP as opposed to this synthetic DGP, which we would prefer but would perform poorly on fit metrics.
Indeed I think much of the structure added to the model - like offset and contamination layer - is likely decreasing in-sample fit by at least increasing the number of parameters, and also actively ignoring in-sample observations (the contamination part of the model effectively pretends 0-s are 1s, for example).

I don’t understand exactly what the model is doing, but in general I would suggest trying to write down a model that captures the true data-generating process as nearly as possible. I think that the contamination layer is part of the assumed true data-generating process that generates the data that the model sees. If it’s not, then I don’t understand the motivation for including it.
Note that loo prioritizes good predictive performance, which is not the same thing as capturing the true data-generating process. With finite data and a complicated DGP, the true DGP might be impossible to fit (i.e. attempting to fit the true DGP might be prone to non-identification and very sensitive to the priors), and might yield really bad predictive performance.
Good predictive performance is not the same thing as accurate uncertainty quantification around causal effect sizes of interest. If your goal is inference about effect sizes and you believe a priori that some particular feature of the model, or some nuisance covariate, is important, then you might choose to leave that model feature in your final model even if loo says that doing so worsens predictive performance.

@RobertoCerina

My final question is about the Min. n_eff: is it at all surprising that we have large n_eff in the “very bad” category ? I would have expected that number to be smaller than for the good and ok categories.

Perhaps @avehtari will chime in about whether this is surprising or not, but I don’t know of any reason to be surprised. For what it’s worth, the loo package documentation says:

the PSIS effective sample size estimate will be over-optimistic when the estimate of 𝑘k is greater than 0.7 .