Bernoulli model with many LOO-PIT values at 1

Modeling
1

I implemented a RL model in Stan. The model uses a TD-learning mechanism to compute a choice probability theta for each trial. Observations (choices) are in turn sampled with y ~ bernoulli(theta).

MCMC diagnostics look good and parameter estimates look reasonable. However, when I check the model using loo I observe some serious miscalibration:

image
image918×826 64.5 KB

My understanding is that the above indicates that the model’s predictive density is over-dispersed compared to the data. And when I look at the QQ plot, I see this:

image
image918×826 49.9 KB

There are many LOO-PIT values of 1. I am not sure what to make of this. Has anyone encountered an anomaly like this before? Does anyone have any ideas about what could cause this?

Thank you for your time…

2

However, a comment from @avehtari on a related post suggests that these plots are not valid for discrete models. So, perhaps I should ignore this.

3

Yes, the current implementation of LOO-PIT is not correct for discrete and we’re working on a correct implementation. In addition even the discrete LOO-PIT is not very good for binary target. See instead Section 4.4 in https://avehtari.github.io/modelselection/diabetes.html
for how to evaluate calibration for Bernoulli model.

1 Like