# Bernoulli model with many LOO-PIT values at 1

**URL:** <https://discourse.mc-stan.org/t/bernoulli-model-with-many-loo-pit-values-at-1/19245>\
**Category:** Modeling\
**Tags:** loo\
**Created:** [November 13, 2020, 1:24am UTC](https://discourse.mc-stan.org/t/bernoulli-model-with-many-loo-pit-values-at-1/19245 "2020-11-13T01:24:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![adkinsty](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/adkinsty/32/9845_2.png) [@adkinsty](https://discourse.mc-stan.org/u/adkinsty)\
**Post date:** [November 13, 2020, 1:24am UTC](https://discourse.mc-stan.org/t/bernoulli-model-with-many-loo-pit-values-at-1/19245/1 "2020-11-13T01:24:15Z")

</div>

I implemented a RL model in Stan. The model uses a TD-learning mechanism to compute a choice probability `theta` for each trial. Observations (choices) are in turn sampled with `y ~ bernoulli(theta)`.

MCMC diagnostics look good and parameter estimates look reasonable. However, when I check the model using `loo` I observe some serious miscalibration:

 ![image](https://canada1.discourse-cdn.com/flex030/uploads/mc_stan/original/2X/6/673cfe2dffa2b06286b61c8b6eec6cb927bf147c.png)

My understanding is that the above indicates that the model’s predictive density is over-dispersed compared to the data. And when I look at the QQ plot, I see this:

 ![image](https://canada1.discourse-cdn.com/flex030/uploads/mc_stan/original/2X/e/eddc729bbb9b8e5d322f09c808ed73359f5acc99.png)

There are many LOO-PIT values of 1. I am not sure what to make of this. Has anyone encountered an anomaly like this before? Does anyone have any ideas about what could cause this?

Thank you for your time…

---

<div class="post-metadata">

**Author:** ![adkinsty](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/adkinsty/32/9845_2.png) [@adkinsty](https://discourse.mc-stan.org/u/adkinsty)\
**Post date:** [November 13, 2020, 1:42am UTC](https://discourse.mc-stan.org/t/bernoulli-model-with-many-loo-pit-values-at-1/19245/2 "2020-11-13T01:42:15Z")

</div>

However, a comment from @avehtari on a related [post](https://discourse.mc-stan.org/t/worst-loo-pit-plots-ever-pp-checks-great/18789) suggests that these plots are not valid for discrete models. So, perhaps I should ignore this.

---

<div class="post-metadata">

**Author:** ![avehtari](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/avehtari/32/5935_2.png) [@avehtari](https://discourse.mc-stan.org/u/avehtari)\
**Post date:** [November 13, 2020, 7:21am UTC](https://discourse.mc-stan.org/t/bernoulli-model-with-many-loo-pit-values-at-1/19245/3 "2020-11-13T07:21:52Z")

</div>

Yes, the current implementation of LOO-PIT is not correct for discrete and we’re working on a correct implementation. In addition even the discrete LOO-PIT is not very good for binary target. See instead Section 4.4 in [https://avehtari.github.io/modelselection/diabetes.html](https://avehtari.github.io/modelselection/diabetes.html)  
for how to evaluate calibration for Bernoulli model.