Posterior Predictive Checks vs. LOO

I have looked at a 6 survey data sets that use a slider scale, or in some cases what is called a feeling thermometer. Such data are range restricted (between 0 and 1), and often show spikes at precisely 0 and 1 (0 and 1 inflation).

Logically it made sense to me to fit a 0, 1 inflated beta. I basically just fit the distributional family with no covariates. Posterior predictive checking showed that the model reproduces the frequency distribution of the data pretty nicely. However, in 5 out of 6 data sets, the Gaussian family has better ELPD LOO values, beyond 2 SEs better.

Visually it doesn’t seem to be a contest with the 0, 1 inflated beta nailing the shape of the frequency distribution and the Gaussian being, well Gaussian, not Beta. It is too symmetric, the slopes of the tails are wrong, it bursts through the 0, 1 boundary of the data, and so forth.

What is going on? Too many parameters? It is 0,1 Beta with 4 parameters, the Gaussian has 2. In at least one of the data sets the sample size is over 8,000. I understand that a simpler, wrong model can sometimes outpredict the correct model, but this seems a bit strange.

0-1-inflated beta is a mixture of discrete and continuous distributions and it is not directly comparable with continuous distribution as you can’t compare probabilities and densities directly. To make a fair comparison, you need to discretize both when computing LOO. See an illustration of such discretization in Nabiximols treatment efficiency case study. I don’t have yet a case study specifically for 0-1-inflated-beta, but hopefully the existing case study is clear enough and don’t hesitate to ask more

2 Likes