I have some data I’m modeling as y ~ ordered_logistic(...), and have y_rep ~ ordered_logistic_rng(...) in my generated quantities section so I can run posterior predictive checks, but I’m unsure as to what quantities to compute from y & y_rep that would make for sensible PPCs. Any suggestions?
I’ve had some success with P(Y > c) for a sensible c thad divides the ordinal response roughly in the middle for most cases.
Here is a custom plot I made and found useful, it combines some elements of PPCheck with exploring the implications of the model. For each data point I make predictions by varying a single variable across the range of its values:
The veritcal bars show posterior credible intervals. Each line connects all predictions from the same sample and the same data point (a random subset shown), green circles are the observed data (size indicates number of cases). Line blue when the probability increases and red when decreases.
Would ppc_bars() or ppc_bars_grouped() in bayesplot be helpful?
I think I created something functionally similar showing violin plots of the counts of responses in each 4 responses X 23 questions in my data, with dots showing the observed data counts:
I’m a little concerned that the dots fall in the 50% intervals in 90 of the 92 distributions, which as I understand it usually means the model is over-fitting somehow. But maybe since these counts are a transform of both y & y_rep, that’s ok?