Is there an easy way to extract an appropriate number for the parameters used in a brms fit?
I see. Then is there a function or a procedure that I can follow by myself to get the number of parameters that can be used to compare with p_loo?
Thank you so much for this detailed answer and all the references! I am not very familiar with the concept of Pareto smooth sampling and related concepts that are involved in
loo() but I will take a look at the materials you suggested and follow your suggestions to check my models. Thanks a lot!
Two quick questions regarding the interpretation of high Pareto k estimates:
- If posterior predictive check looks good, then can this be used as an evidence to say that the model is not misspecified?
- High Pareto k could be due to model is so flexible or the population prior so weak. If that is the case, is this something to worry about or it is good to leave the model as it is and use it?
Only as a partial evidence, as it’s possible that 1) model is overfitting, 2) posterior predictive check is misleading, e.g. 2a) if kernel density estimate is used it can hide problems, or 2b) looking at the marginal doesn’t reveal problems with conditional distributions.
It depends. If you have a simpler equivalent model maybe use that instead (e.g. models where the latent values can be integrated out analytically). If you know you model is likely to be the best one then do enough other checks or do K-fold-CV etc.
Got it. Thank you for your answer!