Spatial model selection with loo

[manjago]

Hi Carlos. Yes, 130 (27.7%) of the observations has values greater than 0.7. I can’t identify a pattern on them. About the priors, yes, I’m using the default brms priors. In the case of the spatial component, I think is ok. For the beta priors, I have no reason to doubt about them either, but the priors can’t be discarted as cause of the problem, I agree with that.

Additional information. As mentioned, n = 470. The p-loo of my base model is 375.3 and the number p of parameters is 1902. The latter was obtained by:

rstan::get_num_upars(your_brms_model$fit)

This was validated by Paul Buerkner here:

Determine number of parameters in brms GAMM to compare to p_loo value

I agree that this could be a reasonable choice indeed.

Based on this, I’m here:

“If p_loo < the number of parameters p and the number of parameters p is relatively large compared to the number of observations p>n/5, it is likely that model is so flexible or population prior is so weak that it’s difficult to predict for left out observation even if the model is true one.”

As discussed above, the prior can be the cause but I don’t know what I’m looking for in order to modified them.

I also made other recommended checks, the posterior predictive check (in this case for each condition) and the pit-loo check. They seem ok, but at least for the posterior predictive check, this does not mean that there is not a problem with the model. Other tests as k-fold cv are recommended, but it seems that this not possible for spatial models:

A quick note what I infer from p_loo and Pareto k values

Only as a partial evidence, as it’s possible that 1) model is overfitting, 2) posterior predictive check is misleading, e.g. 2a) if kernel density estimate is used it can hide problems, or 2b) looking at the marginal doesn’t reveal problems with conditional distributions. It depends. If you have a simpler equivalent model maybe use that instead (e.g. models where the latent values can be integrated out analytically). If you know you model is likely to be the best one then do enough other chec…

This post is actually a spatial model example, but the same solution is recommended:

How to evaluate a Poisson/NB CAR/IAR model

Your model is factorizable, but the prior structure problem is same as in section “Exact LOO-CV with re-fitting” in Leave-one-out cross-validation for non-factorizable models vignette. “The solution is to model y_i as a missing observation and estimate it along with all of the other model parameters.” Instead of exact LOO-CV, you are trying to do exact k-fold-CV, so you can use the same approach.

Maybe I’m completely wrong, but in a model like this, where my response variable is an array of conditions counts, and I only have one per geographical unit, is natural to get this kind of problems:

Practical implications of many high-Pareto K observations loo

Without seeing additional diagnostics (like LOO-PIT) it’s difficult to be certain, but this is the likely reason. You could compare which observations get high Pareto-k and if they all are observations that are the only observation for some species, then that would give additional support to this hypothesis. If the high Pareto-k’s are only for the those observations that are the only one in some group, and we assume there is no model misspecification, then the posterior is still valid. Yes, t…

Thanks again Carlos. If you or other members of the community have an idea, the help will be really appreciated.