MCSE of elpd_loo is NA! How to proceed?

[arumadri]

Hi there,

I’m trying to fit a model including crossbasis matrices generated using Distributed Lag Non-linear Models (dlnm) as predictors using brms (this has been solved/fitted using INLA elsewhere here). My goal is to see if I can replicate this using brms/Stan instead of INLA.

Before defining models including random effects, I have first focused on one area and used only fixed effects variables to get a sense of how models including the crossbasis matrices are fitted using brms.

My basemodel includes 4 parameters; month and 3 crossbasis matrices (basis_tmin, basis_tmax and basis_pdsi– for precipitation). I know these basis_tmin and basis_tmax are correlated but I wanted to include both in the base model for purposes of leave-one-out (loo)cross-validation checks (perhaps a bit naive). See the model specification here

The model prior and posterior predictive checks have both performed well and I moved on to check the leave-one-out (loo)cross-validation checks. After reading about the interpretation of loo outputs on here and here, I see that the model that includes basis_tmin performs better than that with basis_tmax which is the same finding from the main publication.

The problem is that for all the loo outputs, I get MCSE of elpd_ood is NA! And to quote @avehtari from the posts above, “If Monte Carlo SE of elpd_loo is NA, then the result is very unreliable.”

My question therefore is: Is this a problem of my model specification or this is due to the inclusion of predictors of the form crossbasiss matrices?

Here is my base mode and prior set up

How can I proceed?

[avehtari]

MCSE for elpd_loo is NA, as some of the LOO-folds have Pareto-k>0.7 as shown in the Pareto-k diagnostic value tables and mentioned in the warning message. LOO package glossary — loo-glossary • loo tells more about interpretation of Pareto-k values. See also CV-FAQ.

As you are using brms, I recommend testing whether you get rid of high Pareto-k values by using moment matching, as also advised by the warning message. To avoid the need to re-run moment matching, it is good to store the loo results within the brms fit object as

run_basemodel7 <- add_criterion(run_basemodel7, criterion="loo",
                                save_psis=TRUE, moment_match=TRUE)
run_basemodel8 <- add_criterion(run_basemodel8, criterion="loo",
                                save_psis=TRUE, moment_match=TRUE)
```run_basemodel7
Then you can check the loo outputs as before

loo(run_basemodel7, run_basemodel8)

[arumadri]

Thank you for your reply.

Adding moment_match=TRUE eliminates the high Pareto-k values from the previous model comparison and MCSE of elpd_loo is 0.1.

Very useful links to read more about this issue. Thanks again!