loo(moment_match=TRUE) doesn't seem to do anything

Modeling
1

I have fitted a model in RStan and performed loo-cv with the loo package by following this vignette Writing Stan programs for use with the loo package • loo
However some observations have pareto k>0.7. I then try to redo loo with moment_match = TRUE but nothing appears to happen - I get back exactly the same elpd and pareto k values within about 1 second of running the command. I’m also confused because the vignette about moment matching Avoiding model refits in leave-one-out cross-validation with moment matching seems to have different arguments in the loo function to me: loo2 <- loo(fit, moment_match = TRUE) where fit <- sampling(stanmodel, data = standata, seed = seed, refresh = 0). Does this mean I have to use this format for it to work? My code is:

require(rstan)
require(loo)
fit <- stan(file = 'mymodel.stan', 
                   data = mydata, 
                   iter = 2000, chains = 4)

loglik <- extract_log_lik(fit,c('log_lik'),merge_chains=FALSE)
loo_original <- loo(loglik, r_eff=relative_eff(exp(loglik)),cores=4,save_psis = TRUE)
loo_moment_match <- loo(loglik, r_eff=relative_eff(exp(loglik)),cores=1,moment_match = TRUE,overwrite=TRUE)

I have also tried without cores=1, overwrite=TRUE.

2

There seem to be a number of issues related to moment_match. On my system it usually fails: either giving an explicit error message or, as in your case, basically appearing to run but actually doing nothing. Until moment_match is fixed, I suggest using reloo = TRUE instead (to remove the high Pareto-k values), which is slower but definitely works.

1 Like
3

Thanks. reloo = TRUE also doesn’t seem to do anything for me. Is this argument for the loo function in the loo package? The only reference I seem to find for it online appears to be people using it with their brms models so I am wondering if it doesn’t work for my model fitted with stan.

4

You’re correct – reloo is used in brms and there are arguments in rstanarm to do the same thing. Unfortunately, I don’t know if there is a way to quickly automate this in a model fitted directly in Stan.

1 Like
5

Hi, do you have reproducible examples you could post here or in a new issue at github?

At the moment we need more information what to fix. One likely possibility what is breaking is parallelization in certain operating systems, which could explain why we haven’t observed the errors you have observed. Please report also the OS you are using. If it really is an parallelization issue in certain OS, then it probably would need a fix in the parallelization package used.

I hope we can get moment_match to work for you.

6

Thanks Aki! I will aim to get some repros to the LOO GitHub soon – I will need to simulate unsharable data and the models take a while to fit, which is the reason I can’t do this immediately.

I usually work in macOS 11.5.2 but I have attempted moment_match on a Windows 7 laptop with no success there either.

1 Like