Loo moment matching not working

General
, ,
1

I have a stanfit object for which I have some pareto k values that are a bit too high, so I’m trying to get a LOOIC estimate with moment matching. However, when I try to use the loo() function directly on the object with moment_match=T, I get the following error message:

Error in .local(object, ...) : 
  the model object is not created or not valid
Calls: loo ... apply -> FUN -> <Anonymous> -> <Anonymous> -> .local
Execution halted

Any help would be appreciated!

2

Hi Sergej, welcome to the Stan forums. A few questions that will hopefully help us get to the bottom of this:

  • Does regular loo without moment matching work for the same stanfit object? I assume so since you mentioned high pareto k values, but just double checking.
  • Is this a stanfit object loaded in from a previous R session or did you fit the model in the same R session as running loo with moment matchinng? I’m not sure if this is documented (if not then we should definitely mention it) but I think that loo with moment matching requires calling RStan’s unconstrain_pars method internally but that doesn’t work in a fresh R session.

@topipa Any other ideas?

3

Hi @jonah

Thank you so much for the quick response and for welcoming me to the community! My answers are below:

  • Yup, it works just fine and again I do it directly on the stanfit object (instead of extracting the log_lik, then finding r_eff, then doing loo() in the end).
  • It actually is from a previous R session, and also the stanfit object was created using an older version of rstan (I believe 2.18, and I’m trying to extract the loo with moment_match using the 2.21 version). Not sure if that is important as calling loo() withouth moment_match was fine.

Are you saying I need to rerun the model and then get loo with moment matching or something else entirely? I think I can get around Stan pretty well but I’m still a newbie so any detail is helpful!

4

loo() without moment matching should still work because it doesn’t rely on any functionality in rstan that only works in the same R session.

Unfortunately yeah I’m pretty sure you will need to refit the model in order for loo with moment matching to work (hopefully it’s not a super slow one!). Sorry about that!

I’ll make sure that we update the documentation to mention this.

5

@jonah It is a very slow (Wiener DDM) model so just to confirm before I do that: do I literally just refit the model and then call loo(stanfit, moment_match=T) right underneath it? Is that it or do I have to specify something in the model code or sampling() function or loo() function?

Thank you for all your help with this!

6

I think so. And just like for loo without moment matching you’ll need to make sure to calculate and save the pointwise log likelihood in your Stan program.

There’s an example that might be helpful in this vignette:

7

It’s kind of funny that the title of that vignette says it avoids model refits when I just told
you to refit your model! But the title is referring to refitting for cross validation not refitting because the stanfit object is old.

8

I forgot to say that I would also recommend installing the loo package from GitHub. We just submitted an update to CRAN but it’s not up yet. It includes a few improvements for numerical stability that could affect moment matching (probably wouldn’t affect you but to be safe).

Edit: as of December 5th loo 2.4 is now on CRAN so installing from github is unnecessary

9

Thank you so much for your help! The models are running and I’ll check back and update you if this works.

1 Like
10

Hi @jonah,

I tried using the moment matching command in the loo() function, but two things happened that left me LOO-less:

  • With moment_match = T, R used up as much RAM as it could get ahold of. I tried running a smaller Wiener model on my computer than the ones I’m running full-bore (i.e., less chains and iterations), and with my 32 GB it froze my computer in minutes! When I tried to get the LOO using my universities cluster computer (Linux SLURM), it got to using 300-700 GB of RAM (depending on the model), which is insane and the jobs were killed by the admin before they could finish.
  • Running the loo() function with moment matching on the cluster computer also gave me the error Error: 'r_eff' must have one value per observation. which I assume is because the script was stopped prematurely… or is it this because the models had multiple chains and I can only use the moment-matched loo() with one chain in the model? I think a similar issue was reported HERE, but that seems to be an issue with Windows, not Linux.

Any help is greatly appreciated!

1 Like