Cmdstanr loo_moment_match

Interfaces CmdStan
,
1

Does fit$loo(moment_match = TRUE) work?

1 Like
2

Unfortunately not. This is because the loo_moment_match() function in the loo package requires arguments like unconstrain_pars and log_prob_upars (described in Moment matching for efficient approximate leave-one-out cross-validation (LOO) — loo_moment_match • loo). These are possible to specify automatically if using RStan but not yet if using any interface based on CmdStan (e.g., RStan has a function rstan::unconstrain_pars() but we don’t have the equivalent in CmdStan yet). It will hopefully be possible in the future!

4 Likes
3

Note that if you have fitted a costly model with cmdstanr and you need to do loo with moment matching, and you don’t want to re-fit the model, it is possible to recompile the model with rstan for use in conjunction with the draws from cmdstan.

I’ll try to work out exactly how to do this (brms manages it somehow or other) and post back here with a code snippet in the next day or two.

3 Likes
4

@jsocolar good point! Would be cool to see an example of this.

5

Hey, I was looking for a solution to this as well, and I believe I found the brms code @jsocolar mentioned here

Based on that code snippet I created a method that converts my cmdstanr model to an rstan model that lets me use loo with moment matching, without having to fit it again.

cmdstanr2rstan <- function(fit, stan_data, stan_file) {
  rstanfit <- rstan::read_stan_csv(fit$output_files())
  stanfit_ <- rstan::stan(stan_file, 
                          data = stan_data, chains = 0)
  rstanfit@stanmodel <- stanfit_@stanmodel
  rstanfit@.MISC <- stanfit_@.MISC
  rstanfit
}

e: excuse me, I didn’t see a year date, I thought this post was from this week…

4 Likes
6

The new just released CmdStan 2.31 supports log_prob, so we may get loo_moment_match to work with CmdStanR, too, soon

3 Likes
7

any updates for adding this functionality?

8

log_prob functionality is now merged in github version of CmdStanR (Add optional methods for log_prob, grad_log_prob, hessian, un/constrain pars by andrjohns ¡ Pull Request #701 ¡ stan-dev/cmdstanr ¡ GitHub), but there is still on-going discussion for the interface. After the interface part is settled and merged, it is possible to start working on adding loo_moment_match

1 Like
9

Can you give an example of what it would look like to do this manually for a cmdstanr model? Rstan is way behind cmdstanr in terms of keeping up with the Stan language, so mathan’s cmdstanr2rstan() function doesn’t work for me – Rstan won’t compile the model. (I tried making minimal changes to make it compile under Rstan, but did not succeed.)

10

Maybe @andrjohns can help?

11

@Kevin_Van_Horn the GitHub version of cmdstanr supports loo with moment-matching.

You can just call: fit$loo(moment_match=true)

3 Likes
12

Awesome!

3 Likes
13

Seems to be included also in the new cmdstanr 0.6.0 release! Changelog • cmdstanr

1 Like
14

Great!

15

OK, I just tried it and I immediately got this error:

Error: Model methods cannot be used with a pre-compiled Stan executable, the model must be compiled again

The fit object was created using cmdstanr::as_cmdstan_fit from output CSV files. What does “compile the model again” mean in that context, since no CmdStanModel object was used to create the fit object?

16

Because moment-matching requires the log-probability and related functions from the Stan model, cmdstanr needs the Stan code so that it compile those methods and call them.

If you just have the CSV files from fitting a model, then cmdstanr has no way of knowing what Stan code was used to generate it

17

Is there a way to create a suitable fit object to use with loo(moment_match = TRUE) if I have both the CSV files and Stan code available?

18

You can create a ‘dummy’ fit object using the Stan code and data, and then replace its draws with those from the existing CSVs:

fit_csv <- as_cmdstan_fit(list.files(pattern = ".csv"))

mod1 <- cmdstan_model("model.stan", force_recompile = TRUE)
fit_dummy <- mod1$sample(data = "model.data.json",
                         iter_warmup = 1,
                         iter_sampling = 1)

fit_dummy$.__enclos_env__$private$draws_ <- fit_csv$.__enclos_env__$private$draws_
fit_dummy$loo(moment_match = TRUE)
4 Likes