Loo_moment_match not working with cmdstanr model

General
1

Hi,

I have some pareto-k values in (0.7, 1] for my model. It is fitted using cmdstanr, and i convert to a rstan object by using

fit_r <- rstan::read_stan_csv(fit$output_files())

Then, I try to use moment matching using the functions from here

log_lik <- extract_log_lik(fit, parameter_name = "log_lik")
mod_loo <- loo(log_lik, r_eff =  relative_eff(log_lik))

loo3 <- loo::loo_moment_match.default(
  x = fit_r,
  loo = mod_loo,
  post_draws = post_draws_stanfit,
  log_lik_i = log_lik_i_stanfit,
  unconstrain_pars = unconstrain_pars_stanfit,
  log_prob_upars = log_prob_upars_stanfit,
  log_lik_i_upars = log_lik_i_upars_stanfit
)

Which gives the following error

 Error in .local(object, ...) : 
  the model object is not created or not valid

Does anybody know what this error is caused by?

2

I assume this only transform the posterior draws to rstan object, but it’s not creating the binary for model.

Moment matching needs to be able to call these functions which are part of the Stan model binary ,which is likely to be missing in your case. So you need to compile your model with rstan to get access to these functions.

1 Like
3

Thanks for the response Aki

So, does this mean I have to fit the model using rstan? is there a way to use loo_moment_match with cmdstanr?

4

You just need to compile the model with rstan, and then use posterior draws from the csv file but functions from the compiled rstan model.

Not until we have log_prob (and other related functions) for CmdStan. This is on wishlist, but I don’t know when it could happen.

5

How do I use the functions from the compiled rstan model?

So I have tried to do,


meta_model2r <- rstan::read_stan_csv(fit$output_files())

rstan_mod <- stan_model("model.stan")

loo_model <- loo(meta_model2r)

loo3 <- loo::loo_moment_match.default(
  x = meta_model2r,
  loo = loo_model,
  post_draws = post_draws_stanfit,
  log_lik_i = log_lik_i_stanfit,
  unconstrain_pars = unconstrain_pars_stanfit,
  log_prob_upars = log_prob_upars_stanfit,
  log_lik_i_upars = log_lik_i_upars_stanfit,
  cores = 2
)

With functions (from here) ,

# create a named list of draws for use with rstan methods
.rstan_relist <- function(x, skeleton) {
  out <- utils::relist(x, skeleton)
  for (i in seq_along(skeleton)) {
    dim(out[[i]]) <- dim(skeleton[[i]])
  }
  out
}

# rstan helper function to get dims of parameters right
.create_skeleton <- function(pars, dims) {
  out <- lapply(seq_along(pars), function(i) {
    len_dims <- length(dims[[i]])
    if (len_dims < 1) return(0)
    return(array(0, dim = dims[[i]]))
  })
  names(out) <- pars
  out
}

# extract original posterior draws
post_draws_stanfit <- function(x, ...) {
  as.matrix(x)
}

# compute a matrix of log-likelihood values for the ith observation
# matrix contains information about the number of MCMC chains
log_lik_i_stanfit <- function(x, i, parameter_name = "log_lik", ...) {
  loo::extract_log_lik(x, parameter_name, merge_chains = FALSE)[, , i]
}

# transform parameters to the unconstrained space
unconstrain_pars_stanfit <- function(x, pars, ...) {
  skeleton <- .create_skeleton(x@sim$pars_oi, x@par_dims[x@sim$pars_oi])
  upars <- apply(pars, 1, FUN = function(theta) {
    rstan::unconstrain_pars(x, .rstan_relist(theta, skeleton))
  })
  # for one parameter models
  if (is.null(dim(upars))) {
    dim(upars) <- c(1, length(upars))
  }
  t(upars)
}

# compute log_prob for each posterior draws on the unconstrained space
log_prob_upars_stanfit <- function(x, upars, ...) {
  apply(upars, 1, rstan::log_prob, object = x,
        adjust_transform = TRUE, gradient = FALSE)
}

# compute log_lik values based on the unconstrained parameters
log_lik_i_upars_stanfit <- function(x, upars, i, parameter_name = "log_lik",
                                    ...) {
  S <- nrow(upars)
  out <- numeric(S)
  for (s in seq_len(S)) {
    out[s] <- rstan::constrain_pars(x, upars = upars[s, ])[[parameter_name]][i]
  }
  out
}
6

here x should be the rstan object, but you define

and

so maybe it works if you change that to

x = rstan_mod

Although I know that you need the rstan model to get log_prob and other functions, I haven’t tested mixing with cmdstanr csv, so it may still give some another error.

1 Like
7

Thanks, when I try to do that I get a different error

 Error in as.vector(data) : 
  no method for coercing this S4 class to a vector

I guess I’ll just need to run the models using rstan