"Error: Model not compiled." when using brms::kfold with cmdstanr backend

Interfaces brms
,
1

I am having trouble using the brms::kfold function with models that have been fit with brms.backend = "cmdstanr". I get an error telling me that the model [is] not compiled. I only get this error when saving and loading models in two different R sessions.

Here is a minimal, reproducible example:

# Load packages
library(tidyverse)
library(brms)

# Options
options(
  mc.cores = parallel::detectCores(),
  brms.backend = "cmdstanr"
)

# Fit models
m1_fit <- brm(
  count ~ 1 + (1|patient),
  data = epilepsy,
  family = poisson()
)
m2_fit <- brm(
  count ~ 1 + zAge + zBase * Trt + (1|patient),
  data = epilepsy,
  family = poisson()
)

# Save results
write_rds(m1_fit, "m1_fit.rds")
write_rds(m2_fit, "m2_fit.rds")

After running this code, I restart R and run the following code in a new R session:

# Load packages
library(tidyverse)
library(brms)

# Options
options(
  mc.cores = parallel::detectCores(),
  brms.backend = "cmdstanr"
)

# Load results
m1_fit <- read_rds("m1_fit.rds")
m2_fit <- read_rds("m2_fit.rds")

# Compare models
kfold(
  m1_fit,
  m2_fit,
  chains = 1
)

I get the following error message:

Fitting model 1 out of 10
Fitting model 2 out of 10
Fitting model 3 out of 10
Fitting model 4 out of 10
Fitting model 5 out of 10
Fitting model 6 out of 10
Fitting model 7 out of 10
Fitting model 8 out of 10
Fitting model 9 out of 10
Fitting model 10 out of 10
Start sampling
Error: Model not compiled. Try running the compile() method first.

I suspect that it has something to do with the issues described in the cmdstanr documentation on the save_object function:

This method is a wrapper around base::saveRDS() that ensures that all posterior draws and diagnostics are saved when saving a fitted model object. Because the contents of the CmdStan output CSV files are only read into R lazily (i.e., as needed), the $save_object() method is the safest way to guarantee that everything has been read in before saving.

  • Operating System: Windows 10 x64 (build 19043)
  • brms Version: 2.15.0

Thank you in advance!

2

Hi @nkreimer, sorry for taking a while to get to you. Can you by any chance provide a true reproducible example, by including data or code to simulate data that reproduces the issue for you? Also, just in case, you might try upgrading to the latest brms with remotes::install_github("paul-buerkner/brms").

3

Hi @jsocolar, thanks for your response. Is the example I provided not reproducible? I use the epilepsy dataset that comes included with the brms package.

1 Like
4

Oh jeez, I’m sorry, it is! Sorry for not running. I didn’t see epilepsy defined in the script and assumed it was external. Seeing if I can reproduce your behavior now.

1 Like
5

I can run your code without problems on my system, which uses the development version of brms. Can you try installing the development version of brms with remotes::install_github('paul-buerkner/brms') and see if the problem persists?

6

I installed the development version of brms of GitHub (2.15.9) but the problem persits.

Fitting model 1 out of 10
Fitting model 2 out of 10
Fitting model 3 out of 10
Fitting model 4 out of 10
Fitting model 5 out of 10
Fitting model 6 out of 10
Fitting model 7 out of 10
Fitting model 8 out of 10
Fitting model 9 out of 10
Fitting model 10 out of 10
Start sampling
Error: Model not compiled. Try running the compile() method first.

Did you restart R before running the second chunk of code?

7

yep! And I just did it again for good measure.

I don’t know why this would be the issue, but perhaps you could also try upgrading cmdstanr to the development version (remotes::install_github('stan-dev/cmdstanr'))?

8

How peculiar! I did that and still get the same error. For good measure, I rewrote the example so that it doesn’t depend on the tidyverse but am still getting the same error.

# Load packages
library(brms)

# Options
options(
  mc.cores = parallel::detectCores(),
  brms.backend = "cmdstanr"
)

# Fit models
m1_fit <- brm(
  count ~ 1 + (1|patient),
  data = epilepsy,
  family = poisson()
)
m2_fit <- brm(
  count ~ 1 + zAge + zBase * Trt + (1|patient),
  data = epilepsy,
  family = poisson()
)

# Save results
saveRDS(m1_fit, "m1_fit.rds")
saveRDS(m2_fit, "m2_fit.rds")

# Load packages
library(brms)

# Options
options(
  mc.cores = parallel::detectCores(),
  brms.backend = "cmdstanr"
)

# Load results
m1_fit <- readRDS("m1_fit.rds")
m2_fit <- readRDS("m2_fit.rds")

# Compare models
kfold(
  m1_fit,
  m2_fit,
  chains = 1
)
9

@rok_cesnovar @paul.buerkner any ideas here?

10

Have there been any developments on this? or workarounds? I am having the exact same issue.

11

I am having a similar issue recently when using LOO with reloo = TRUE; but only on one type of multivariate model I’m running.

Environment:

R version 4.1.1 (2021-08-10)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.7

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] posterior_1.1.0  cmdstanr_0.4.0   kableExtra_1.3.4 ggExtra_0.9      tidybayes_3.0.1  forcats_0.5.1    stringr_1.4.0    dplyr_1.0.7      purrr_0.3.4      readr_2.0.2      tidyr_1.1.4     
[12] tibble_3.1.5     ggplot2_3.3.5    tidyverse_1.3.1  brms_2.16.1      Rcpp_1.0.7

Model:

mod.BayesNASA0_wslopes <- brm(mvbind(NASAtlx_performance_round, NASAtlx_mental_round,
                                     NASAtlx_effort_round, NASAtlx_temporal_round,
                                     NASAtlx_frustration_round, NASAtlx_physical_round) ~ 1 +
                                  scale(iblock) + (1 + scale(iblock) | p | ID),
                              data=blockData,
                              family = cumulative(link = "logit"),
                              prior = mypriorsNT, sample_prior = "yes",
                              warmup = 750, iter = 4000, chains = 2,
                              inits= "0", seed = 123,
                              backend="cmdstanr",
                              file = "./modelcache/mod.BayesNASA0_wslopes")

LOO code:

loo_nullNASA_wslopes <- loo(mod.BayesNASA0_wslopes, save_psis = TRUE, moment_match = TRUE, reloo = TRUE)

Traceback:

Error: Model not compiled. Try running the compile() method first.
27. stop(condition)
26. signalConditions(future, exclude = getOption("future.relay.immediate", 
"immediateCondition"), resignal = TRUE)
25. value.Future(futures[[j]])
24. future::value(futures[[j]])
23. reloo.brmsfit(x = .x1, loo = .x2, newdata = .x3, resp = .x4, 
k_threshold = .x5, check = .x6)
22. reloo(x = .x1, loo = .x2, newdata = .x3, resp = .x4, k_threshold = .x5, 
check = .x6)
21. eval(expr, envir, ...)
20. eval(expr, envir, ...)
19. eval2(call, envir = args, enclos = envir)
18. do_call("reloo", reloo_args)
17. .loo(x = .x1, newdata = .x2, resp = .x3, model_name = .x4, pointwise = .x5, 
k_threshold = .x6, save_psis = .x7, moment_match = .x8, reloo = .x9, 
moment_match_args = .x10, reloo_args = .x11)
16. eval(expr, envir, ...)
15. eval(expr, envir, ...)
14. eval2(call, envir = args, enclos = envir)
13. do_call(paste0(".", criterion), args)
12. .fun(criterion = .x1, pointwise = .x2, resp = .x3, k_threshold = .x4, 
save_psis = .x5, moment_match = .x6, reloo = .x7, moment_match_args = .x8, 
reloo_args = .x9, x = .x10, model_name = .x11, use_stored = .x12)
11. eval(expr, envir, ...)
10. eval(expr, envir, ...)
9. eval2(call, envir = args, enclos = envir)
8. do_call(compute_loo, args)
7. .fun(models = .x1, criterion = .x2, pointwise = .x3, compare = .x4, 
resp = .x5, k_threshold = .x6, save_psis = .x7, moment_match = .x8, 
reloo = .x9, moment_match_args = .x10, reloo_args = .x11)
6. eval(expr, envir, ...)
5. eval(expr, envir, ...)
4. eval2(call, envir = args, enclos = envir)
3. do_call(compute_loolist, args)
2. loo.brmsfit(mod.BayesNASA0_wslopes, save_psis = TRUE, moment_match = TRUE, 
reloo = TRUE)
1. loo(mod.BayesNASA0_wslopes, save_psis = TRUE, moment_match = TRUE, 
reloo = TRUE)
12

I added some functionality in the github version of brms that automatically performs recompilation, if necessary. You can also avoid repeated recompilations within the same session by running recompile_model once on your brmsfit object.

1 Like
13

Thanks Paul; I have tried moving to the BRMS version and running recompile_model, but it is still giving me some problems. Though I can pull up the man page successfully with ?recompile_model, it seems like it’s not recognizing it as a function?

> recompile_model(mod.BayesNASA0_wslopes, recompile = TRUE)
Error in recompile_model(mod.BayesNASA0_wslopes, recompile = TRUE) : 
  could not find function "recompile_model"

Traceback for the loo command also changes after installing github brms version:

Error in FUN(X[[i]], ...) : 
lazy-load database '/Library/Frameworks/R.framework/Versions/4.1/Resources/library/brms/data/Rdata.rdb' is corrupt
32. FUN(X[[i]], ...)
31. vapply(li, exists, NA, where = i, mode = mode, inherits = FALSE)
30. apropos(model_re, mode = "S4", ignore.case = FALSE)
29. stan_model(file, model_name = model_name, model_code = model_code, 
stanc_ret = NULL, boost_lib = boost_lib, eigen_lib = eigen_lib, 
save_dso = save_dso, verbose = verbose)
28. rstan::stan(model_code = stancode(x, threads = threading()), 
data = standata(x), chains = 0)
27. withCallingHandlers(expr, message = function(c) if (inherits(c, 
classes)) tryInvokeRestart("muffleMessage"))
26. suppressMessages(rstan::stan(model_code = stancode(x, threads = threading()), 
data = standata(x), chains = 0))
25. add_rstan_model(x)
24. update_misc_env(x)
23. loo_moment_match.brmsfit(x = .x1, loo = .x2, newdata = .x3, resp = .x4, 
k_threshold = .x5, check = .x6)
22. loo_moment_match(x = .x1, loo = .x2, newdata = .x3, resp = .x4, 
k_threshold = .x5, check = .x6)
21. eval(expr, envir, ...)
20. eval(expr, envir, ...)
19. eval2(call, envir = args, enclos = envir)
18. do_call("loo_moment_match", moment_match_args)
17. .loo(x = .x1, newdata = .x2, resp = .x3, model_name = .x4, pointwise = .x5, k_threshold = .x6, save_psis = .x7, moment_match = .x8, reloo = .x9, moment_match_args = .x10, reloo_args = .x11)
16. eval(expr, envir, ...)
15. eval(expr, envir, ...)
14. eval2(call, envir = args, enclos = envir)
13. do_call(paste0(".", criterion), args)
12. .fun(criterion = .x1, pointwise = .x2, resp = .x3, k_threshold = .x4, save_psis = .x5, moment_match = .x6, reloo = .x7, moment_match_args = .x8, reloo_args = .x9, x = .x10, model_name = .x11, use_stored = .x12)
11. eval(expr, envir, ...)
10. eval(expr, envir, ...)
9. eval2(call, envir = args, enclos = envir)
8. do_call(compute_loo, args)
7. .fun(models = .x1, criterion = .x2, pointwise = .x3, compare = .x4,  resp = .x5, k_threshold = .x6, save_psis = .x7, moment_match = .x8, reloo = .x9, moment_match_args = .x10, reloo_args = .x11)
6. eval(expr, envir, ...)
5. eval(expr, envir, ...)
4. eval2(call, envir = args, enclos = envir)
3. do_call(compute_loolist, args)
2. loo.brmsfit(mod.BayesNASA0_wslopes, save_psis = TRUE, moment_match = TRUE,  reloo = TRUE)
1. loo(mod.BayesNASA0_wslopes, save_psis = TRUE, moment_match = TRUE, reloo = TRUE)
14

perhaps R needs to be restarted again?

I realize I implemented the internal auto checking only for kfold but not for reloo. will add this today.

3 Likes
15

I re-installed cmdstan and brms from the github sources, updated any other packages that could be updated, and restarted R-- not sure which one did the trick, but things seem to be working now with LOO; it recompiles without me having to call it again explicitly. Thanks Paul! @nkreimer are you still stuck with the same error?

16

I ended up circumventing the problem by running the kfold cross-validation in the same script instead of saving the intermediate results.

17

Hi all,

I don’t know if there has been a follow-up with this error as to why it happens, but I also encountered it and reported it in this thread [Multiple attemps to run k-fold cross-validation fail with brms]. I tried to do the solution suggested by @nkreimer but it did not work for me and returned a new error.

Thank you and have a good day.

1 Like