Error when using roxygen2:roxygenize() when creating a package that depends on rstan

Hello everyone,

I am trying to create a package that depends on rstan. I closely followed the step by step guide Step by step guide for creating a package that depends on RStan • rstantools. However, when I try to call roxygen2::roxygenize(), I receive a lengthy output which ends with the following lines (including an error message):

C:/Users/k_jans13/AppData/Local/Programs/R/R-4.2.2/library/StanHeaders/include/src/stan/mcmc/hmc/hamiltonians/dense_e_metric.hpp:21:10: required from here
C:/Users/k_jans13/AppData/Local/Programs/R/R-4.2.2/library/RcppEigen/include/Eigen/src/Core/DenseCoeffsBase.h:55:30:warning: noring attributes on template argument 'Eigen::internal::packet_traits::type{aka '__m128d-Wignored-attributes]
as: stanExports_bbm_Harrison.o: too many sections (40516)
C:\Users\k_jans13\AppData\Local\Temp\ccNE2LLC.s: Assembler messages:
C:\Users\k_jans13\AppData\Local\Temp\ccNE2LLC.s: Fatal error: can’t write 87 bytes to section .text of stanExports_bbm_Harrison.o: ‘file too big’
as: stanExports_bbm_Harrison.o: too many sections (40516)
C:\Users\k_jans13\AppData\Local\Temp\ccNE2LLC.s: Fatal error: stanExports_bbm_Harrison.o: file too big
make: *** [C:/Users/k_jans13/AppData/Local/Programs/R/R-42~1.2/etc/x64/Makeconf:258: stanExports_bbm_Harrison.o] Error 1
ERROR: compilation failed for package ‘bmetabbin’
─ removing ‘C:/Users/k_jans13/AppData/Local/Temp/RtmpuYpFW0/devtools_install_1c345b906580/bmetabbin’
Error in (function (command = NULL, args = character(), error_on_status = TRUE, …:
! System command ‘Rcmd.exe’ failed

Exit status: 1
stdout & stderr:

Type .Last.error to see the more details.

The output of .Last.error is

<system_command_status_error/rlib_error_3_0/rlib_error/error>
Error in (function (command = NULL, args = character(), error_on_status = TRUE, …:
! System command ‘Rcmd.exe’ failed

Exit status: 1
stdout & stderr:

Backtrace:

  1. roxygen2::roxygenise()
  2. local load_code(base_path)
  3. pkgload::load_all(path, helpers = FALSE, attach_testthat = FALSE)
  4. pkgbuild::compile_dll(path, quiet = quiet)
  5. pkgbuild:::install_min(path, dest = install_dir, components = “libs”, args = if (needs_c…
  6. pkgbuild::rcmd_build_tools(“INSTALL”, c(path, paste(“–library=”, dest, …
  7. pkgbuild::with_build_tools({ …
  8. withr::with_path(rtools_path(), code)
  9. base::force(code)
  10. base::withCallingHandlers(callr::rcmd_safe(…, env = env, spinner = FALSE, …
  11. callr::rcmd_safe(…, env = env, spinner = FALSE, show = FALSE, …
  12. callr:::run_r(options)
  13. base::with(options, with_envvar(env, do.call(processx::run, c(list(bin, …
  14. base::with.default(options, with_envvar(env, do.call(processx::run, …
  15. base::eval(substitute(expr), data, enclos = parent.frame())
  16. base::eval(substitute(expr), data, enclos = parent.frame())
  17. callr:::with_envvar(env, do.call(processx::run, c(list(bin, args = real_cmdargs, …
  18. base::force(code)
  19. base::do.call(processx::run, c(list(bin, args = real_cmdargs, stdout_line_callback …
  20. (function (command = NULL, args = character(), error_on_status = TRUE, …
  21. base::throw(new_process_error(res, call = sys.call(), echo = echo, …
  22. | base::signalCondition(cond)
  23. (function (e) …
  24. asNamespace(“callr”)$err$throw(e)

I am using Rstan version 2.26.13 on Windows.

The output of devtools::session_info(“rstan”) is:

─ Session info ──────────────────────────────────────────────────────────────────
setting value
version R version 4.2.2 (2022-10-31 ucrt)
os Windows 10 x64 (build 19044)
system x86_64, mingw32
ui RStudio
language (EN)
collate German_Germany.utf8
ctype German_Germany.utf8
tz Europe/Berlin
date 2022-12-20
rstudio 2022.07.2+576 Spotted Wakerobin (desktop)
pandoc NA

─ Packages ──────────────────────────────────────────────────────────────────────
! package * version date (UTC) lib source
backports 1.4.1 2021-12-13 [1] CRAN (R 4.2.0)
BH 1.78.0-0 2021-12-15 [1] CRAN (R 4.2.0)
callr 3.7.3 2022-11-02 [1] CRAN (R 4.2.2)
checkmate 2.1.0 2022-04-21 [1] CRAN (R 4.2.2)
cli 3.4.1 2022-09-23 [1] CRAN (R 4.2.2)
colorspace 2.0-3 2022-02-21 [1] CRAN (R 4.2.2)
crayon 1.5.2 2022-09-29 [1] CRAN (R 4.2.2)
curl 4.3.3 2022-10-06 [1] CRAN (R 4.2.2)
desc 1.4.2 2022-09-08 [1] CRAN (R 4.2.2)
fansi 1.0.3 2022-03-24 [1] CRAN (R 4.2.2)
farver 2.1.1 2022-07-06 [1] CRAN (R 4.2.2)
ggplot2 3.4.0 2022-11-04 [1] CRAN (R 4.2.2)
glue 1.6.2 2022-02-24 [1] CRAN (R 4.2.2)
gridExtra 2.3 2017-09-09 [1] CRAN (R 4.2.2)
gtable 0.3.1 2022-09-01 [1] CRAN (R 4.2.2)
inline 0.3.19 2021-05-31 [1] CRAN (R 4.2.2)
isoband 0.2.6 2022-10-06 [1] CRAN (R 4.2.2)
jsonlite 1.8.4 2022-12-06 [1] CRAN (R 4.2.2)
labeling 0.4.2 2020-10-20 [1] CRAN (R 4.2.0)
lattice 0.20-45 2021-09-22 [1] CRAN (R 4.2.2)
lifecycle 1.0.3 2022-10-07 [1] CRAN (R 4.2.2)
loo 2.5.1 2022-03-24 [1] CRAN (R 4.2.2)
magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.2.2)
MASS 7.3-58.1 2022-08-03 [1] CRAN (R 4.2.2)
D Matrix 1.5-1 2022-09-13 [1] CRAN (R 4.2.2)
matrixStats 0.63.0 2022-11-18 [1] CRAN (R 4.2.2)
mgcv 1.8-41 2022-10-21 [1] CRAN (R 4.2.2)
munsell 0.5.0 2018-06-12 [1] CRAN (R 4.2.2)
nlme 3.1-160 2022-10-10 [1] CRAN (R 4.2.2)
pillar 1.8.1 2022-08-19 [1] CRAN (R 4.2.2)
pkgbuild 1.4.0 2022-11-27 [1] CRAN (R 4.2.2)
pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.2.2)
prettyunits 1.1.1 2020-01-24 [1] CRAN (R 4.2.2)
processx 3.8.0 2022-10-26 [1] CRAN (R 4.2.2)
ps 1.7.2 2022-10-26 [1] CRAN (R 4.2.2)
R6 2.5.1 2021-08-19 [1] CRAN (R 4.2.2)
RColorBrewer 1.1-3 2022-04-03 [1] CRAN (R 4.2.0)
Rcpp 1.0.9 2022-07-08 [1] CRAN (R 4.2.2)
RcppEigen 0.3.3.9.3 2022-11-05 [1] CRAN (R 4.2.2)
D RcppParallel 5.1.5 2022-01-05 [1] CRAN (R 4.2.2)
rlang 1.0.6 2022-09-24 [1] CRAN (R 4.2.2)
rprojroot 2.0.3 2022-04-02 [1] CRAN (R 4.2.2)
rstan * 2.26.13 2022-06-25 [1] local
scales 1.2.1 2022-08-20 [1] CRAN (R 4.2.2)
StanHeaders * 2.26.13 2022-06-25 [1] local
tibble 3.1.8 2022-07-22 [1] CRAN (R 4.2.2)
utf8 1.2.2 2021-07-24 [1] CRAN (R 4.2.2)
V8 4.2.2 2022-11-03 [1] CRAN (R 4.2.2)
vctrs 0.5.1 2022-11-16 [1] CRAN (R 4.2.2)
viridisLite 0.4.1 2022-08-22 [1] CRAN (R 4.2.2)
withr 2.5.0 2022-03-03 [1] CRAN (R 4.2.2)

[1] C:/Users/k_jans13/AppData/Local/Programs/R/R-4.2.2/library

The output of `writeLines(readLines(file.path(Sys.getenv(“HOME”), “.R/Makevars”))) is

Error in file(con, “r”) : cannot open the connection
In addition: Warning message:
In file(con, “r”) :
cannot open file ‘C:/Users/k_jans13/Documents/.R/Makevars’: No such file or directory

However, the Makevars file is actually in the folder C:/Users/k_jans13/Documents/.R/Makevars and looks like this:

CXX14FLAGS += -O3 -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2

I already tried to reinstall rstan and StanHeaders, but the same error still occurred. I found this thread Error when trying to create R package that involves stan, using rstantools - #4 by ebuhle which seems to discuss a similar issue, but I am new to using Stan and do not really understand the solution which is being proposed there.

Compiling Stan models using rstan outside of my package works without any issues.

I would appreciate any help! If there is any more information I need to provide, I am happy to do so.

Best regards
Katrin

This can happen when using larger/complex models. Let’s try requesting more aggressive optimisation when compiling c++.

In your package directory you should have a src/Makevars.win file, at the bottom of the file add the line:

PKG_CXXFLAGS += -O3

Then try the compilation again

Thank you so much for your quick reply. I added PKG_CXXFLAGS += -03 to the Makevars.win file in the src directory, but received the same error as before (the output from .Last.error is also the same as before).

Do you have the package source on github or similar so I could debug locally? You can also PM me a zip file of the source if it’s not meant to be public

Hi @katrija, I should probably leave this to the expert, and if Andrew does come up with a solution related to the package source I’ll be interested to see what it is. Still, I can weigh in on this specific point:

I can’t say with confidence that your issue is the same one I encountered, but if the “lengthy output” you’re getting includes many warnings like this

/include/Eigen/src/Core/DenseCoeffsBase.h:55:30:warning: noring attributes on template argument 'Eigen::internal::packet_traits::type{aka '__m128d-Wignored-attributes]

then it does sound similar. (Caveat: I am not a computer scientist, if that wasn’t obvious.) The solution in my case was to add lines to Makevars that suppress these warnings, e.g.

CXX14FLAGS += -Wno-ignored-attributes

Looking through the output, I found several different types of warnings that needed to be silenced. The GitHub Issue linked in my thread shows that these excessively verbose warnings started happening with recent versions of the toolchain (precisely which component(s) are at fault was never resolved, I don’t think) and affect compilation of TMB as well as Stan code. Those TMB folks didn’t report crashes, just obnoxiously lengthy and unhelpful output, but I was indeed getting fatal errors with 'file too big' until I added the aforementioned compiler flags.

I don’t know if this is more likely to affect larger or more complex models – that certainly applies to the package I was working on when this problem started, but I then experienced the same behavior with all of my Stan-dependent packages, including the toy rstanlm from the vignette. (@katrija, are you able to compile that one?)

Hi @ebuhle,

thank you so much for your detailled explanation. My “lengthy output” indeed contains many warnings like the one you described.

I tried to compile the rstanlm package, but received the same error as I did for my own package.

I also tried to add

CXX14FLAGS += -Wno-ignored-attributes

to my Makevars file (i.e., the src/Makevars.win file and the src/Makevars file). When I call roxygen2::roxygenize() after adding this line to both files, I get the same error as before and on top of that, the line CXX14FLAGS += -Wno-ignored-attributes is removed from the Makevars files (this seems to happen the moment I call roxygen2::roxygenize()).

OK, you don’t need these flags in the package Makevars.win, only in your user Makevars. And you may well have other warnings besides Wignored-attributes that need to be suppressed. In my case, scrolling through the output before the fatal error led me to the following (note that the line breaks are just for readability):

CXX14FLAGS += -Wno-unused -Wno-ignored-attributes -Wno-sign-compare 
CXX14FLAGS += -Wno-deprecated-declarations -Wno-attributes -Wno-parentheses 
CXX14FLAGS += -Wno-unknown-pragmas -Wno-infinite-recursion

But just to be clear, you’re saying that this

happens to your user Makevars as well? That is extremely weird. Are you sure that R is finding and invoking your Makevars, given that it’s in a non-default location?

EDIT: After reading your post again more carefully, it looks like you only added -Wno-ignored-attributes to the Makevars files in the package itself. Those are not the droids you’re looking for, and it makes sense that compilation will reset them. You want to modify your user Makevars (and also double-check that R knows where to find it).

Hi @katrija, just curious, did you ever get this resolved?

Hi @ebuhle, sorry for my late reply.

Via PM, Andrew suggested to add

PKG_CXXFLAGS += -Wa,-mbig-obj

to the src/Makevars.win file, which did the job. That is, I am now able to compile, install and use my package locally. I still get the lengthy output when compiling (I didn’t test yet whether it disappears when I edit the user Makevars file as you suggested, but I will try when I have to recompile the package. How can I double-check that R knows where my user Makevars file is?). Also, compilation took some time (10-15 minutes - not sure whether this is normal, but for now I don’t mind). Most importantly, as far as I can tell, everything now works as it should.

Interesting, that’s a new one on me. I gather compilation no longer resets src/Makevars.win to the default? I’ll have to try this on my package; it would be nice to be able to resolve this issue internally so I don’t have to instruct users to edit their Makevars.

One way to check is to look at the output that is printed to the console during compilation. The first block (before any warnings) contains the compiler flags. More simply, though, my understanding is that if your Makevars.win is in a non-default location, you need to set the R environment variable R_MAKEVARS_USER to a text string giving the path. That said, the easiest thing is just to move Makevars.win to the default .R directory if possible.

You can speed up compilation by using multiple cores. Set the R environment variable MAKEFLAGS to -jX, where X is the number of cores. For example, my .Renviron file contains MAKEFLAGS = -j4, and compiling my package (which contains 9 fairly complex Stan models) takes around 5 min on a 2.9 GHz machine with 16 GB RAM.

Part of the reason I wanted to follow up is because this is now a reproducible issue for at least two rstantools users (i.e., it’s not just me) even with the minimal example package, so I wonder if it’s worth addressing it in the rstantools vignette(s). @andrjohns, any thoughts?