Pkgbuild::compile_dll() on previously working rstantools-based package fails with "file too big" error

Interfaces
1

I’m encountering a new (to me) problem when trying to compile my Stan-dependent packages built with rstantools in RStudio. My workflow, following the vignette, has been:

pkgbuild::compile_dll()
roxygen2::roxygenize()
devtools::install(upgrade = FALSE)

But now compile_dll(), which in the past has worked quickly and mostly silently, has suddenly started puking out a massive stream of warnings to the console. After several minutes of this it terminates in the following fatal error:

 Error in (function (command = NULL, args = character(), error_on_status = TRUE,  : 
  System command 'Rcmd.exe' failed, exit status: 1, stdout + stderr (last 10 lines):
E> make: *** [C:/R/R-40~1.5/etc/x64/Makeconf:227: stanExports_IPM_ICchinook_pp.o] Error 1
E> make: *** Waiting for unfinished jobs....
E> C:/rtools40/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.3.0/../../../../x86_64-w64-mingw32/bin/as.exe: stanExports_IPM_LCRchum_pp.o: too many sections (66611)
E> C:\Users\Buhle\AppData\Local\Temp\ccuYkIrh.s: Assembler messages:
E> C:\Users\Buhle\AppData\Local\Temp\ccuYkIrh.s: Fatal error: can't write 56 bytes to section .text of stanExports_IPM_LCRchum_pp.o: 'file too big'
E> C:/rtools40/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.3.0/../../../../x86_64-w64-mingw32/bin/as.exe: stanExports_IPM_LCRchum_pp.o: too many sections (66611)
E> C:\Users\Buhle\AppData\Local\Temp\ccuYkIrh.s: Fatal error: can't close stanExports_IPM_LCRchum_pp.o: file too big
E> make: *** [C:/R/R-40~1.5/etc/x64/Makeconf:
[...]

What’s really baffling is that the package where this first started happening (the example shown above) had just been compiled without incident two days earlier, and literally nothing in the package code had changed, nor had I made any changes (that I’m aware of) to any relevant system files. I’ve since verified that the problem affects all my Stan / rstantools-dependent packages, including some I hadn’t touched in months. The previously compiled and installed “good” version of the package shown above still runs fine, as does rstanarm. Stan models called with rstan() still compile and run fine.

Things I’ve tried that haven’t worked:

  • Deleting the contents of the /src directory.
  • Deleting the file C:\Users\Buhle\AppData\Local\Temp\ccuYkIrh.s mentioned in the error message, and any other similar-looking temp files.
  • Calling rstantools::rstan_config() before compile_dll(), as described here.
  • Checking to see if the failed compile attempt had left one of those 00-LOCK files in my library directory (it hadn’t).
  • Based on the cryptic “file too big” error and this thread, adding the -g0 flag to my Makevars.win.
  • Deleting the entire contents of Makevars.win and just using defaults.
  • Uninstalling and reinstalling pkgbuild.

I haven’t yet uninstalled and reinstalled rstan and StanHeaders as mentioned here, mostly because it’s not clear to me whether the versions on mc-stan.org are the more recent ones that might resurrect a deal-breaking issue with Stan files in RStudio that I just recently, finally, resolved.

I’d be grateful for any suggestions. As always, thanks!

Session info:

R version 4.0.5 (2021-03-31)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.6           compiler_4.0.5       pillar_1.5.1         prettyunits_1.1.1   
 [5] tools_4.0.5          pkgbuild_1.2.0       jsonlite_1.7.2       lifecycle_1.0.0     
 [9] tibble_3.1.0         gtable_0.3.0         pkgconfig_2.0.3      rlang_0.4.10        
[13] DBI_1.1.1            cli_2.4.0            parallel_4.0.5       curl_4.3            
[17] loo_2.4.1            gridExtra_2.3        withr_2.4.1          dplyr_1.0.5         
[21] generics_0.1.0       vctrs_0.3.7          tidyselect_1.1.0     stats4_4.0.5        
[25] grid_4.0.5           glue_1.4.2           inline_0.3.17        R6_2.5.0            
[29] processx_3.5.1       fansi_0.4.2          rstan_2.21.2         purrr_0.3.4         
[33] ggplot2_3.3.3        callr_3.6.0          magrittr_2.0.1       codetools_0.2-18    
[37] matrixStats_0.58.0   scales_1.1.1         ps_1.6.0             StanHeaders_2.21.0-7
[41] ellipsis_0.3.1       assertthat_0.2.1     colorspace_2.0-0     V8_3.4.0            
[45] utf8_1.2.1           RcppParallel_5.0.3   munsell_0.5.0        crayon_1.4.1
2

I checked for you on the web and I think the problem is not related specifically to rstantools but more to the size of the compiled files. Maybe this post or that post may help.

3

Thanks @Kbus007, your Google-fu and/or C++ knowledge is stronger than mine. I’m afraid I have no idea how to implement the solutions mentioned in those posts, but I get the impression they are necessitated by the nature of the source file (e.g., “us[ing] templates intensively”). I don’t see how the source file(s) could be the problem here when, again, literally nothing in the package code has changed since a few days ago when compilation worked correctly. Seems like a toolchain issue, but having eliminated the obvious suspects mentioned in the OP, I’m at a loss.

4

My knowledge of the toolchain is also limited but I think you have to change your makevars file.

In this post, someone encountered a similar issue and the solution proposed by @bgoodri was

This seems to have solved the problem.

5

Thanks, that’s an interesting thread, but I’ve had -O3 in CXX14FLAGS, CXX11FLAGS, and even CXXFLAGS all along (except for when I tried a blank Makevars.win as mentioned above).

I guess tomorrow I’ll start uninstalling and reinstalling everything and see where that gets me.

6

I apparently resolved this by doing Clean and Rebuild under the Build tab in RStudio. I didn’t think these GUI options even worked for rstantools-based packages or I would have been using them instead of the command line all along, but I tested it with my copy of rstanlm, the toy package from the vignette, which had the same issue with pkgbuild::compile_dll(). After Clean and Rebuild-ing once, I was able to compile again using the workflow in the OP.

It worked on my real package of interest too, but that initial Clean and Rebuild took, no kidding, 15 hours for what is normally a 15-minute compile. Subsequent recompiles with the devtools framework were no slower than usual.

I’ll close this topic under “all’s well that ends well”, but I still have no idea what caused this. Also, I notice that the warnings devtools::install() spits out to the console (many of which I can suppress with -Wno-ignored-attributes, etc., but I’m testing with a minimal Makevars.win) seem very similar to the ones pkgbuild::compile_dll() was choking on. Maybe they were somehow getting “pushed back” in the compiler chain? Very mysterious.

7

Your story is interesting; glad you solved the problem.

If I understand well, your package which was compiling within 15 minutes is now taking 15 hours ?
I encountered a similar problem with a rstantools-package when I moved from R 3.6 to R 4.0, see my post.
In fact, the problem was that the C++ toolchain was misconfigured. I followed the instructions for R4.0 in this vignette and I was then able to compile “as fast as before”.

8

Now that you mention it, @Kbus007, I remember seeing that thread. I was intrigued by the possibility of putting prebuilt binaries on GitHub, but unclear on the details (e.g., “I would not put them in git”, but they can still be on GH?) and haven’t had time to follow up.

In my case, though:

Same toolchain configuration in both cases. I haven’t tried Clean and Rebuild-ing a second time, but I’m guessing that method, like the devtools-based method, would also be “as fast as before” now that I’ve cleared out the problem – whatever it was. I’m not sure what Clean and Rebuild does that’s different from manually emptying the /src directory (which didn’t work, as noted in the OP), but evidently it did the trick…eventually.

9

Well, if you are not yet using Github Actions (GA) and Continuous Integration, I would highly recommend it. I really regret that I didn’t know that before. With GA, you are able to check and build your package on any OS (Windows, Mac, etc) any time you want and for free of course. Moreover, with the drat package, you can put all your binaries in a github repository and people can install them via the standard install.packages command as if they were on CRAN.

In the past, I was working on Windows and got some trouble when checking/building the rstantools-package on Mac or vice-versa. Now, with GA, I just need to make a git push and the package is checked and built on Windows and Mac on the cloud for me, that’s wonderful !

10

Thanks, this does sound very appealing. GitHub Actions and drat are both new to me, and I suspect they will take a while to figure out, but the package at issue here is unlikely ever to be on CRAN. Since precompiled models were what originally sold me on the rstantools framework (I even refactored this package from an earlier version where models compiled on the fly), it would be nice to actually offer users a compile-free installation.

11

If you have to figure out everything (as I did some weeks ago), it takes a while indeed but in this github issue, I have explained the main steps to build binary packages via Github Actions for Windows and Mac and insert them into a drat repository so that people can directly install the pre-compiled binary packages via the standard install.packages command.

If there is something unclear, feel free to ask :)

12

Turns out I spiked the ball prematurely. Things were fine for a while after that initial Clean and Rebuild, but this is a recurring problem. I’ll try to summarize what I’ve learned, with the disclaimer that I am way out of my depth when it comes to C++ compilers, R package internals, etc., so I may not be understanding or explaining things correctly. Feel free to ask for clarification if that is the case. Thanks in advance for reading a long post and for any advice you can offer.

My working hypothesis is that the root of the problem is that current versions of R / Rtools / RcppEigen cause the compiler to emit a spectacularly verbose stream of warnings, thousands of lines long, where this was not the case with previous versions. For example, some colleagues encountered a similar issue when compiling an R package that depends on TMB (more on this in a sec). As in their case, most (though not all) of the warnings that I get when compiling my package are -Wignored-attributes.

In my case, these warnings are now invariably emitted (to the R console) when I do the third and final step of the rstantools Getting Started workflow, i.e. devtools::install(upgrade = FALSE). However, sometimes these warnings, or at least superficially similar ones, are emitted when I call pkgbuild::compile_dll() or even roxygen2::roxygenize(). I am not 100% certain that they are the exact same warnings as in the former case, but -Wignored-attributes does feature prominently, as does the aptly named -Wpedantic. The most conspicuous difference is that the warnings printed by the latter two function calls have weird formatting (bold, purple, turquoise). I don’t know the significance of that, if any.

When the warnings come from install(), they do not prevent installation from completing successfully. However, when they come from compile_dll() or roxygenize(), they eventually terminate in the error message shown in the OP. I emphasize again that when things are working properly, compile_dll() is silent and virtually instantaneous. My admittedly vague impression is that the warnings have somehow been “pushed up” in the chain of commands to a point where they cannot be handled correctly, but this may be pure superstition.

I have not been able to discern any rhyme or reason as to why compile_dll() sometimes emits these warnings and fails. As mentioned in the OP, the first such incident happened when the package source had not changed whatsoever since a previous successful install. Subsequent occurrences have come (inconsistently) after editing Stan code or even just R code. But once the first such failure happens, it always recurs until it is “cleared” (see below). Furthermore, as I mentioned, the error in Focal Package then “infects” my other Stan-dependent packages if I try to compile them – even the toy rstanlm example. It’s as if the “state” of the tools themselves has somehow changed.

Now, you’d think that if warnings are the problem, adding flags to Makevars.win to suppress those warnings would fix it, but I actually had -Wno-unused-attributes as part of CXXFLAGS, CXX11FLAGS and CXX14FLAGS in my Makevars.win already. I believe these used to work, but apparently no longer. (Or maybe it’s just that previous R / Rtools / RcppEigen versions didn’t emit all these warnings to begin with.) I also tried a more radical suggestion from one of the aforementioned colleagues working with TMB, to edit R’s system Makeconf file to remove -Wall. That also had no effect on the behavior in my case. Finally, I tried passing minimal flags (just -O3) using pkgbuild::with_debug(), but that didn’t work either.

The only thing I have found that will “clear” the compile_dll() error once it has started is the Clean and Rebuild button in RStudio. This has worked 100% of the time so far, but unfortunately it is not a viable solution because it turns out the aforementioned 15-hour compile time is far from the worst-case scenario. In one instance it took FOUR AND A HALF DAYS. I wish I were kidding. I resisted the temptation to kill it, and at the end I had a successful install. Apparently this is an RStudio bug that prevents the Build pane from displaying the thousands of lines of warnings in a timely manner – meaning the warnings themselves, rather than anything about the actual compile, are the problem. This strikes me as totally mindblowing, pathological behavior, but then I don’t know much about 'puters. Things were good until maybe the third or fourth iteration through the rstantools Getting Started Vignette workflow, when the error happened again.

Perhaps, then, an R command-line equivalent of Clean and Rebuild would print the warnings to the console, just like devtools::install() does, and therefore might not take forever. According to this, all of the RStudio Build buttons have devtools equivalents…except Clean and Rebuild. That button executes

R CMD INSTALL --preclean --no-multiarch --with-keep.source <pkgNameGoesHere>

I am not familiar with running R from the shell or with R CMD INSTALL, but I tried to figure it out. No luck. I’ve seen some of the Stan-devs say that they use this instead of devtools, but I haven’t seen a step-by-step, plain-English description of how to do it. This might be a useful addition to the Getting Started vignette. I also tried pkgbuild::clean_dll(), which didn’t seem to have any effect.

To my surprise, devtools::install() still works, even after compile_dll() and roxygenize() start choking. I don’t know the implications of skipping to the last step in the workflow, though; I’ve only ever done it in the prescribed order. My guess is it’s problematic, and anyway I’ll have to recompile the doc sooner or later.

I guess my question is basically “Help!”, but I’ll try to narrow it down:

  • Do you have any idea why the behavior I’m describing might be happening? Is my hypothesis about the newly verbose compiler warnings plausible, or is that a red herring?

  • Do you know of any way besides the obvious (and not-so-obvious) methods I’ve mentioned to suppress the newly verbose compiler warnings, or why the obvious isn’t working?

  • Is there a devtools-based approach that might accomplish the “reset” of compile_dll() once it’s borked, similar to Clean and Rebuild but not glacially slow?

  • If not, can you give me a simple step-by-step description of how to do the R CMD INSTALL way, under different scenarios (what if I’ve only changed the R code; only the documentation; only the Stan code; etc.), starting from “open a terminal window”? Explain like I’m 5.

Again, thanks!

13

OK, I’ll be more specific about this part in hopes that someone might have some ideas. Karl Broman’s R package primer says

If I do this as written, I get

Buhle@Buhle-W541 MINGW64 ~/My Application Data/R/salmonIPM (covariates-formula)
$ R CMD build salmonIPM
 ERROR
cannot change to directory 'salmonIPM'

despite the fact that I am already in that directory, as the line immediately above indicates. After some trial and error I figured out that I have to type the full path, with the directory with spaces enclosed in quotes.

[EDIT: Note to future readers (or my future self), the problem here was my reading comprehension. Karl even italicized it, but I read it as the package directory rather than the directory above it. Blame it on computer fatigue. However, after correcting this, the error below still happens.]

$ R CMD build ~/"My Application Data"/R/salmonIPM
* checking for file 'C:/Users/Buhle/My Application Data/R/salmonIPM/DESCRIPTION'
 ... OK
* preparing 'salmonIPM':
* checking DESCRIPTION meta-information ... OK
* cleaning src
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
* building 'salmonIPM_1.0.2.9000.tar.gz'
Warning: file 'salmonIPM/configure' did not have execute permissions: corrected

Success! I do indeed have a .tar.gz file in my directory. Now I do

$ R CMD INSTALL --preclean --no-multiarch --with-keep.source salmonIPM_1.0.2.90
00.tar.gz

First I get the usual and accustomed series of Stan diagnostics about possible nonlinear transforms, which always occur and I know are not a problem:

* installing to library 'C:/R/R-4.0.5/library'
* installing *source* package 'salmonIPM' ...
** using staged installation
DIAGNOSTIC(S) FROM PARSER:
Info:
Left-hand side of sampling statement (~) may contain a non-linear transform of a
 parameter or local variable.
If it does, you need to include a target += statement with the log absolute dete
rminant of the Jacobian of the transform.
Left-hand-side of sampling statement:
    logit(s_D[which_prior_D]) ~ normal(...)

[truncated]

But then I get this:

** libs
/usr/bin/rm: missing operand
Try '/usr/bin/rm --help' for more information.
make: *** [C:/R/R-4.0.5/share/make/winshlib.mk:28: shlib-clean] Error 1
sh-4.4$

which dumps me out into this sh-4.4$ prompt. I have no idea what this means and exit doesn’t work to get me out of it, so I just kill the terminal window.

Needless to say, this does not happen when I click the Clean and Rebuild button in the RStudio Build pane. The result in that case, if one has zen-like patience, is a successful install. So is the claim in that StackOverflow comment that Clean and Rebuild is equivalent to

R CMD INSTALL --preclean --no-multiarch --with-keep.source <pkgNameGoesHere>

incorrect? (I mean, apart from the fact that it’s not the package name but the package name appended by a string of numbers and .tar.gz.)

14

In trying to figure out why the devtools functions as well as the RStudio Build pane buttons seem to be ignoring the instructions in my Makevars.win such as -Wno-unused-attributes, I discovered something unexpected:

> tools::makevars_user()
character(0)

> tools::makevars_site()
character(0)

And yet:

> file.exists(file.path(Sys.getenv('HOME'), '.R', 'Makevars.win'))
[1] TRUE

which of course it does, because I’m looking at it in an editor window right now.

Question:

Does anyone know why the functions are telling me my user- or site-specific Makevars file does not exist? Is this an indication that I need to tell R the path to Makevars.win (and if so, how do I do that, as I have never seen such a requirement before) or is it a bug in the tools functions that I should disregard?

Thanks.

15

Well, amazingly enough that led me to what appears to be (for now, at least) a solution!

Turns out I had a Windows environment variable R_MAKEVARS_USER, which I forgot I had created in order to resolve one of the many previous bewildering Stan / C++ roadblocks over the years. The problem was, that variable was set to Makevars, not Makevars.win (I changed the name at some point to be more compliant with the Stan development guidelines). After updating it, R now respects my compiler flags, including -Wno-unused-attributes, and the normal workflow of

pkgbuild::compile_dll()
roxygen2::roxygenize()
devtools::install(upgrade = FALSE)

works once again! The compile_dll() call is still slower and more verbose than it used to be, but I’ll take a 10-min wait over a 4-day wait anytime. At least this allows me to proceed. Perhaps the R_MAKEVARS_USER environment variable is no longer needed at all; I’ll try deleting it.

Thanks for reading, if you have, and I hope this thread proves useful to someone in the future. I also hope I don’t come to regret marking this as a solution, again.

1 Like
16

Thanks for this thread. It was really useful, and saved me a lot of time as I had the same problem. Incidentally, I was able to run the command line install in the R console using:

callr::rcmd("R CMD INSTALL --preclean --no-multiarch --with-keep-source E:/Routines/MyPackage ")

which seemed to be pretty fast with no output.

17

For future reference as well, this can also be resolved by adding:

PKG_CXXFLAGS += -Wa,-mbig-obj

To the package’s Makevars file. For packages using rstantools-based installation, this can be implemented by adding the following to the bottom of the package’s configure and configure.win scripts:

configure:

# Add configuration needed for big object files
echo "PKG_CXXFLAGS += -Wa,-mbig-obj" >> ./src/Makevars

configure.win:

# Add configuration needed for big object files
echo "PKG_CXXFLAGS += -Wa,-mbig-obj" >> ./src/Makevars.win
2 Likes
18

@paul, delighted to hear my tedious novella was useful to at least two other people. Makes me feel less embarrassed for having written it.

@andrjohns, thank you so much for this fix! This came up at the end of the other thread too, but it wasn’t clear to me how to permanently edit the package Makevars and I didn’t follow up. I tried it, and it works! Without the extra user Makevars flags it does still puke out a lot of warnings to the console, and (as a result?) takes ~ 15% longer in my case. But the important thing is it compiles successfully, and this way I don’t have to instruct users to add all those -Wno flags, which is obviously much better.

2 Likes
19

@andrjohns when implementing this change in an R package that uses rstantools and submitting to CRAN, CRAN sends it back due to the NOTE: “Compilation used the following non-portable flag(s): ‘-Wa,-mbig-obj’”.

The internet seems to suggest adding this to one’s personal ~/.R/Makevars file, but will not including these flags cause problems for others trying to install the package? Is there an easier way to get CRAN to go along with the edited configure files?

20

If you remove the flags and resubmit it should be fine, the file too big errors only seem to pop up with rstudio and local development on windows (anecdotally)