Turns out I spiked the ball prematurely. Things were fine for a while after that initial Clean and Rebuild, but this is a recurring problem. I’ll try to summarize what I’ve learned, with the disclaimer that I am way out of my depth when it comes to C++ compilers, R package internals, etc., so I may not be understanding or explaining things correctly. Feel free to ask for clarification if that is the case. Thanks in advance for reading a long post and for any advice you can offer.
My working hypothesis is that the root of the problem is that current versions of R / Rtools / RcppEigen cause the compiler to emit a spectacularly verbose stream of warnings, thousands of lines long, where this was not the case with previous versions. For example, some colleagues encountered a similar issue when compiling an R package that depends on TMB (more on this in a sec). As in their case, most (though not all) of the warnings that I get when compiling my package are -Wignored-attributes.
In my case, these warnings are now invariably emitted (to the R console) when I do the third and final step of the rstantools Getting Started workflow, i.e. devtools::install(upgrade = FALSE). However, sometimes these warnings, or at least superficially similar ones, are emitted when I call pkgbuild::compile_dll() or even roxygen2::roxygenize(). I am not 100% certain that they are the exact same warnings as in the former case, but -Wignored-attributes does feature prominently, as does the aptly named -Wpedantic. The most conspicuous difference is that the warnings printed by the latter two function calls have weird formatting (bold, purple, turquoise). I don’t know the significance of that, if any.
When the warnings come from install(), they do not prevent installation from completing successfully. However, when they come from compile_dll() or roxygenize(), they eventually terminate in the error message shown in the OP. I emphasize again that when things are working properly, compile_dll() is silent and virtually instantaneous. My admittedly vague impression is that the warnings have somehow been “pushed up” in the chain of commands to a point where they cannot be handled correctly, but this may be pure superstition.
I have not been able to discern any rhyme or reason as to why compile_dll() sometimes emits these warnings and fails. As mentioned in the OP, the first such incident happened when the package source had not changed whatsoever since a previous successful install. Subsequent occurrences have come (inconsistently) after editing Stan code or even just R code. But once the first such failure happens, it always recurs until it is “cleared” (see below). Furthermore, as I mentioned, the error in Focal Package then “infects” my other Stan-dependent packages if I try to compile them – even the toy rstanlm example. It’s as if the “state” of the tools themselves has somehow changed.
Now, you’d think that if warnings are the problem, adding flags to Makevars.win to suppress those warnings would fix it, but I actually had -Wno-unused-attributes as part of CXXFLAGS, CXX11FLAGS and CXX14FLAGS in my Makevars.win already. I believe these used to work, but apparently no longer. (Or maybe it’s just that previous R / Rtools / RcppEigen versions didn’t emit all these warnings to begin with.) I also tried a more radical suggestion from one of the aforementioned colleagues working with TMB, to edit R’s system Makeconf file to remove -Wall. That also had no effect on the behavior in my case. Finally, I tried passing minimal flags (just -O3) using pkgbuild::with_debug(), but that didn’t work either.
The only thing I have found that will “clear” the compile_dll() error once it has started is the Clean and Rebuild button in RStudio. This has worked 100% of the time so far, but unfortunately it is not a viable solution because it turns out the aforementioned 15-hour compile time is far from the worst-case scenario. In one instance it took FOUR AND A HALF DAYS. I wish I were kidding. I resisted the temptation to kill it, and at the end I had a successful install. Apparently this is an RStudio bug that prevents the Build pane from displaying the thousands of lines of warnings in a timely manner – meaning the warnings themselves, rather than anything about the actual compile, are the problem. This strikes me as totally mindblowing, pathological behavior, but then I don’t know much about 'puters. Things were good until maybe the third or fourth iteration through the rstantools Getting Started Vignette workflow, when the error happened again.
Perhaps, then, an R command-line equivalent of Clean and Rebuild would print the warnings to the console, just like devtools::install() does, and therefore might not take forever. According to this, all of the RStudio Build buttons have devtools equivalents…except Clean and Rebuild. That button executes
R CMD INSTALL --preclean --no-multiarch --with-keep.source <pkgNameGoesHere>
I am not familiar with running R from the shell or with R CMD INSTALL, but I tried to figure it out. No luck. I’ve seen some of the Stan-devs say that they use this instead of devtools, but I haven’t seen a step-by-step, plain-English description of how to do it. This might be a useful addition to the Getting Started vignette. I also tried pkgbuild::clean_dll(), which didn’t seem to have any effect.
To my surprise, devtools::install() still works, even after compile_dll() and roxygenize() start choking. I don’t know the implications of skipping to the last step in the workflow, though; I’ve only ever done it in the prescribed order. My guess is it’s problematic, and anyway I’ll have to recompile the doc sooner or later.
I guess my question is basically “Help!”, but I’ll try to narrow it down:
-
Do you have any idea why the behavior I’m describing might be happening? Is my hypothesis about the newly verbose compiler warnings plausible, or is that a red herring?
-
Do you know of any way besides the obvious (and not-so-obvious) methods I’ve mentioned to suppress the newly verbose compiler warnings, or why the obvious isn’t working?
-
Is there a devtools-based approach that might accomplish the “reset” of
compile_dll()once it’s borked, similar to Clean and Rebuild but not glacially slow? -
If not, can you give me a simple step-by-step description of how to do the
R CMD INSTALLway, under different scenarios (what if I’ve only changed the R code; only the documentation; only the Stan code; etc.), starting from “open a terminal window”? Explain like I’m 5.
Again, thanks!