CmdStanR - install_cmdstan() not finishing build of cmdstan

Yes

yields:

clang: start
CFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
CCFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
CXXFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
CPPFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include

SHLIB_CXXLDFLAGS+=-Wl,-rpath,${R_HOME}/lib ${R_HOME}/lib/libc++abi.1.dylib
clang: end

yields:

# clang: start
PATH="/usr/local/clang7/bin:${PATH}"
# clang: end
R version 3.6.0 (2019-04-26)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS  10.15

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base     

other attached packages:
 [1] rstan_2.19.2          StanHeaders_2.18.1-10 forcats_0.4.0        
 [4] stringr_1.4.0         dplyr_0.8.3           purrr_0.3.2          
 [7] readr_1.3.1           tidyr_1.0.0           tibble_2.1.3         
[10] ggplot2_3.2.0         tidyverse_1.2.1      

loaded via a namespace (and not attached):
 [1] httr_1.4.1         jsonlite_1.6       splines_3.6.0     
 [4] foreach_1.4.4      prodlim_2019.10.13 modelr_0.1.4      
 [7] assertthat_0.2.1   stats4_3.6.0       cellranger_1.1.0  
[10] ipred_0.9-9        pillar_1.4.2       backports_1.1.5   
[13] lattice_0.20-38    glue_1.3.1         rvest_0.3.4       
[16] colorspace_1.4-1   recipes_0.1.7      Matrix_1.2-17     
[19] plyr_1.8.4         timeDate_3043.102  pkgconfig_2.0.3   
[22] broom_0.5.2        haven_2.1.0        caret_6.0-84      
[25] mvtnorm_1.0-11     scales_1.0.0       processx_3.4.1    
[28] gower_0.2.1        lava_1.6.6         generics_0.0.2    
[31] withr_2.1.2        nnet_7.3-12        lazyeval_0.2.2    
[34] cli_1.1.0          survival_2.44-1.1  magrittr_1.5      
[37] crayon_1.3.4       readxl_1.3.1       ps_1.3.0          
[40] nlme_3.1-139       MASS_7.3-51.4      xml2_1.2.1        
[43] class_7.3-15       pkgbuild_1.0.4     tools_3.6.0       
[46] loo_2.1.0          data.table_1.12.2  prettyunits_1.0.2 
[49] hms_0.4.2          lifecycle_0.1.0    matrixStats_0.54.0
[52] munsell_0.5.0      callr_3.3.1        compiler_3.6.0    
[55] rlang_0.4.1        grid_3.6.0         iterators_1.0.10  
[58] rstudioapi_0.10    rethinking_1.88    gtable_0.3.0      
[61] ModelMetrics_1.2.2 codetools_0.2-16   inline_0.3.15     
[64] reshape2_1.4.3     R6_2.4.1           gridExtra_2.3     
[67] lubridate_1.7.4    zeallot_0.1.0      stringi_1.4.3     
[70] parallel_3.6.0     Rcpp_1.0.2         vctrs_0.2.0       
[73] rpart_4.1-15       tidyselect_0.2.5   coda_0.19-3

Thanks for starting a new topic. Deleting ~/.Renviron is what got cmdstanr::install_cmdstan() working for @ssp3nc3r

perhaps because of the clang 7 path in there. But I don’t know if doing that would mess up other things (e.g. rstan).

So, I can confirm that the make file used by install_cmdstan() is cycling compilations, but only with stan/src/stan/lang/grammars files.

* Installing CmdStan 2.21.0 in /Users/ronin/.cmdstanr
* Downloading cmdstan-2.21.0.tar.gz from GitHub.
* Download complete.
* Unpacking archive ...
* Building CmdStan binaries.
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT      -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS      -c -MT src/cmdstan/stanc.o -MM -E -MG -MP -MF src/cmdstan/stanc.d src/cmdstan/stanc.cpp
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT      -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS      -c -MT bin/cmdstan/lang/ast_def.o -MT stan/src/stan/lang/ast_def.d -MM -E -MG -MP -MF stan/src/stan/lang/ast_def.d stan/src/stan/lang/ast_def.cpp
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT      -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS      -c -MT bin/cmdstan/lang/grammars/semantic_actions_def.o -MT stan/src/stan/lang/grammars/semantic_actions_def.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/semantic_actions_def.d stan/src/stan/lang/grammars/semantic_actions_def.cpp
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT      -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS      -c -MT bin/cmdstan/lang/grammars/statement_grammar_inst.o -MT stan/src/stan/lang/grammars/statement_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/statement_grammar_inst.d stan/src/stan/lang/grammars/statement_grammar_inst.cpp

I’m wondering if its related to the bug previously encountered with clang on TravisCI?

The clang version shipping on Catalina is Clang 11 vs. R’s clang7.

Apple clang version 11.0.0 (clang-1100.0.33.8)
Target: x86_64-apple-darwin19.0.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Relevant portion of: inst/make_cmdstan.sh

From there, the makefile shipping with cmdstan seems to be handling the stan/src/stan/lang/grammars/* here:

1 Like

Yeah this does seem similar to that issue from @rok_cesnovar.

@coatless thanks for helping with this! since the Apple clang 11 seems to work for CmdStanR, could we just have CmdStanR set the PATH environment variable?(i.e., overriding .Renviron but then changing it back when it finishes)

I can confirm that deleting ~/.Renviron allowed the build to complete and models are now compiling successfully.

2 Likes

I’m not sure how viable of a solution that would be if the goal is for having cmdstanr being on CRAN.

Good point!

This does seems similar to that Travis issue. We never got around to fixing it, because we opted to exclude Travis in favor of doing all tests on Jenkins instead.

One thing worth trying is to install the current develop version of cmdstan. I think this latest PR from @mitzimorris could fix this issue. Not 100% sure, but worth a try.

In order to install from github call install_cmdstan(repo_clone = TRUE) (see https://mc-stan.org/cmdstanr/reference/install_cmdstan.html)

2 Likes

I’m not sure either - it would be worth a try!

After I deleted the ~/.Renviron the compilation worked fine but when I tried compiling a test program, I go the following error message:

> mod <- cmdstan_model("bern.stan")
Compiling Stan program...
error: PCH file built from a different branch ((clang-1103.0.32.29)) than the compiler ((tags/RELEASE_700/final))
1 error generated.
make: *** [/var/folders/rq/rz57m_px0sscbvx56fx18k5w0000gn/T/RtmpuuUqsW/model-5c7951ecc530] Error 1
Error in processx::run(command = make_cmd(), args = c(tmp_exe, include_paths,  : 
  System command 'make' failed, exit status: 2, stderr:
E> error: PCH file built from a different branch ((clang-1103.0.32.29)) than the compiler ((tags/RELEASE_700/final))
E> 1 error generated.
E> make: *** [/var/folders/rq/rz57m_px0sscbvx56fx18k5w0000gn/T/RtmpuuUqsW/model-5c7951ecc530] Error 1
Type .Last.error.trace to see where the error occured

I am on Apple clang version 11.0.3 (clang-1103.0.32.29)

Problem solved. Before running the above I restored ~/.Renviron and that was apparently causing the problem. After removing it (again) the compilation succeeded.

1 Like

Great that you solved it!

Just to follow up, from the looks of it I would say that the previous ~/.Renviron enforced the use of another version of clang? Is that possible?

I am guessing there was a leftover model_header PCH file that was build using the older version of Clang. This issue would also be solved if you would clean and rebuild the cmdstan (make clean-all`` and make build`). We need to expose this cleanup feature via a R command anyways (cmdstan_clean_and_rebuild() or something along those lines).

Yeah, .Renviron had: PATH="/usr/local/clang7/bin:${PATH}"

* Building CmdStan binaries.
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT      -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS      -c -MT src/cmdstan/stanc.o -MM -E -MG -MP -MF src/cmdstan/stanc.d src/cmdstan/stanc.cpp
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT      -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS      -c -MT bin/cmdstan/lang/ast_def.o -MT stan/src/stan/lang/ast_def.d -MM -E -MG -MP -MF stan/src/stan/lang/ast_def.d stan/src/stan/lang/ast_def.cpp
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT      -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS      -c -MT bin/cmdstan/lang/grammars/semantic_actions_def.o -MT stan/src/stan/lang/grammars/semantic_actions_def.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/semantic_actions_def.d stan/src/stan/lang/grammars/semantic_actions_def.cpp
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT      -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS      -c -MT bin/cmdstan/lang/grammars/statement_grammar_inst.o -MT stan/src/stan/lang/grammars/statement_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/statement_grammar_inst.d stan/src/stan/lang/grammars/statement_grammar_inst.cpp

Aren’t these files part of stanc2? Why would they get built anyway? Is this a makefile bug?

Edit: whoops those questions are for @mitzimorris.

Because we still build stanc2. Its used for debugging.
make STANC2=true model uses bin/stanc2.

what Rok said - stanc2 is available as a way to check/workaround failures of the new compiler.
but I thought that there was another reason as well - some other kind of debugging or error messages?

Its still used for cmdstan interface tests, which isnt ideal to say the least.

that might have been the case last fall, when the stanc3 compiler wasn’t the default, but is this still needed?

It isnt needed (we can switch it). Currently libstanc.a is being used here https://github.com/stan-dev/cmdstan/blob/develop/make/stanc#L107

1 Like

As a two cents, I understand .Renviron is part of the R installs on Mac OS X, so I don’t like removing it as the standard thing to do. The easiest thing to do might be to not build stanc2 in cmdstanr and sidestep the issue. That’s my motivation for asking about stanc2 here.