CmdStanR - install_cmdstan() not finishing build of cmdstan

mattwilliamson13 · November 15, 2019, 1:04am

Yes

yields:

clang: start
CFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
CCFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
CXXFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
CPPFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include

SHLIB_CXXLDFLAGS+=-Wl,-rpath,${R_HOME}/lib ${R_HOME}/lib/libc++abi.1.dylib
clang: end

yields:

# clang: start
PATH="/usr/local/clang7/bin:${PATH}"
# clang: end

R version 3.6.0 (2019-04-26)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS  10.15

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base     

other attached packages:
 [1] rstan_2.19.2          StanHeaders_2.18.1-10 forcats_0.4.0        
 [4] stringr_1.4.0         dplyr_0.8.3           purrr_0.3.2          
 [7] readr_1.3.1           tidyr_1.0.0           tibble_2.1.3         
[10] ggplot2_3.2.0         tidyverse_1.2.1      

loaded via a namespace (and not attached):
 [1] httr_1.4.1         jsonlite_1.6       splines_3.6.0     
 [4] foreach_1.4.4      prodlim_2019.10.13 modelr_0.1.4      
 [7] assertthat_0.2.1   stats4_3.6.0       cellranger_1.1.0  
[10] ipred_0.9-9        pillar_1.4.2       backports_1.1.5   
[13] lattice_0.20-38    glue_1.3.1         rvest_0.3.4       
[16] colorspace_1.4-1   recipes_0.1.7      Matrix_1.2-17     
[19] plyr_1.8.4         timeDate_3043.102  pkgconfig_2.0.3   
[22] broom_0.5.2        haven_2.1.0        caret_6.0-84      
[25] mvtnorm_1.0-11     scales_1.0.0       processx_3.4.1    
[28] gower_0.2.1        lava_1.6.6         generics_0.0.2    
[31] withr_2.1.2        nnet_7.3-12        lazyeval_0.2.2    
[34] cli_1.1.0          survival_2.44-1.1  magrittr_1.5      
[37] crayon_1.3.4       readxl_1.3.1       ps_1.3.0          
[40] nlme_3.1-139       MASS_7.3-51.4      xml2_1.2.1        
[43] class_7.3-15       pkgbuild_1.0.4     tools_3.6.0       
[46] loo_2.1.0          data.table_1.12.2  prettyunits_1.0.2 
[49] hms_0.4.2          lifecycle_0.1.0    matrixStats_0.54.0
[52] munsell_0.5.0      callr_3.3.1        compiler_3.6.0    
[55] rlang_0.4.1        grid_3.6.0         iterators_1.0.10  
[58] rstudioapi_0.10    rethinking_1.88    gtable_0.3.0      
[61] ModelMetrics_1.2.2 codetools_0.2-16   inline_0.3.15     
[64] reshape2_1.4.3     R6_2.4.1           gridExtra_2.3     
[67] lubridate_1.7.4    zeallot_0.1.0      stringi_1.4.3     
[70] parallel_3.6.0     Rcpp_1.0.2         vctrs_0.2.0       
[73] rpart_4.1-15       tidyselect_0.2.5   coda_0.19-3

jonah · November 15, 2019, 1:21am

Thanks for starting a new topic. Deleting ~/.Renviron is what got cmdstanr::install_cmdstan() working for @ssp3nc3r

perhaps because of the clang 7 path in there. But I don’t know if doing that would mess up other things (e.g. rstan).

coatless · November 15, 2019, 1:46am

So, I can confirm that the make file used by install_cmdstan() is cycling compilations, but only with stan/src/stan/lang/grammars files.

* Installing CmdStan 2.21.0 in /Users/ronin/.cmdstanr
* Downloading cmdstan-2.21.0.tar.gz from GitHub.
* Download complete.
* Unpacking archive ...
* Building CmdStan binaries.
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT      -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS      -c -MT src/cmdstan/stanc.o -MM -E -MG -MP -MF src/cmdstan/stanc.d src/cmdstan/stanc.cpp
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT      -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS      -c -MT bin/cmdstan/lang/ast_def.o -MT stan/src/stan/lang/ast_def.d -MM -E -MG -MP -MF stan/src/stan/lang/ast_def.d stan/src/stan/lang/ast_def.cpp
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT      -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS      -c -MT bin/cmdstan/lang/grammars/semantic_actions_def.o -MT stan/src/stan/lang/grammars/semantic_actions_def.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/semantic_actions_def.d stan/src/stan/lang/grammars/semantic_actions_def.cpp
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT      -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS      -c -MT bin/cmdstan/lang/grammars/statement_grammar_inst.o -MT stan/src/stan/lang/grammars/statement_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/statement_grammar_inst.d stan/src/stan/lang/grammars/statement_grammar_inst.cpp

I’m wondering if its related to the bug previously encountered with clang on TravisCI?

The clang version shipping on Catalina is Clang 11 vs. R’s clang7.

Apple clang version 11.0.0 (clang-1100.0.33.8)
Target: x86_64-apple-darwin19.0.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Relevant portion of: inst/make_cmdstan.sh

github.com

stan-dev/cmdstanr/blob/6ed71d0f456bb008b661a9f4c79689baa1fb2578/inst/make_cmdstan.sh#L81-L87


build_cmdstan() {
    if [[ ${WIN} -ne 0 ]]; then
        mingw32-make -j${JOBS} build
    else
        make -j${JOBS} build
    fi
}

From there, the makefile shipping with cmdstan seems to be handling the stan/src/stan/lang/grammars/* here:

github.com

stan-dev/cmdstan/blob/1834691a90e709011cb6f11def7a55047b65cdd4/make/stanc#L45


endif
else
bin/stanc$(EXE) : $(shell find $(STANC3)/src/ -type f -name '*.ml*') $(STANC#)
	@mkdir -p $(dir $@)
	cd $(STANC3) && echo "--- Rebuilding stanc ---\n" && dune build @install
	cp $(STANC3)/_build/default/src/stanc/stanc.exe $@
endif


# libstanc build rules
# just used by cmdstan tests and should soon be replaced with tests based on stanc$(EXE)
STANC_TEMPLATE_INSTANTIATION_CPP := $(shell find $(STAN)src/stan/lang -type f -name '*_inst.cpp') $(shell find $(STAN)src/stan/lang -type f -name '*_def.cpp')
STANC_TEMPLATE_INSTANTIATION := $(STANC_TEMPLATE_INSTANTIATION_CPP:$(STAN)src/stan/%.cpp=bin/cmdstan/%.o)


bin/cmdstan/libstanc.a : $(STANC_TEMPLATE_INSTANTIATION)
	@mkdir -p $(dir $@)
	$(AR) -rs $@ $^


$(STANC_TEMPLATE_INSTANTIATION) : O = $(O_STANC)
$(STANC_TEMPLATE_INSTANTIATION) : bin/cmdstan/%.o : $(STAN)src/stan/%.cpp
	@mkdir -p $(dir $@)
	$(COMPILE.cpp) $(OUTPUT_OPTION) $<

jonah · November 15, 2019, 2:10am

Yeah this does seem similar to that issue from @rok_cesnovar.

@coatless thanks for helping with this! since the Apple clang 11 seems to work for CmdStanR, could we just have CmdStanR set the PATH environment variable?(i.e., overriding .Renviron but then changing it back when it finishes)

mattwilliamson13 · November 15, 2019, 2:12am

I can confirm that deleting ~/.Renviron allowed the build to complete and models are now compiling successfully.

coatless · November 15, 2019, 2:12am

I’m not sure how viable of a solution that would be if the goal is for having cmdstanr being on CRAN.

jonah · November 15, 2019, 2:24am

Good point!

rok_cesnovar · November 15, 2019, 8:23am

This does seems similar to that Travis issue. We never got around to fixing it, because we opted to exclude Travis in favor of doing all tests on Jenkins instead.

One thing worth trying is to install the current develop version of cmdstan. I think this latest PR from @mitzimorris could fix this issue. Not 100% sure, but worth a try.

In order to install from github call install_cmdstan(repo_clone = TRUE) (see https://mc-stan.org/cmdstanr/reference/install_cmdstan.html)

mitzimorris · November 15, 2019, 2:37pm

I’m not sure either - it would be worth a try!

ericnovik · April 1, 2020, 3:13am

After I deleted the ~/.Renviron the compilation worked fine but when I tried compiling a test program, I go the following error message:

> mod <- cmdstan_model("bern.stan")
Compiling Stan program...
error: PCH file built from a different branch ((clang-1103.0.32.29)) than the compiler ((tags/RELEASE_700/final))
1 error generated.
make: *** [/var/folders/rq/rz57m_px0sscbvx56fx18k5w0000gn/T/RtmpuuUqsW/model-5c7951ecc530] Error 1
Error in processx::run(command = make_cmd(), args = c(tmp_exe, include_paths,  : 
  System command 'make' failed, exit status: 2, stderr:
E> error: PCH file built from a different branch ((clang-1103.0.32.29)) than the compiler ((tags/RELEASE_700/final))
E> 1 error generated.
E> make: *** [/var/folders/rq/rz57m_px0sscbvx56fx18k5w0000gn/T/RtmpuuUqsW/model-5c7951ecc530] Error 1
Type .Last.error.trace to see where the error occured

I am on Apple clang version 11.0.3 (clang-1103.0.32.29)

ericnovik · April 1, 2020, 3:49am

Problem solved. Before running the above I restored ~/.Renviron and that was apparently causing the problem. After removing it (again) the compilation succeeded.

rok_cesnovar · April 1, 2020, 7:15am

Great that you solved it!

Just to follow up, from the looks of it I would say that the previous ~/.Renviron enforced the use of another version of clang? Is that possible?

I am guessing there was a leftover model_header PCH file that was build using the older version of Clang. This issue would also be solved if you would clean and rebuild the cmdstan (make clean-all`` and make build`). We need to expose this cleanup feature via a R command anyways (cmdstan_clean_and_rebuild() or something along those lines).

ericnovik · April 1, 2020, 1:04pm

Yeah, .Renviron had: PATH="/usr/local/clang7/bin:${PATH}"

bbbales2 · April 16, 2020, 10:25am

* Building CmdStan binaries.
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT      -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS      -c -MT src/cmdstan/stanc.o -MM -E -MG -MP -MF src/cmdstan/stanc.d src/cmdstan/stanc.cpp
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT      -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS      -c -MT bin/cmdstan/lang/ast_def.o -MT stan/src/stan/lang/ast_def.d -MM -E -MG -MP -MF stan/src/stan/lang/ast_def.d stan/src/stan/lang/ast_def.cpp
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT      -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS      -c -MT bin/cmdstan/lang/grammars/semantic_actions_def.o -MT stan/src/stan/lang/grammars/semantic_actions_def.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/semantic_actions_def.d stan/src/stan/lang/grammars/semantic_actions_def.cpp
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT      -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.69.0 -I stan/lib/stan_math/lib/sundials_4.1.0/include    -DBOOST_DISABLE_ASSERTS      -c -MT bin/cmdstan/lang/grammars/statement_grammar_inst.o -MT stan/src/stan/lang/grammars/statement_grammar_inst.d -MM -E -MG -MP -MF stan/src/stan/lang/grammars/statement_grammar_inst.d stan/src/stan/lang/grammars/statement_grammar_inst.cpp

Aren’t these files part of stanc2? Why would they get built anyway? Is this a makefile bug?

Edit: whoops those questions are for @mitzimorris.

rok_cesnovar · April 16, 2020, 10:31am

Because we still build stanc2. Its used for debugging.
make STANC2=true model uses bin/stanc2.

mitzimorris · April 16, 2020, 1:14pm

what Rok said - stanc2 is available as a way to check/workaround failures of the new compiler.
but I thought that there was another reason as well - some other kind of debugging or error messages?

rok_cesnovar · April 16, 2020, 1:16pm

Its still used for cmdstan interface tests, which isnt ideal to say the least.

mitzimorris · April 16, 2020, 1:33pm

that might have been the case last fall, when the stanc3 compiler wasn’t the default, but is this still needed?

rok_cesnovar · April 16, 2020, 1:46pm

It isnt needed (we can switch it). Currently libstanc.a is being used here https://github.com/stan-dev/cmdstan/blob/develop/make/stanc#L107

bbbales2 · April 17, 2020, 5:30pm

As a two cents, I understand .Renviron is part of the R installs on Mac OS X, so I don’t like removing it as the standard thing to do. The easiest thing to do might be to not build stanc2 in cmdstanr and sidestep the issue. That’s my motivation for asking about stanc2 here.