Error in rstan_2.16.2 after upgrading to R 3.4.1 with Clang 4.0.0


#21

I installed it and set my Makevars to

CXXFLAGS=-O3 -mtune=native -march=native -Wno-unused-variable -Wno-unused-function
CC=/usr/local/clang4/bin/clang
CXX=/usr/local/clang4/bin/clang++ -arch x86_64 -ftemplate-depth-256
CXX1X=/usr/local/clang4/bin/clang++
CXX98=/usr/local/clang4/bin/clang++
CXX11=/usr/local/clang4/bin/clang++
CXX14=/usr/local/clang4/bin/clang++
CXX17=/usr/local/clang4/bin/clang++
LDFLAGS=-L/usr/local/clang4/lib

I’m running latest rstan from develop branch on GitHub.

A few observations:

  • When I run stan_demo("birats", chains = 1) I get
Iteration:    1 / 2000 [  0%]  (Warmup)
[1] "Error in sampler$call_sampler(args_list[[i]]) : "
[2] "  c++ exception (unknown reason)"                
error occurred during calling the sampler; sampling not done
  • I can successfully run stan_demo("dugongs") sometimes, but other times I get the same as for "birats".

  • If I put the Fibonacci code in test.stan and call rstan::expose_stan_functions("test.stan") I get a lot of output and eventually it errors out:

In file included from file4d8958ca6820.cpp:6:
In file included from /Library/Frameworks/R.framework/Versions/3.4/Resources/library/StanHeaders/include/src/stan/model/standalone_functions_header.hpp:4:
In file included from /Library/Frameworks/R.framework/Versions/3.4/Resources/library/StanHeaders/include/stan/math.hpp:4:
In file included from /Library/Frameworks/R.framework/Versions/3.4/Resources/library/StanHeaders/include/stan/math/rev/mat.hpp:4:
In file included from /Library/Frameworks/R.framework/Versions/3.4/Resources/library/StanHeaders/include/stan/math/rev/core.hpp:12:
In file included from /Library/Frameworks/R.framework/Versions/3.4/Resources/library/StanHeaders/include/stan/math/rev/core/gevv_vvv_vari.hpp:5:
In file included from /Library/Frameworks/R.framework/Versions/3.4/Resources/library/StanHeaders/include/stan/math/rev/core/var.hpp:7:
In file included from /Library/Frameworks/R.framework/Versions/3.4/Resources/library/BH/include/boost/math/tools/config.hpp:13:
In file included from /Library/Frameworks/R.framework/Versions/3.4/Resources/library/BH/include/boost/config.hpp:39:
/Library/Frameworks/R.framework/Versions/3.4/Resources/library/BH/include/boost/config/compiler/clang.hpp:200:11: warning: 'BOOST_NO_CXX11_RVALUE_REFERENCES' macro redefined [-Wmacro-redefined]
#  define BOOST_NO_CXX11_RVALUE_REFERENCES
          ^
<command line>:6:9: note: previous definition is here
#define BOOST_NO_CXX11_RVALUE_REFERENCES 1
        ^
file4d8958ca6820.cpp:13:19: error: target of using declaration conflicts with declaration already in scope
using stan::math::lgamma;
                  ^
/Library/Frameworks/R.framework/Versions/3.4/Resources/library/StanHeaders/include/stan/math/prim/scal/fun/lgamma.hpp:36:19: note: target of using declaration
    inline double lgamma(double x) {
                  ^
/usr/include/math.h:462:15: note: conflicting declaration
extern double lgamma(double);
              ^
file4d8958ca6820.cpp:34:40: error: parameter declarator cannot be qualified
fib(const int& n, std::ostream* &Rcpp::Rcout);
                                 ~~~~~~^
file4d8958ca6820.cpp:58:91: error: call to 'fib' is ambiguous
        return stan::math::promote_scalar<fun_return_scalar_t__>((logical_lte(n,2) ? 1 : (fib((n - 1), pstream__) + fib((n - 2), pstream__)) ));
                                                                                          ^~~
file4d8958ca6820.cpp:34:1: note: candidate function
fib(const int& n, std::ostream* &Rcpp::Rcout);
^
file4d8958ca6820.cpp:38:1: note: candidate function
fib(const int& n, std::ostream* pstream__) {
^
file4d8958ca6820.cpp:58:117: error: call to 'fib' is ambiguous
        return stan::math::promote_scalar<fun_return_scalar_t__>((logical_lte(n,2) ? 1 : (fib((n - 1), pstream__) + fib((n - 2), pstream__)) ));
                                                                                                                    ^~~
file4d8958ca6820.cpp:34:1: note: candidate function
fib(const int& n, std::ostream* &Rcpp::Rcout);
^
file4d8958ca6820.cpp:38:1: note: candidate function
fib(const int& n, std::ostream* pstream__) {
^
file4d8958ca6820.cpp:80:44: error: parameter declarator cannot be qualified
int fib(const int& n, std::ostream*& Rcpp::Rcout);
                                     ~~~~~~^
file4d8958ca6820.cpp:81:56: error: parameter declarator cannot be qualified
RcppExport SEXP sourceCpp_1_fib(SEXP nSEXP, SEXP Rcpp::RcoutSEXP) {
                                                 ~~~~~~^
file4d8958ca6820.cpp:86:77: error: no type named 'RcoutSEXP' in namespace 'Rcpp'
    Rcpp::traits::input_parameter< std::ostream*& >::type Rcpp::Rcout(Rcpp::RcoutSEXP);
                                                                      ~~~~~~^
file4d8958ca6820.cpp:86:65: error: definition or redeclaration of 'Rcout' not allowed inside a function
    Rcpp::traits::input_parameter< std::ostream*& >::type Rcpp::Rcout(Rcpp::RcoutSEXP);
                                                          ~~~~~~^
file4d8958ca6820.cpp:86:70: warning: parentheses were disambiguated as a function declaration [-Wvexing-parse]
    Rcpp::traits::input_parameter< std::ostream*& >::type Rcpp::Rcout(Rcpp::RcoutSEXP);
                                                                     ^~~~~~~~~~~~~~~~~
file4d8958ca6820.cpp:86:71: note: add a pair of parentheses to declare a variable
    Rcpp::traits::input_parameter< std::ostream*& >::type Rcpp::Rcout(Rcpp::RcoutSEXP);
                                                                      ^
                                                                      (
file4d8958ca6820.cpp:87:34: error: no matching function for call to 'fib'
    rcpp_result_gen = Rcpp::wrap(fib(n, Rcpp::Rcout));
                                 ^~~
file4d8958ca6820.cpp:38:1: note: candidate function not viable: no known conversion from 'Rostream<true>' to 'std::ostream *' (aka 'basic_ostream<char> *') for 2nd argument; take the address of the argument with &
fib(const int& n, std::ostream* pstream__) {
^
file4d8958ca6820.cpp:80:5: note: candidate function not viable: no known conversion from 'Rostream<true>' to 'std::ostream *&' (aka 'basic_ostream<char> *&') for 2nd argument
int fib(const int& n, std::ostream*& Rcpp::Rcout);
    ^
file4d8958ca6820.cpp:93:17: error: redefinition of 'sourceCpp_1_fib'
RcppExport SEXP sourceCpp_1_fib(SEXP nSEXP, SEXP pstream__SEXP) {
                ^
file4d8958ca6820.cpp:81:17: note: previous definition is here
RcppExport SEXP sourceCpp_1_fib(SEXP nSEXP, SEXP Rcpp::RcoutSEXP) {
                ^
In file included from file4d8958ca6820.cpp:4:
In file included from /Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include/Rcpp.h:27:
In file included from /Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include/RcppCommon.h:160:
In file included from /Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include/Rcpp/as.h:25:
/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include/Rcpp/internal/Exporter.h:31:28: error: cannot initialize a member subobject of type 'std::__1::basic_ostream<char> *' with an lvalue of type 'SEXP' (aka 'SEXPREC *')
                    Exporter( SEXP x ) : t(x){}
                                         ^ ~
/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include/Rcpp/as.h:87:41: note: in instantiation of member function 'Rcpp::traits::Exporter<std::__1::basic_ostream<char> *>::Exporter' requested here
            ::Rcpp::traits::Exporter<T> exporter(x);
                                        ^
/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include/Rcpp/as.h:152:26: note: in instantiation of function template specialization 'Rcpp::internal::as<std::__1::basic_ostream<char> *>' requested here
        return internal::as<T>(x, typename traits::r_type_traits<T>::r_category());
                         ^
/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include/Rcpp/InputParameter.h:34:38: note: in instantiation of function template specialization 'Rcpp::as<std::__1::basic_ostream<char> *>' requested here
        inline operator T() { return as<T>(x) ; }
                                     ^
file4d8958ca6820.cpp:99:41: note: in instantiation of member function 'Rcpp::InputParameter<std::__1::basic_ostream<char> *>::operator std::__1::basic_ostream<char> *' requested here
    rcpp_result_gen = Rcpp::wrap(fib(n, pstream__));
                                        ^
2 warnings and 11 errors generated.
make: *** [file4d8958ca6820.o] Error 1
/usr/local/clang4/bin/clang++ -arch x86_64 -ftemplate-depth-256 -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include/"  -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/RcppEigen/include/"  -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/RcppEigen/include/unsupported"  -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/rstan/include/boost_not_in_BH" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/BH/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/StanHeaders/include/src/"  -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/StanHeaders/include/"  -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/rstan/include" -DEIGEN_NO_DEBUG  -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -DBOOST_NO_CXX11_RVALUE_REFERENCES  -I"/Library/Frameworks/R.framework/Versions/... <truncated>

 Error in Rcpp::sourceCpp(code = paste(code, collapse = "\n"), ...) : 
  Error 1 occurred building shared library. 
  • If I put the throw_exception Rcpp code in test.cpp and then run Rcpp::sourceCpp("test.cpp") it appears to compile fine. But then if I run throw_exception() I get
Error in throw_exception() : c++ exception (unknown reason)

#22

I suspect that Stan’s sampling algorithms are programmed to stop if an unrecognized exception is thrown, even though in this case it should be recognized. @Bob_Carpenter is there anything we can do on the Stan side to change the fallback behavior to continue sampling?


#23

Can one person with a Mac report this as a bug against the clang build of R for Mac and provide a link to it here? I guess the place to do it is https://stat.ethz.ch/mailman/listinfo/r-sig-mac . The expected behavior when the following file

#include <Rcpp.h>
using namespace Rcpp;

// [[Rcpp::export]]
int throw_exception() {
  std::stringstream errmsg;
  errmsg << "this is an exception";
  throw std::domain_error(errmsg.str());
  return 0;
}

is processed by Rcpp::sourceCpp to is create a function throw_exception that when called in R yields

Error in throw_exception() : this is an exception

But with the clang build of R it seems to say

Error in throw_exception() : c++ exception (unknown reason)

#24

They catch std::exception, but being C++, a function can throw anything as an exception. All the ones we throw come from us, from Boost, or from Eigen, and as far as I know, they all extend std::exception.

If there’s an error during generated quantities, I think we may still not be catching it properly.


#25

Done:

https://stat.ethz.ch/pipermail/r-sig-mac/2017-September/012488.html

although line breaks got removed from the example Rcpp file.


#26

We got a response from Brian Ripley. Here’s the link https://stat.ethz.ch/pipermail/r-sig-mac/2017-September/012490.html to his response. but I’ll copy the response here too:


#27

And a much more thorough response from Kevin Ushey:

https://stat.ethz.ch/pipermail/r-sig-mac/2017-September/012491.html


#28

Can someone who is encountering this problem try adding -nostdinc++ to CXXFLAGS in ~/.R/Makevars in order to avoid the system C++ headers and then add some -isystem directives to use the C++ headers that come with Att’s version of clang4.


#29

Without loading rstan (just with Rcpp and the core libraries loaded) I have two copies of libc++:

matt$ lsof -p 71128 | grep libc++
R 71128 436256 8773504 /usr/lib/libc++abi.dylib
R 71128 1436752 3292488 /usr/lib/libc++.1.dylib
R 71128 862316 11011372 /usr/local/clang4/lib/libc++.1.0.dylib
R 71128 297040 11011376 /usr/local/clang4/lib/libc++abi.1.0.dylib

Once I load the rstan package, I have 3 copies:

matt$ lsof -p 71128 | grep libc++
R 71128 436256 8773504 /usr/lib/libc++abi.dylib
R 71128 1436752 3292488 /usr/lib/libc++.1.dylib
R 71128 862316 11011372 /usr/local/clang4/lib/libc++.1.0.dylib
R 71128 297040 11011376 /usr/local/clang4/lib/libc++abi.1.0.dylib
R 71128 1205156 11014605 /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libc++.1.dylib
R 71128 358888 11014606 /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libc++abi.1.dylib

This might be because I have the CRAN copy of rstan, instead of compiling it from source?

In any case, the minimal Rcpp example reproduces the bug, so the problem is not with Stan.


#30

Adding -nostdinc++ -isystem/usr/local/clang4/include/c++/v1 to CXXFLAGS didn’t appear to make any difference. I think you’d have to recompile R from source to get rid of the dependency. Even that might not work, given Kevin Ushey’s diagnosis of the issue.


#31

OK. Maybe forget about changing ~/.R/Makevars but do Kevin’s hack

source <- "/usr/local/clang4/lib/libc++.1.dylib"
target <- "/usr/lib/libc++.1.dylib"
cmd <- paste(
  "install_name_tool",
  "-change",
  source,
  target,
  file.path("path", "to", "shared", "object")
)
system(cmd)

before running it again.


#32

Perhaps followed by a dyn.unload and a dyn.load of the shared object before attempting to execute it.


#33

What happens if you add -static to CXXFLAGS? Or maybe set LDFLAGS=-Wl,-Bstatic -lc++ -Wl,-Bdynamic?


#34

OK. Can anyone with the broken clang4 compiler tell me if

devtools::install_github("stan-dev/rstanarm", args = "--preclean")
example(pp_check.stanreg, package = "rstanarm")

works?


#35

I get an error during installation:

/usr/local/clang4/bin/clang++ -arch x86_64 -ftemplate-depth-256 -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I"`"/Library/Frameworks/R.framework/Resources/bin/Rscript" --vanilla -e "cat(system.file('include', 'src', package = 'StanHeaders'))"`" -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -DEIGEN_NO_DEBUG -DBOOST_MATH_OVERFLOW_ERROR_POLICY=ignore_error -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/StanHeaders/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/rstan/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/BH/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/RcppEigen/include" -I/usr/local/include   -fPIC  -O3 -mtune=native -march=native -Wno-unused-variable -Wno-unused-function -c Modules.cpp -o Modules.o
/usr/local/clang4/bin/clang++ -arch x86_64 -ftemplate-depth-256 -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I"`"/Library/Frameworks/R.framework/Resources/bin/Rscript" --vanilla -e "cat(system.file('include', 'src', package = 'StanHeaders'))"`" -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -DEIGEN_NO_DEBUG -DBOOST_MATH_OVERFLOW_ERROR_POLICY=ignore_error -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/StanHeaders/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/rstan/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/BH/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.4/Resources/library/RcppEigen/include" -I/usr/local/include   -fPIC  -O3 -mtune=native -march=native -Wno-unused-variable -Wno-unused-function -c init.cpp -o init.o
/usr/local/clang4/bin/clang++ -arch x86_64 -ftemplate-depth-256 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/clang4/lib -o rstanarm.so Modules.o init.o -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
if test -e "/usr/bin/install_name_tool" && test -e "/usr/local/clang4/lib/libc++.1.dylib" && test -e "/usr/lib/libc++.1.dylib"; then /usr/bin/install_name_tool --change /usr/local/clang4/lib/libc++.1.dylib /usr/lib/libc++.1.dylib rstanarm.so; fi
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: more than one input file specified (/usr/local/clang4/lib/libc++.1.dylib and --change)
Usage: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool [-change old new] ... [-rpath old new] ... [-add_rpath new] ... [-delete_rpath old] ... [-id name] input
make: *** [Dirk] Error 1
ERROR: compilation failed for package ‘rstanarm’
* removing ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/rstanarm’
* restoring previous ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/rstanarm’
Installation failed: Command failed (1)

My .R/Makevars file is currently:

CXXFLAGS=-O3 -mtune=native -march=native -Wno-unused-variable -Wno-unused-function
CC=/usr/local/clang4/bin/clang
CXX=/usr/local/clang4/bin/clang++ -arch x86_64 -ftemplate-depth-256
CXX1X=/usr/local/clang4/bin/clang++
CXX98=/usr/local/clang4/bin/clang++
CXX11=/usr/local/clang4/bin/clang++
CXX14=/usr/local/clang4/bin/clang++
CXX17=/usr/local/clang4/bin/clang++
LDFLAGS=-L/usr/local/clang4/lib

#36

Try again. I had --change instead of -change. If that still doesn’t work, make src/Makevars look like Kevin’s hack at
https://github.com/jeroen/V8/issues/37#issuecomment-305266135


#37

It worked when I tried it again. I can install and run the example.


#38

And it is broken if you do install.packages("rstanarm", type = "source")?


#39

That actually seems to work too.


#40

What about the develop version of rstanarm with the XCode compiler?