CmdStan development branch model compilation error on cluster

caesoma · January 8, 2019, 5:45pm

I am trying to run the development branch of cmdstan on a cluster (which runs Red Hat-based Linux), but I am getting errors when trying to compile the stan model into an executable.
The behavior is odd because I am able to clone, build, compile and run the bernoulli example model locally (Debian-based Linux) for the development branch as well as versions 2.18.0 and 2.18.1. On the cluster these two releases also work without problem, but development builds but then fails to compile the executable.

Some additional information/questions:

I think I got the same, or similar, error on a separate cluster (otherwise very similar to the problematic one), but I dismissed it as a mismatch in the -jN flag and the number of cores or something and tried again with a different number or without the flag altogether, I can’t recall precisely; it just went away.
Could some of that have fixed it anyhow, and will this affect the sampling performance?
In the case of a cluster, where the number of cores/CPUs can change according to the job, should stan be built again each time with the corresponding -jN to match the number of cores?

Thanks in advance for the help.

The output of make examples/bernoulli/bernoulli after building CmdStan with make build (no -jN flag) is below:

--- CmdStan v2.18.1 built ---
$ make examples/bernoulli/bernoulli

--- Translating Stan model to C++ code ---
bin/stanc  --o=examples/bernoulli/bernoulli.hpp examples/bernoulli/bernoulli.stan
Model name=bernoulli_model
Input file=examples/bernoulli/bernoulli.stan
Output file=examples/bernoulli/bernoulli.hpp
g++ -std=c++1y -pthread -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.66.0 -I stan/lib/stan_math/lib/sundials_3.1.0/include    -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -DBOOST_PHOENIX_NO_VARIADIC_EXPRESSION     -c -MT examples/bernoulli/bernoulli.o -MT examples/bernoulli/bernoulli -include examples/bernoulli/bernoulli.hpp -include src/cmdstan/main.cpp -MM -E -MG -MP -MF examples/bernoulli/bernoulli.d examples/bernoulli/bernoulli.hpp

--- Linking C++ model ---
g++ -std=c++1y -pthread -Wno-sign-compare     -O3 -I src -I stan/src -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.66.0 -I stan/lib/stan_math/lib/sundials_3.1.0/include    -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -DBOOST_PHOENIX_NO_VARIADIC_EXPRESSION             -include examples/bernoulli/bernoulli.hpp src/cmdstan/main.cpp        stan/lib/stan_math/lib/sundials_3.1.0/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_3.1.0/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_3.1.0/lib/libsundials_idas.a  -o examples/bernoulli/bernoulli
In file included from stan/lib/stan_math/stan/math/rev/mat/fun/ordered_constrain.hpp:6:0,
                 from stan/lib/stan_math/stan/math/rev/mat.hpp:41,
                 from stan/lib/stan_math/stan/math.hpp:4,
                 from stan/src/stan/model/model_header.hpp:4,
                 from ./examples/bernoulli/bernoulli.hpp:3,
                 from <command-line>:0:
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:31:32: error: ‘std::index_sequence’ has not been declared
                           std::index_sequence<I...> i) {
                                ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:31:46: error: expected ‘,’ or ‘...’ before ‘<’ token
                           std::index_sequence<I...> i) {
                                              ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp: In function ‘constexpr auto stan::math::internal::apply(const F&, const Tuple&)’:
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:49:27: error: ‘make_index_sequence’ is not a member of ‘std’
   return apply_impl(f, t, std::make_index_sequence<std::tuple_size<Tuple>{}>{});
                           ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:49:27: note: suggested alternative:
In file included from stan/lib/stan_math/lib/boost_1.66.0/boost/fusion/container/vector/vector.hpp:28:0,
                 from stan/lib/stan_math/lib/boost_1.66.0/boost/fusion/container/vector/vector10.hpp:25,
                 from stan/lib/stan_math/lib/boost_1.66.0/boost/fusion/view/transform_view/transform_view.hpp:22,
                 from stan/lib/stan_math/lib/boost_1.66.0/boost/fusion/algorithm/transformation/transform.hpp:11,
                 from stan/lib/stan_math/lib/boost_1.66.0/boost/fusion/view/zip_view/detail/begin_impl.hpp:14,
                 from stan/lib/stan_math/lib/boost_1.66.0/boost/fusion/view/zip_view/zip_view.hpp:16,
                 from stan/lib/stan_math/lib/boost_1.66.0/boost/fusion/view/zip_view.hpp:12,
                 from stan/lib/stan_math/lib/boost_1.66.0/boost/fusion/include/zip_view.hpp:11,
                 from stan/lib/stan_math/lib/boost_1.66.0/boost/numeric/odeint/util/resize.hpp:26,
                 from stan/lib/stan_math/lib/boost_1.66.0/boost/numeric/odeint/util/state_wrapper.hpp:26,
                 from stan/lib/stan_math/lib/boost_1.66.0/boost/numeric/odeint/util/ublas_wrapper.hpp:33,
                 from stan/lib/stan_math/lib/boost_1.66.0/boost/numeric/odeint.hpp:25,
                 from stan/lib/stan_math/stan/math/prim/arr/functor/integrate_ode_rk45.hpp:17,
                 from stan/lib/stan_math/stan/math/prim/arr.hpp:43,
                 from stan/lib/stan_math/stan/math/prim/mat.hpp:325,
                 from stan/lib/stan_math/stan/math/rev/mat.hpp:12,
                 from stan/lib/stan_math/stan/math.hpp:4,
                 from stan/src/stan/model/model_header.hpp:4,
                 from ./examples/bernoulli/bernoulli.hpp:3,
                 from <command-line>:0:
stan/lib/stan_math/lib/boost_1.66.0/boost/fusion/support/detail/index_sequence.hpp:59:12: note:   ‘boost::fusion::detail::make_index_sequence’
     struct make_index_sequence
            ^
In file included from stan/lib/stan_math/stan/math/rev/mat/fun/ordered_constrain.hpp:6:0,
                 from stan/lib/stan_math/stan/math/rev/mat.hpp:41,
                 from stan/lib/stan_math/stan/math.hpp:4,
                 from stan/src/stan/model/model_header.hpp:4,
                 from ./examples/bernoulli/bernoulli.hpp:3,
                 from <command-line>:0:
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:49:77: error: expected primary-expression before ‘{’ token
   return apply_impl(f, t, std::make_index_sequence<std::tuple_size<Tuple>{}>{});
                                                                             ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:49:77: error: expected ‘)’ before ‘{’ token
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp: At global scope:
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:151:9: error: expected type-specifier
       = std::result_of_t<F(decltype(is_var_), decltype(value_of(Targs()))...)>;
         ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:156:42: error: ‘FReturnType’ was not declared in this scope
   std::array<int, internal::compute_dims<FReturnType>::value> M_;
                                          ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:156:53: error: template argument 1 is invalid
   std::array<int, internal::compute_dims<FReturnType>::value> M_;
                                                     ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:156:61: error: template argument 2 is invalid
   std::array<int, internal::compute_dims<FReturnType>::value> M_;
                                                             ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp: In member function ‘std::vector<stan::math::var> stan::math::adj_jac_vari<F, Targs>::build_return_varis_and_vars(const std::vector<double>&)’:
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:349:9: error: invalid types ‘int[int]’ for array subscript
     M_[0] = val_y.size();
         ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:350:32: error: invalid types ‘int[int]’ for array subscript
     std::vector<var> var_y(M_[0]);
                                ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp: In member function ‘Eigen::Matrix<stan::math::var, R, C> stan::math::adj_jac_vari<F, Targs>::build_return_varis_and_vars(const Eigen::Matrix<double, R, C>&)’:
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:375:9: error: invalid types ‘int[int]’ for array subscript
     M_[0] = val_y.rows();
         ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:376:9: error: invalid types ‘int[int]’ for array subscript
     M_[1] = val_y.cols();
         ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:377:40: error: invalid types ‘int[int]’ for array subscript
     Eigen::Matrix<var, R, C> var_y(M_[0], M_[1]);
                                        ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:377:47: error: invalid types ‘int[int]’ for array subscript
     Eigen::Matrix<var, R, C> var_y(M_[0], M_[1]);
                                               ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp: In member function ‘void stan::math::adj_jac_vari<F, Targs>::chain()’:
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:530:5: error: ‘FReturnType’ was not declared in this scope
     FReturnType y_adj;
     ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:530:17: error: expected ‘;’ before ‘y_adj’
     FReturnType y_adj;
                 ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:532:38: error: ‘y_adj’ was not declared in this scope
     internal::build_y_adj(y_vi_, M_, y_adj);
                                      ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:536:26: error: expansion pattern ‘auto&&’ contains no argument packs
         [this](auto&&... args) { this->accumulate_adjoints(args...); },
                          ^
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp: In lambda function:
stan/lib/stan_math/stan/math/rev/mat/functor/adj_jac_apply.hpp:536:60: error: ‘args’ was not declared in this scope
         [this](auto&&... args) { this->accumulate_adjoints(args...); },
                                                            ^
make: *** [examples/bernoulli/bernoulli] Error 1

jjramsey · January 8, 2019, 7:26pm

What version of compiler do you have? std::index_sequence (what the compiler is complaining about) is from C++14. The compiler that you are using may not be new enough to handle that standard.

caesoma · January 8, 2019, 8:42pm

I have

$ gcc --version                                                                                           
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36) 
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO                                         warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

if that’s the problem I can definitely ask the administrators to make any other version available, but does that make sense with there being no errors with the current release ?

jjramsey · January 8, 2019, 9:15pm

Why wouldn’t it make sense? Just because the CmdStan utilities and libraries compiled with g++ 4.8.5 doesn’t mean that the code generated by stanc will compile with the same compiler.

caesoma · January 8, 2019, 9:23pm

What I meant that wouldn’t make sense is the same model compiling with the current release (CmdStan 2.18.1), but not with the development branch. I assume whatever changed could be the cause of the error, unless it’s some esoteric interaction, does that make sense?

jjramsey · January 8, 2019, 9:27pm

Then offhand it looks to me like the development branch is using a C++14 feature (namely std::index_sequence) that the release version didn’t. I suppose it’s possible that a bug might have been introduced so that the header that defines std::index_sequence isn’t #include'd, but I’d try seeing if you can get a more recent compiler. You’ll probably need one eventually anyway.

caesoma · January 8, 2019, 11:01pm

Thanks. I’ll ask SysAdmin to make a newer compiler available and see if that solves it.

tedward · July 18, 2019, 8:58pm

@caesoma I think I am in the same situation as you. Did you resolve the problem with a new compiler?

caesoma · July 20, 2019, 4:02am

It’s been a while, but I think it was the specific version of the gcc compiler, I think I had problems with both the most recent and an older version, the cluster required that I loaded a specific module, so I had to keep track what was the right one. Didn’t have any problems since, but I’m not sure I can describe the exact solution right now.

mitzimorris · July 20, 2019, 12:53pm

@tedward are you using the latest version of development and/or 2.20 release?
2.20 release went out July 18 and yesterday (July 19) there was an additional fix specifically for Linux clusters - https://github.com/stan-dev/cmdstan/pull/717

if someone could compare develop vs. 2.20 release, that would be awesome.

tedward · July 24, 2019, 2:46am

Thanks for your responses to this. I have switched to using a docker image which meets my needs. If I have time I may go back to the way I was doing this previously and test the new release. I had been using 2.20.0, but haven’t tried since July 18. I’ll try to get around to testing this in the next week.

Topic		Replies	Views
Cmdstan 2.24.1 Compile Error CmdStan	3	668	September 21, 2020
Can't compile exemple/bernoulli model after adding threading flags to CmdStan [Ubuntu 18.04.3] CmdStan	4	1198	May 16, 2020
Unable to compile model using cmdstan on linux CmdStan	2	452	July 4, 2023
CmdStan Bernoulli Example Compiling but Not Running (Windows 10, CmdStan 2.26.1) General cmdstan	5	1186	March 30, 2021
Unable to compile `Bernoulli` example on macOS Monterey 12.6.1 CmdStan example-models	2	461	November 15, 2022

CmdStan development branch model compilation error on cluster

Related topics