ERROR(s) during compilation with rstan in HPC

I’m trying to use rstan and brms in a High-Performance Computing (HPC) system, which runs Linux 3.10.0-327.36.3.el7.x86_64 x86_64. I run the following codes in a Conda environment.

I created the Conda environment with following codes (Conda version: 4.10.1):

conda create -n testenv
conda activate testenv
conda install -c conda-forge r-brms
conda install -c conda-forge r-bh 

Then I open R in the env and run the following codes:

library(rstan)
example(stan_model, package = "rstan", run.dontrun = TRUE)

With above code I got the error:
ERROR(s) during compilation: source code errors or compiler configuration errors!

The full output is saved as full_output.txt (35.1 KB). And the part relating to the error is:

make cmd is
  make -f '/scratch/hj23/miniconda3/envs/test2/lib/R/etc/Makeconf' -f '/scratch/hj23/miniconda3/envs/test2/lib/R/share/make/shlib.mk' -f '/home/hj23/.R/Makevars' CXX='$(CXX14) $(CXX14STD)' CXXFLAGS='$(CXX14FLAGS)' CXXPICFLAGS='$(CXX14PICFLAGS)' SHLIB_LDFLAGS='$(SHLIB_CXX14LDFLAGS)' SHLIB_LD='$(SHLIB_CXX14LD)' SHLIB='file39106a922ce2.so' OBJECTS='file39106a922ce2.o'

make would use
x86_64-conda-linux-gnu-c++ -std=gnu++14 -I"/scratch/hj23/miniconda3/envs/test2/lib/R/include" -DNDEBUG   -I"/scratch/hj23/miniconda3/envs/test2/lib/R/library/Rcpp/include/"  -I"/scratch/hj23/miniconda3/envs/test2/lib/R/library/RcppEigen/include/"  -I"/scratch/hj23/miniconda3/envs/test2/lib/R/library/RcppEigen/include/unsupported"  -I"/scratch/hj23/miniconda3/envs/test2/lib/R/library/BH/include" -I"/scratch/hj23/miniconda3/envs/test2/lib/R/library/StanHeaders/include/src/"  -I"/scratch/hj23/miniconda3/envs/test2/lib/R/library/StanHeaders/include/"  -I"/scratch/hj23/miniconda3/envs/test2/lib/R/library/RcppParallel/include/"  -I"/scratch/hj23/miniconda3/envs/test2/lib/R/library/rstan/include" -DEIGEN_NO_DEBUG  -DBOOST_DISABLE_ASSERTS  -DBOOST_PENDING_INTEGER_LOG2_HPP  -DSTAN_THREADS  -DBOOST_NO_AUTO_PTR  -include '/scratch/hj23/miniconda3/envs/test2/lib/R/library/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp'  -D_REENTRANT -DRCPP_PARALLEL_USE_TBB=1   -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /scratch/hj23/miniconda3/envs/test2/include -I/scratch/hj23/miniconda3/envs/test2/include -Wl,-rpath-link,/scratch/hj23/miniconda3/envs/test2/lib   -fpic  -O3 -march=native -mtune=native -fPIC -c file39106a922ce2.cpp -o file39106a922ce2.o
if test  "zfile39106a922ce2.o" != "z"; then \
  echo x86_64-conda-linux-gnu-c++ -std=gnu++14 -shared -L"/scratch/hj23/miniconda3/envs/test2/lib/R/lib" -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/scratch/hj23/miniconda3/envs/test2/lib -Wl,-rpath-link,/scratch/hj23/miniconda3/envs/test2/lib -L/scratch/hj23/miniconda3/envs/test2/lib -o file39106a922ce2.so file39106a922ce2.o  '/scratch/hj23/miniconda3/envs/test2/lib/R/library/rstan/lib//libStanServices.a' -L'/scratch/hj23/miniconda3/envs/test2/lib/R/library/StanHeaders/lib/' -lStanHeaders -L'/scratch/hj23/miniconda3/envs/test2/lib/R/library/RcppParallel/lib/' -ltbb  -L"/scratch/hj23/miniconda3/envs/test2/lib/R/lib" -lR; \
  x86_64-conda-linux-gnu-c++ -std=gnu++14 -shared -L"/scratch/hj23/miniconda3/envs/test2/lib/R/lib" -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/scratch/hj23/miniconda3/envs/test2/lib -Wl,-rpath-link,/scratch/hj23/miniconda3/envs/test2/lib -L/scratch/hj23/miniconda3/envs/test2/lib -o file39106a922ce2.so file39106a922ce2.o  '/scratch/hj23/miniconda3/envs/test2/lib/R/library/rstan/lib//libStanServices.a' -L'/scratch/hj23/miniconda3/envs/test2/lib/R/library/StanHeaders/lib/' -lStanHeaders -L'/scratch/hj23/miniconda3/envs/test2/lib/R/library/RcppParallel/lib/' -ltbb  -L"/scratch/hj23/miniconda3/envs/test2/lib/R/lib" -lR; \
fi

ERROR(s) during compilation: source code errors or compiler configuration errors!

Program source:
  1: 
  2: // includes from the plugin
  3: // [[Rcpp::plugins(cpp14)]]
  4: 
  5: 
  6: // user includes
  7: #include <Rcpp.h>
  8: #include <rstan/io/rlist_ref_var_context.hpp>
  9: #include <rstan/io/r_ostream.hpp>
 10: #include <rstan/stan_args.hpp>
 11: #include <boost/integer/integer_log2.hpp>
 12: // Code generated by Stan version 2.21.0
 13: 
 14: #include <stan/model/model_header.hpp>
 15: 
 16: namespace model3910519e785d_16a540c6086086816528e4524def24d9_namespace {
 17: 
 18: using std::istream;
 19: using std::string;
 20: using std::stringstream;
 21: using std::vector;
 22: using stan::io::dump;
 23: using stan::math::lgamma;
 24: using stan::model::prob_grad;
 25: using namespace stan::math;
 26: 
 27: static int current_statement_begin__;
 28: 
 29: stan::io::program_reader prog_reader__() {
 30:     stan::io::program_reader reader;
 31:     reader.add_event(0, 0, "start", "model3910519e785d_16a540c6086086816528e4524def24d9");
 32:     reader.add_event(3, 1, "end", "model3910519e785d_16a540c6086086816528e4524def24d9");
 33:     return reader;
 34: }
 35: 
 36: class model3910519e785d_16a540c6086086816528e4524def24d9
 37:   : public stan::model::model_base_crtp<model3910519e785d_16a540c6086086816528e4524def24d9> {
 38: private:
 39:         double y_mean;
 40: public:
 41:     model3910519e785d_16a540c6086086816528e4524def24d9(rstan::io::rlist_ref_var_context& context__,
 42:         std::ostream* pstream__ = 0)
 43:         : model_base_crtp(0) {
 44:         ctor_body(context__, 0, pstream__);
 45:     }
 46: 
 47:     model3910519e785d_16a540c6086086816528e4524def24d9(stan::io::var_context& context__,
 48:         unsigned int random_seed__,
 49:         std::ostream* pstream__ = 0)
 50:         : model_base_crtp(0) {
 51:         ctor_body(context__, random_seed__, pstream__);
 52:     }
 53: 
 54:     void ctor_body(stan::io::var_context& context__,
 55:                    unsigned int random_seed__,
 56:                    std::ostream* pstream__) {
 57:         typedef double local_scalar_t__;
 58: 
 59:         boost::ecuyer1988 base_rng__ =
 60:           stan::services::util::create_rng(random_seed__, 0);
 61:         (void) base_rng__;  // suppress unused var warning
 62: 
 63:         current_statement_begin__ = -1;
 64: 
 65:         static const char* function__ = "model3910519e785d_16a540c6086086816528e4524def24d9_namespace::model3910519e785d_16a540c6086086816528e4524def24d9";
 66:         (void) function__;  // dummy to suppress unused var warning
 67:         size_t pos__;
 68:         (void) pos__;  // dummy to suppress unused var warning
 69:         std::vector<int> vals_i__;
 70:         std::vector<double> vals_r__;
 71:         local_scalar_t__ DUMMY_VAR__(std::numeric_limits<double>::quiet_NaN());
 72:         (void) DUMMY_VAR__;  // suppress unused var warning
 73: 
 74:         try {
 75:             // initialize data block variables from context__
 76:             current_statement_begin__ = 1;
 77:             context__.validate_dims("data initialization", "y_mean", "double", context__.to_vec());
 78:             y_mean = double(0);
 79:             vals_r__ = context__.vals_r("y_mean");
 80:             pos__ = 0;
 81:             y_mean = vals_r__[pos__++];
 82: 
 83: 
 84:             // initialize transformed data variables
 85:             // execute transformed data statements
 86: 
 87:             // validate transformed data
 88: 
 89:             // validate, set parameter ranges
 90:             num_params_r__ = 0U;
 91:             param_ranges_i__.clear();
 92:             current_statement_begin__ = 1;
 93:             num_params_r__ += 1;
 94:         } catch (const std::exception& e) {
 95:             stan::lang::rethrow_located(e, current_statement_begin__, prog_reader__());
 96:             // Next line prevents compiler griping about no return
 97:             throw std::runtime_error("*** IF YOU SEE THIS, PLEASE REPORT A BUG ***");
 98:         }
 99:     }
100: 
101:     ~model3910519e785d_16a540c6086086816528e4524def24d9() { }
102: 
103: 
104:     void transform_inits(const stan::io::var_context& context__,
105:                          std::vector<int>& params_i__,
106:                          std::vector<double>& params_r__,
107:                          std::ostream* pstream__) const {
108:         typedef double local_scalar_t__;
109:         stan::io::writer<double> writer__(params_r__, params_i__);
110:         size_t pos__;
111:         (void) pos__; // dummy call to supress warning
112:         std::vector<double> vals_r__;
113:         std::vector<int> vals_i__;
114: 
115:         current_statement_begin__ = 1;
116:         if (!(context__.contains_r("y")))
117:             stan::lang::rethrow_located(std::runtime_error(std::string("Variable y missing")), current_statement_begin__, prog_reader__());
118:         vals_r__ = context__.vals_r("y");
119:         pos__ = 0U;
120:         context__.validate_dims("parameter initialization", "y", "double", context__.to_vec());
121:         double y(0);
122:         y = vals_r__[pos__++];
123:         try {
124:             writer__.scalar_unconstrain(y);
125:         } catch (const std::exception& e) {
126:             stan::lang::rethrow_located(std::runtime_error(std::string("Error transforming variable y: ") + e.what()), current_statement_begin__, prog_reader__());
127:         }
128: 
129:         params_r__ = writer__.data_r();
130:         params_i__ = writer__.data_i();
131:     }
132: 
133:     void transform_inits(const stan::io::var_context& context,
134:                          Eigen::Matrix<double, Eigen::Dynamic, 1>& params_r,
135:                          std::ostream* pstream__) const {
136:       std::vector<double> params_r_vec;
137:       std::vector<int> params_i_vec;
138:       transform_inits(context, params_i_vec, params_r_vec, pstream__);
139:       params_r.resize(params_r_vec.size());
140:       for (int i = 0; i < params_r.size(); ++i)
141:         params_r(i) = params_r_vec[i];
142:     }
143: 
144: 
145:     template <bool propto__, bool jacobian__, typename T__>
146:     T__ log_prob(std::vector<T__>& params_r__,
147:                  std::vector<int>& params_i__,
148:                  std::ostream* pstream__ = 0) const {
149: 
150:         typedef T__ local_scalar_t__;
151: 
152:         local_scalar_t__ DUMMY_VAR__(std::numeric_limits<double>::quiet_NaN());
153:         (void) DUMMY_VAR__;  // dummy to suppress unused var warning
154: 
155:         T__ lp__(0.0);
156:         stan::math::accumulator<T__> lp_accum__;
157:         try {
158:             stan::io::reader<local_scalar_t__> in__(params_r__, params_i__);
159: 
160:             // model parameters
161:             current_statement_begin__ = 1;
162:             local_scalar_t__ y;
163:             (void) y;  // dummy to suppress unused var warning
164:             if (jacobian__)
165:                 y = in__.scalar_constrain(lp__);
166:             else
167:                 y = in__.scalar_constrain();
168: 
169:             // model body
170: 
171:             current_statement_begin__ = 1;
172:             lp_accum__.add(normal_log<propto__>(y, y_mean, 1));
173: 
174:         } catch (const std::exception& e) {
175:             stan::lang::rethrow_located(e, current_statement_begin__, prog_reader__());
176:             // Next line prevents compiler griping about no return
177:             throw std::runtime_error("*** IF YOU SEE THIS, PLEASE REPORT A BUG ***");
178:         }
179: 
180:         lp_accum__.add(lp__);
181:         return lp_accum__.sum();
182: 
183:     } // log_prob()
184: 
185:     template <bool propto, bool jacobian, typename T_>
186:     T_ log_prob(Eigen::Matrix<T_,Eigen::Dynamic,1>& params_r,
187:                std::ostream* pstream = 0) const {
188:       std::vector<T_> vec_params_r;
189:       vec_params_r.reserve(params_r.size());
190:       for (int i = 0; i < params_r.size(); ++i)
191:         vec_params_r.push_back(params_r(i));
192:       std::vector<int> vec_params_i;
193:       return log_prob<propto,jacobian,T_>(vec_params_r, vec_params_i, pstream);
194:     }
195: 
196: 
197:     void get_param_names(std::vector<std::string>& names__) const {
198:         names__.resize(0);
199:         names__.push_back("y");
200:     }
201: 
202: 
203:     void get_dims(std::vector<std::vector<size_t> >& dimss__) const {
204:         dimss__.resize(0);
205:         std::vector<size_t> dims__;
206:         dims__.resize(0);
207:         dimss__.push_back(dims__);
208:     }
209: 
210:     template <typename RNG>
211:     void write_array(RNG& base_rng__,
212:                      std::vector<double>& params_r__,
213:                      std::vector<int>& params_i__,
214:                      std::vector<double>& vars__,
215:                      bool include_tparams__ = true,
216:                      bool include_gqs__ = true,
217:                      std::ostream* pstream__ = 0) const {
218:         typedef double local_scalar_t__;
219: 
220:         vars__.resize(0);
221:         stan::io::reader<local_scalar_t__> in__(params_r__, params_i__);
222:         static const char* function__ = "model3910519e785d_16a540c6086086816528e4524def24d9_namespace::write_array";
223:         (void) function__;  // dummy to suppress unused var warning
224: 
225:         // read-transform, write parameters
226:         double y = in__.scalar_constrain();
227:         vars__.push_back(y);
228: 
229:         double lp__ = 0.0;
230:         (void) lp__;  // dummy to suppress unused var warning
231:         stan::math::accumulator<double> lp_accum__;
232: 
233:         local_scalar_t__ DUMMY_VAR__(std::numeric_limits<double>::quiet_NaN());
234:         (void) DUMMY_VAR__;  // suppress unused var warning
235: 
236:         if (!include_tparams__ && !include_gqs__) return;
237: 
238:         try {
239:             if (!include_gqs__ && !include_tparams__) return;
240:             if (!include_gqs__) return;
241:         } catch (const std::exception& e) {
242:             stan::lang::rethrow_located(e, current_statement_begin__, prog_reader__());
243:             // Next line prevents compiler griping about no return
244:             throw std::runtime_error("*** IF YOU SEE THIS, PLEASE REPORT A BUG ***");
245:         }
246:     }
247: 
248:     template <typename RNG>
249:     void write_array(RNG& base_rng,
250:                      Eigen::Matrix<double,Eigen::Dynamic,1>& params_r,
251:                      Eigen::Matrix<double,Eigen::Dynamic,1>& vars,
252:                      bool include_tparams = true,
253:                      bool include_gqs = true,
254:                      std::ostream* pstream = 0) const {
255:       std::vector<double> params_r_vec(params_r.size());
256:       for (int i = 0; i < params_r.size(); ++i)
257:         params_r_vec[i] = params_r(i);
258:       std::vector<double> vars_vec;
259:       std::vector<int> params_i_vec;
260:       write_array(base_rng, params_r_vec, params_i_vec, vars_vec, include_tparams, include_gqs, pstream);
261:       vars.resize(vars_vec.size());
262:       for (int i = 0; i < vars.size(); ++i)
263:         vars(i) = vars_vec[i];
264:     }
265: 
266:     std::string model_name() const {
267:         return "model3910519e785d_16a540c6086086816528e4524def24d9";
268:     }
269: 
270: 
271:     void constrained_param_names(std::vector<std::string>& param_names__,
272:                                  bool include_tparams__ = true,
273:                                  bool include_gqs__ = true) const {
274:         std::stringstream param_name_stream__;
275:         param_name_stream__.str(std::string());
276:         param_name_stream__ << "y";
277:         param_names__.push_back(param_name_stream__.str());
278: 
279:         if (!include_gqs__ && !include_tparams__) return;
280: 
281:         if (include_tparams__) {
282:         }
283: 
284:         if (!include_gqs__) return;
285:     }
286: 
287: 
288:     void unconstrained_param_names(std::vector<std::string>& param_names__,
289:                                    bool include_tparams__ = true,
290:                                    bool include_gqs__ = true) const {
291:         std::stringstream param_name_stream__;
292:         param_name_stream__.str(std::string());
293:         param_name_stream__ << "y";
294:         param_names__.push_back(param_name_stream__.str());
295: 
296:         if (!include_gqs__ && !include_tparams__) return;
297: 
298:         if (include_tparams__) {
299:         }
300: 
301:         if (!include_gqs__) return;
302:     }
303: 
304: }; // model
305: 
306: }  // namespace
307: 
308: typedef model3910519e785d_16a540c6086086816528e4524def24d9_namespace::model3910519e785d_16a540c6086086816528e4524def24d9 stan_model;
309: 
310: #ifndef USING_R
311: 
312: stan::model::model_base& new_model(
313:         stan::io::var_context& data_context,
314:         unsigned int seed,
315:         std::ostream* msg_stream) {
316:   stan_model* m = new stan_model(data_context, seed, msg_stream);
317:   return *m;
318: }
319: 
320: #endif
321: 
322: 
323: 
324: #include <rstan_next/stan_fit.hpp>
325: 
326: struct stan_model_holder {
327:     stan_model_holder(rstan::io::rlist_ref_var_context rcontext,
328:                       unsigned int random_seed)
329:     : rcontext_(rcontext), random_seed_(random_seed)
330:      {
331:      }
332: 
333:    //stan::math::ChainableStack ad_stack;
334:    rstan::io::rlist_ref_var_context rcontext_;
335:    unsigned int random_seed_;
336: };
337: 
338: Rcpp::XPtr<stan::model::model_base> model_ptr(stan_model_holder* smh) {
339:   Rcpp::XPtr<stan::model::model_base> model_instance(new stan_model(smh->rcontext_, smh->random_seed_), true);
340:   return model_instance;
341: }
342: 
343: Rcpp::XPtr<rstan::stan_fit_base> fit_ptr(stan_model_holder* smh) {
344:   return Rcpp::XPtr<rstan::stan_fit_base>(new rstan::stan_fit(model_ptr(smh), smh->random_seed_), true);
345: }
346: 
347: std::string model_name(stan_model_holder* smh) {
348:   return model_ptr(smh).get()->model_name();
349: }
350: 
351: RCPP_MODULE(stan_fit4model3910519e785d_16a540c6086086816528e4524def24d9_mod){
352:   Rcpp::class_<stan_model_holder>("stan_fit4model3910519e785d_16a540c6086086816528e4524def24d9")
353:   .constructor<rstan::io::rlist_ref_var_context, unsigned int>()
354:   .method("model_ptr", &model_ptr)
355:   .method("fit_ptr", &fit_ptr)
356:   .method("model_name", &model_name)
357:   ;
358: }
359: 
360: 
361: // declarations
362: extern "C" {
363: SEXP file39106a922ce2( ) ;
364: }
365: 
366: // definition
367: SEXP file39106a922ce2() {
368:  return Rcpp::wrap("16a540c6086086816528e4524def24d9");
369: }

Compilation ERROR, function(s)/method(s) not created!
Error in compileCode(f, code, language = language, verbose = verbose) : 
    650 |   return internal::first_aligned<int(unpacket_traits<DefaultPacketType>::alignment),Derived>(m);      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~/scratch/hj23/miniconda3/envs/test2/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /scratch/hj23/miniconda3/envs/test2/lib/R/library/rstan/lib//libStanServices.a(stan_fit.o): relocation R_X86_64_PC32 against undefined hidden symbol `_ZTCN5boost10wrapexceptISt14overflow_errorEE0_NS_16exception_detail10clone_implINS3_19error_info_injectorIS1_EEEE' can not be used when making a shared object/scratch/hj23/miniconda3/envs/test2/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: final link failed: bad valuecollect2: error: ld returned 1 exit statusmake: *** [/scratch/hj23/miniconda3/envs/test2/lib/R/share/make/shlib.mk:10: file39106a922ce2.so] Error 1

I also tried the above codes in Conda env in a local iMac and it works well (it can sample). Thus, I guess the error is relating to HPC.

Please let me know what I can try and the potential limitations that the HPC may have.

Thanks in advance!

As a preface, conda is notorious for not working with RStan/brms since it causes issues with the compiler linking.

How did you install rstan on this system? Through conda or through R?

I installed rstan through conda. When I run conda install -c conda-forge r-brms, it also installed rstan.

Good to know this. The reason I use Conda is that it is kind of the only way HPC people suggest using R in HPC. It works when I use some other packages (e.g., lme4). It will save lots of time if I can use rstan and brms in HPC.
I only have experience using rstan to fit Stan models. Maybe I should try to use cmdStan instead? Any suggestion?

Thanks!

Can you try reinstalling rstan and StanHeaders through R?

Launch R (making sure that rstan and brms don’t get loaded), and run:

remove.packages(c("StanHeaders", "rstan")) 

# Compile packages using all cores 
Sys.setenv(MAKEFLAGS = paste0("-j",parallel::detectCores()))

install.packages(c("StanHeaders","rstan"),type="source")
1 Like

It works now!

Many thanks for the suggestion.

Do you mind explaining a little bit what the issue is?

Sometimes the binary packages can have issues if they’re compiled against different versions of the dependency packages (e.g., BH and RcppEigen) then what is installed on your system. Re-compiling them against your local packages can fix this

1 Like

Thanks again!

1 Like