Compiling errors after previous successful runs

Interfaces brms
1

Hi friends,

I’m running brms on a cluster with repeatedly drawn data samples. The codes work well for some samples but stop with compiling error. I copied the error message here. Any idea to fix this? It runs for a few cases but then halts. The data were repeatedly simulated from a pre-specified model. Thanks!

----cut off-----
Chain 4: Iteration: 1600 / 2000 [ 80%]  (Sampling)
Chain 4: Iteration: 1800 / 2000 [ 90%]  (Sampling)
Chain 4: Iteration: 2000 / 2000 [100%]  (Sampling)
Chain 4: 
Chain 4:  Elapsed Time: 260.78 seconds (Warm-up)
Chain 4:                113.24 seconds (Sampling)
Chain 4:                374.02 seconds (Total)
Chain 4: 
Compiling Stan program...
recompiling to avoid crashing R session
make cmd is
  make -f '/opt/R/4.0.5/lib/R/etc/Makeconf' -f '/opt/R/4.0.5/lib/R/share/make/shlib.mk' CXX='$(CXX14) $(CXX14STD)' CXXFLAGS='$(CXX14FLAGS)' CXXPICFLAGS='$(CXX14PICFLAGS)' SHLIB_LDFLAGS='$(SHLIB_CXX14LDFLAGS)' SHLIB_LD='$(SHLIB_CXX14LD)' SHLIB='filebe695637f8ad.so' OBJECTS='filebe695637f8ad.o'

make would use
g++ -std=gnu++14 -I"/opt/R/4.0.5/lib/R/include" -DNDEBUG   -I"/usr/local/Rsite/4.0/Rcpp/include/"  -I"/usr/local/Rsite/4.0/RcppEigen/include/"  -I"/usr/local/Rsite/4.0/RcppEigen/include/unsupported"  -I"/usr/local/Rsite/4.0/BH/include" -I"/home/yajuan/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/yajuan/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/usr/local/Rsite/4.0/RcppParallel/include/"  -I"/home/yajuan/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -DBOOST_DISABLE_ASSERTS  -DBOOST_PENDING_INTEGER_LOG2_HPP  -DSTAN_THREADS  -DBOOST_NO_AUTO_PTR  -include '/home/yajuan/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp'  -D_REENTRANT -DRCPP_PARALLEL_USE_TBB=1   -I/usr/local/include   -fpic  -g -O2  -c filebe695637f8ad.cpp -o filebe695637f8ad.o
if test  "zfilebe695637f8ad.o" != "z"; then \
  echo g++ -std=gnu++14 -shared -L"/opt/R/4.0.5/lib/R/lib" -L/usr/local/lib -o filebe695637f8ad.so filebe695637f8ad.o  '/home/yajuan/R/x86_64-pc-linux-gnu-library/4.0/rstan/lib//libStanServices.a' -L'/home/yajuan/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/lib/' -lStanHeaders -L'/usr/local/Rsite/4.0/RcppParallel/lib/' -ltbb  -L"/opt/R/4.0.5/lib/R/lib" -lR; \
  g++ -std=gnu++14 -shared -L"/opt/R/4.0.5/lib/R/lib" -L/usr/local/lib -o filebe695637f8ad.so filebe695637f8ad.o  '/home/yajuan/R/x86_64-pc-linux-gnu-library/4.0/rstan/lib//libStanServices.a' -L'/home/yajuan/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/lib/' -lStanHeaders -L'/usr/local/Rsite/4.0/RcppParallel/lib/' -ltbb  -L"/opt/R/4.0.5/lib/R/lib" -lR; \
fi
Error in compileCode(f, code, language = language, verbose = verbose) : 
  /home/yajuan/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/model_base_crtp.hpp:149:20:   required from here/usr/local/Rsite/4.0/RcppEigen/include/Eigen/src/Core/CoreEvaluators.h:960:8: warning: ignoring attributes on template argument ‘Eigen::internal::packet_traits<double>::type’ {aka ‘__vector(2) double’} [-Wignored-attributes]/tmp/cc1MwmvL.s: Assembler messages:/tmp/cc1MwmvL.s: Fatal error: can't write 3888 bytes to section .debug_info of filebe695637f8ad.o: 'No space left on device'/tmp/cc1MwmvL.s: Fatal error: can't close filebe695637f8ad.o: No space left on devicemake: *** [/opt/R/4.0.5/lib/R/etc/Makeconf:179: filebe695637f8ad.o] Error 1
Calls: trial ... cxxfunctionplus -> <Anonymous> -> cxxfunction -> compileCode
In addition: There were 39 warnings (use warnings() to see them)
Error in sink(type = "output") : invalid connection
Calls: trial ... eval2 -> eval -> eval -> .fun -> cxxfunctionplus -> sink
Execution halted

[edit: escaped console output]

2

Hi, @Yajuan_Si and welcome to the forums!

It looks like your setup is trying to recompile on individual nodes of the cluster, which can lead to trouble if they’re not configured with the software you need.

Are you using brms with cmdstanr or with RStan?

3

Thanks Bob. I was calling brms/Rstan and running it on a Linux cluster. I agree with you that the nodes can be messed up. Do you know how to change the setting? Thanks!

4

The important part of the error message here is:

Fatal error: can't write 3888 bytes to section .debug_info of filebe695637f8ad.o: 'No space left on device

So it seems that the node run out of memory.

1 Like