Cmdstanr not working with oneTBB on ubuntu

Interfaces CmdStan
,
1

Hi all,

I recently installed oneTBB because I had an issue with within-chain parallelization in rstan. However, now I cannot use cmdstan anymore.

  • Operating System: Unbuntu 20.04
  • CmdStan Version: 2.26.1
  • Compiler/Toolkit: gcc 4.8

I tried installing it via cmdstanr::install_cmdstan() in R as well as make clean-all and make build from terminal.
I also tried removing the entire folder and reinstalling it after that.

with the former method, it fails when trying to run compile the bernoulli.stan model during the installation with the following error:

--- Translating Stan model to C++ code ---
bin/stanc  --o=examples/bernoulli/bernoulli.hpp examples/bernoulli/bernoulli.stan

--- Compiling, linking C++ code ---
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes      -I /home/julian/oneapi-tbb-2021.1.1/include   -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.72.0 -I stan/lib/stan_math/lib/sundials_5.6.1/include    -DBOOST_DISABLE_ASSERTS   -DTBB_INTERFACE_NEW  -DTBB_INTERFACE_NEW    -c -Wno-ignored-attributes   -x c++ -o examples/bernoulli/bernoulli.o examples/bernoulli/bernoulli.hpp
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes      -I /home/julian/oneapi-tbb-2021.1.1/include   -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.72.0 -I stan/lib/stan_math/lib/sundials_5.6.1/include    -DBOOST_DISABLE_ASSERTS   -DTBB_INTERFACE_NEW  -DTBB_INTERFACE_NEW
   -Wl,-L,"/home/julian/oneapi-tbb-2021.1.1/lib/intel64/gcc4.8" -Wl,-rpath,"/home/julian/oneapi-tbb-2021.1.1/lib/intel64/gcc4.8 -ltbb"    examples/bernoulli/bernoulli.o src/cmdstan/main.o         stan/lib/stan_math/lib/sundials_5.6.1/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_5.6.1/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_5.6.1/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_5.6.1/lib/libsundials_kinsol.a   -o examples/bernoulli/bernoulli
/usr/bin/ld: examples/bernoulli/bernoulli.o: in function `tbb::detail::d1::task_scheduler_observer::~task_scheduler_observer()':
bernoulli.hpp:(.text._ZN3tbb6detail2d123task_scheduler_observerD0Ev[_ZN3tbb6detail2d123task_scheduler_observerD5Ev]+0x34): undefined reference to `tbb::detail::r1::observe(tbb::detail::d1::task_scheduler_observer&, bool)'
/usr/bin/ld: examples/bernoulli/bernoulli.o: in function `tbb::detail::d1::task_scheduler_observer::~task_scheduler_observer()':
bernoulli.hpp:(.text._ZN3tbb6detail2d123task_scheduler_observerD2Ev[_ZN3tbb6detail2d123task_scheduler_observerD5Ev]+0x27): undefined reference to `tbb::detail::r1::observe(tbb::detail::d1::task_scheduler_observer&, bool)'
/usr/bin/ld: examples/bernoulli/bernoulli.o: in function `stan::math::ad_tape_observer::~ad_tape_observer()':
bernoulli.hpp:(.text._ZN4stan4math16ad_tape_observerD0Ev[_ZN4stan4math16ad_tape_observerD5Ev]+0x1e): undefined reference to `tbb::detail::r1::observe(tbb::detail::d1::task_scheduler_observer&, bool)'
/usr/bin/ld: bernoulli.hpp:(.text._ZN4stan4math16ad_tape_observerD0Ev[_ZN4stan4math16ad_tape_observerD5Ev]+0xde): undefined reference to `tbb::detail::r1::observe(tbb::detail::d1::task_scheduler_observer&, bool)'
/usr/bin/ld: examples/bernoulli/bernoulli.o: in function `stan::math::ad_tape_observer::~ad_tape_observer()':
bernoulli.hpp:(.text._ZN4stan4math16ad_tape_observerD2Ev[_ZN4stan4math16ad_tape_observerD5Ev]+0x1e): undefined reference to `tbb::detail::r1::observe(tbb::detail::d1::task_scheduler_observer&, bool)'
/usr/bin/ld: examples/bernoulli/bernoulli.o:bernoulli.hpp:(.text._ZN4stan4math16ad_tape_observerD2Ev[_ZN4stan4math16ad_tape_observerD5Ev]+0xd6): more undefined references to `tbb::detail::r1::observe(tbb::detail::d1::task_scheduler_observer&, bool)' follow
/usr/bin/ld: examples/bernoulli/bernoulli.o: in function `__static_initialization_and_destruction_0(int, int) [clone .constprop.0]':
bernoulli.hpp:(.text.startup+0xb25): undefined reference to `tbb::detail::r1::assertion_failure(char const*, int, char const*, char const*)'
/usr/bin/ld: src/cmdstan/main.o: in function `cmdstan::command(int, char const**)':
main.cpp:(.text+0x7479): undefined reference to `tbb::detail::r1::create(tbb::detail::d1::global_control&)'
/usr/bin/ld: main.cpp:(.text+0x85c3): undefined reference to `tbb::detail::r1::assertion_failure(char const*, int, char const*, char const*)'
/usr/bin/ld: src/cmdstan/main.o: in function `tbb::detail::d1::global_control::~global_control()':
main.cpp:(.text._ZN3tbb6detail2d114global_controlD2Ev[_ZN3tbb6detail2d114global_controlD5Ev]+0x12): undefined reference to `tbb::detail::r1::destroy(tbb::detail::d1::global_control&)'
/usr/bin/ld: main.cpp:(.text._ZN3tbb6detail2d114global_controlD2Ev[_ZN3tbb6detail2d114global_controlD5Ev]+0x3b): undefined reference to `tbb::detail::r1::assertion_failure(char const*, int, char const*, char const*)'
/usr/bin/ld: main.cpp:(.text._ZN3tbb6detail2d114global_controlD2Ev[_ZN3tbb6detail2d114global_controlD5Ev]+0x43): undefined reference to `tbb::detail::r1::destroy(tbb::detail::d1::global_control&)'
/usr/bin/ld: src/cmdstan/main.o: in function `_GLOBAL__sub_I__ZN7cmdstan18write_compile_infoERN4stan9callbacks6writerERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISA_EE':
main.cpp:(.text.startup+0x3f9): undefined reference to `tbb::detail::r1::observe(tbb::detail::d1::task_scheduler_observer&, bool)'
/usr/bin/ld: main.cpp:(.text.startup+0xa99): undefined reference to `tbb::detail::r1::assertion_failure(char const*, int, char const*, char const*)'
collect2: error: ld returned 1 exit status
make: *** [make/program:62: examples/bernoulli/bernoulli] Error 1

When trying to use make build it fails with :

touch stan/lib/stan_math/lib/tbb/tbb-make-check
tbb_root="../oneapi-tbb-2021.1.1" CXX="g++" CC="gcc" LDFLAGS='-Wl,-L,"/home/julian/.cmdstanr/cmdstan/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/home/julian/.cmdstanr/cmdstan/stan/lib/stan_math/lib/tbb"  ' 'make' -C "stan/lib/stan_math/lib/tbb" -r -f "/home/julian/oneapi-tbb-2021.1.1/build/Makefile.tbbmalloc" compiler=gcc cfg=release stdver=c++1y malloc CXXFLAGS="-Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation  "
make[1]: Entering directory '/home/julian/.cmdstanr/cmdstan/stan/lib/stan_math/lib/tbb'
make[1]: /home/julian/oneapi-tbb-2021.1.1/build/Makefile.tbbmalloc: No such file or directory
make[1]: *** No rule to make target '/home/julian/oneapi-tbb-2021.1.1/build/Makefile.tbbmalloc'.  Stop.
make[1]: Leaving directory '/home/julian/.cmdstanr/cmdstan/stan/lib/stan_math/lib/tbb'
make: *** [stan/lib/stan_math/make/libraries:173: stan/lib/stan_math/lib/tbb/tbbmalloc.def] Error 2

Afterwards, I have a cmdstan/stan/lib/stan_math/lib/tbb folder, that only includes the tbb-make-check file and a cmdstan/stan/lib/stan_math/lib/tbb_2020.3 folder that includes all the other tbb files, so it seems to be looking for tbbmalloc.def in the wrong folder (tbb instead of tbb_2020.3).

I appreciate any pointers on how to resolve this.

Thanks in advance!

Julian

1 Like
3

Hi,

it seems that this is picking up the oneTBB installation you added for rstan.

If you wish it to work with oneTBB you need to run

cmdstan_make_local(cpp_options = list("TBB_INTERFACE_NEW" = TRUE))
rebuild_cmdstan(cores = 4)

and then try again.

1 Like
4

Thanks for your help! The building does indeed succeed, but when running the example:

file <- file.path(cmdstan_path(), "examples", "bernoulli", "bernoulli.stan")
mod <- cmdstan_model(file)

it throws a different error:

Compiling Stan program...
/usr/bin/ld: /tmp/RtmpPV7KAT/model-c26bc99994.o: in function `tbb::detail::d1::task_scheduler_observer::~task_scheduler_observer()':
model-c26bc99994.hpp:(.text._ZN3tbb6detail2d123task_scheduler_observerD0Ev[_ZN3tbb6detail2d123task_scheduler_observerD5Ev]+0x34): undefined reference to `tbb::detail::r1::observe(tbb::detail::d1::task_scheduler_observer&, bool)'
/usr/bin/ld: /tmp/RtmpPV7KAT/model-c26bc99994.o: in function `tbb::detail::d1::task_scheduler_observer::~task_scheduler_observer()':
model-c26bc99994.hpp:(.text._ZN3tbb6detail2d123task_scheduler_observerD2Ev[_ZN3tbb6detail2d123task_scheduler_observerD5Ev]+0x27): undefined reference to `tbb::detail::r1::observe(tbb::detail::d1::task_scheduler_observer&, bool)'
/usr/bin/ld: /tmp/RtmpPV7KAT/model-c26bc99994.o: in function `stan::math::ad_tape_observer::~ad_tape_observer()':
model-c26bc99994.hpp:(.text._ZN4stan4math16ad_tape_observerD0Ev[_ZN4stan4math16ad_tape_observerD5Ev]+0x1e): undefined reference to `tbb::detail::r1::observe(tbb::detail::d1::task_scheduler_observer&, bool)'
/usr/bin/ld: model-c26bc99994.hpp:(.text._ZN4stan4math16ad_tape_observerD0Ev[_ZN4stan4math16ad_tape_observerD5Ev]+0xde): undefined reference to `tbb::detail::r1::observe(tbb::detail::d1::task_scheduler_observer&, bool)'
/usr/bin/ld: /tmp/RtmpPV7KAT/model-c26bc99994.o: in function `stan::math::ad_tape_observer::~ad_tape_observer()':
model-c26bc99994.hpp:(.text._ZN4stan4math16ad_tape_observerD2Ev[_ZN4stan4math16ad_tape_observerD5Ev]+0x1e): undefined reference to `tbb::detail::r1::observe(tbb::detail::d1::task_scheduler_observer&, bool)'
/usr/bin/ld: /tmp/RtmpPV7KAT/model-c26bc99994.o:model-c26bc99994.hpp:(.text._ZN4stan4math16ad_tape_observerD2Ev[_ZN4stan4math16ad_tape_observerD5Ev]+0xd6): more undefined references to `tbb::detail::r1::observe(tbb::detail::d1::task_scheduler_observer&, bool)' follow
/usr/bin/ld: /tmp/RtmpPV7KAT/model-c26bc99994.o: in function `__static_initialization_and_destruction_0(int, int) [clone .constprop.0]':
model-c26bc99994.hpp:(.text.startup+0xb25): undefined reference to `tbb::detail::r1::assertion_failure(char const*, int, char const*, char const*)'
/usr/bin/ld: src/cmdstan/main.o: in function `cmdstan::command(int, char const**)':
main.cpp:(.text+0x7479): undefined reference to `tbb::detail::r1::create(tbb::detail::d1::global_control&)'
/usr/bin/ld: main.cpp:(.text+0x85c3): undefined reference to `tbb::detail::r1::assertion_failure(char const*, int, char const*, char const*)'
/usr/bin/ld: src/cmdstan/main.o: in function `tbb::detail::d1::global_control::~global_control()':
main.cpp:(.text._ZN3tbb6detail2d114global_controlD2Ev[_ZN3tbb6detail2d114global_controlD5Ev]+0x12): undefined reference to `tbb::detail::r1::destroy(tbb::detail::d1::global_control&)'
/usr/bin/ld: main.cpp:(.text._ZN3tbb6detail2d114global_controlD2Ev[_ZN3tbb6detail2d114global_controlD5Ev]+0x3b): undefined reference to `tbb::detail::r1::assertion_failure(char const*, int, char const*, char const*)'
/usr/bin/ld: main.cpp:(.text._ZN3tbb6detail2d114global_controlD2Ev[_ZN3tbb6detail2d114global_controlD5Ev]+0x43): undefined reference to `tbb::detail::r1::destroy(tbb::detail::d1::global_control&)'
/usr/bin/ld: src/cmdstan/main.o: in function `_GLOBAL__sub_I__ZN7cmdstan18write_compile_infoERN4stan9callbacks6writerERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISA_EE':
main.cpp:(.text.startup+0x3f9): undefined reference to `tbb::detail::r1::observe(tbb::detail::d1::task_scheduler_observer&, bool)'
/usr/bin/ld: main.cpp:(.text.startup+0xa99): undefined reference to `tbb::detail::r1::assertion_failure(char const*, int, char const*, char const*)'
collect2: error: ld returned 1 exit status
make: *** [make/program:62: /tmp/RtmpPV7KAT/model-c26bc99994] Error 1
Error: An error occured during compilation! See the message above for more information.
5

Arght, that is weird.

Do you want cmdstan to work with oneTBB or you dont care as long as it works? The solutions I can suggest are different depending on your goals.

6

Well, I thought that it would be best to have both (rstan and cmdstan) use oneTBB but I dont undestand much of it and trust your opinion here, so if you think that it does not have any advantages to use oneTBB here and there is an easy/easier way to have rstan work (with oneTBB) while cmdstan works with another tbb library, then it really does not matter to me as long as both work. Thanks again!

7

I was able to fix it by cloning the cmdstan repo directly into the home directory and then use make from console and set the cmdstan path manually in cmdstanr. Seems like it uses the tbb libraries in stan-math now if I interpret the g++ arguments during compilation correctly, but as long as it works :)

2 Likes