Runtime error: symbol lookup error with multithreaded compile

Interfaces
1

Hi,

I’m trying to run a multithreaded model (using reduce_sum) but whenever I try running it use cmdstanr::sample I get the below error

Running MCMC with 16 parallel chains, with 2 thread(s) per chain...

Chain 1 ./ts_model: symbol lookup error: ./ts_model: undefined symbol: _ZN3tbb10interface78internal20isolate_within_arenaERNS1_13delegate_baseEl
Chain 2 ./ts_model: symbol lookup error: ./ts_model: undefined symbol: _ZN3tbb10interface78internal20isolate_within_arenaERNS1_13delegate_baseEl
Chain 3 ./ts_model: symbol lookup error: ./ts_model: undefined symbol: _ZN3tbb10interface78internal20isolate_within_arenaERNS1_13delegate_baseEl
Chain 4 ./ts_model: symbol lookup error: ./ts_model: undefined symbol: _ZN3tbb10interface78internal20isolate_within_arenaERNS1_13delegate_baseEl
Chain 5 ./ts_model: symbol lookup error: ./ts_model: undefined symbol: _ZN3tbb10interface78internal20isolate_within_arenaERNS1_13delegate_baseEl
Chain 6 ./ts_model: symbol lookup error: ./ts_model: undefined symbol: _ZN3tbb10interface78internal20isolate_within_arenaERNS1_13delegate_baseEl
Chain 7 ./ts_model: symbol lookup error: ./ts_model: undefined symbol: _ZN3tbb10interface78internal20isolate_within_arenaERNS1_13delegate_baseEl
Chain 8 ./ts_model: symbol lookup error: ./ts_model: undefined symbol: _ZN3tbb10interface78internal20isolate_within_arenaERNS1_13delegate_baseEl
Chain 9 ./ts_model: symbol lookup error: ./ts_model: undefined symbol: _ZN3tbb10interface78internal20isolate_within_arenaERNS1_13delegate_baseEl
Chain 10 ./ts_model: symbol lookup error: ./ts_model: undefined symbol: _ZN3tbb10interface78internal20isolate_within_arenaERNS1_13delegate_baseEl
Chain 11 ./ts_model: symbol lookup error: ./ts_model: undefined symbol: _ZN3tbb10interface78internal20isolate_within_arenaERNS1_13delegate_baseEl
Chain 12 ./ts_model: symbol lookup error: ./ts_model: undefined symbol: _ZN3tbb10interface78internal20isolate_within_arenaERNS1_13delegate_baseEl
Chain 13 ./ts_model: symbol lookup error: ./ts_model: undefined symbol: _ZN3tbb10interface78internal20isolate_within_arenaERNS1_13delegate_baseEl
Chain 14 ./ts_model: symbol lookup error: ./ts_model: undefined symbol: _ZN3tbb10interface78internal20isolate_within_arenaERNS1_13delegate_baseEl
Chain 15 ./ts_model: symbol lookup error: ./ts_model: undefined symbol: _ZN3tbb10interface78internal20isolate_within_arenaERNS1_13delegate_baseEl
Chain 16 ./ts_model: symbol lookup error: ./ts_model: undefined symbol: _ZN3tbb10interface78internal20isolate_within_arenaERNS1_13delegate_baseEl

I’m using cmdstanr 0.4.0.9001 and CmdStan version 2.29. I’m running on Ubuntu Linux.

This is the output from compiling my model (notice that I don’t find a *_threads version of my model being compiled).

ts_model <- cmdstan_model("ts_model.stan", include_paths = here::here(),  force = T, quiet = FALSE, cpp_options = list(stan_threads = TRUE))
Compiling Stan program...
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes     -DSTAN_THREADS -I stan/lib/stan_math/lib/tbb_2020.3/include    -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials    -DBOOST_DISABLE_ASSERTS          -c -MT stan/src/stan/model/model_header_threads.hpp.gch -MT stan/src/stan/model/model_header_threads.d -MM -E -MG -MP -MF stan/src/stan/model/model_header_threads.d stan/src/stan/model/model_header.hpp

--- Compiling the main object file. This might take up to a minute. ---
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes     -DSTAN_THREADS -I stan/lib/stan_math/lib/tbb_2020.3/include    -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials    -DBOOST_DISABLE_ASSERTS          -c -o src/cmdstan/main_threads.o src/cmdstan/main.cpp

--- Compiling pre-compiled header. This might take a few seconds. ---
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes     -DSTAN_THREADS -I stan/lib/stan_math/lib/tbb_2020.3/include    -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials    -DBOOST_DISABLE_ASSERTS          -c stan/src/stan/model/model_header.hpp -o stan/src/stan/model/model_header_threads.hpp.gch

--- Translating Stan model to C++ code ---
bin/stanc --include-paths=/home/karim/Code/science/cont-net-cash-elasticity --name='ts_model_model' --o=/tmp/RtmpXo4x9z/model-10b631f469e0.hpp /tmp/RtmpXo4x9z/model-10b631f469e0.stan

--- Compiling, linking C++ code ---
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes     -DSTAN_THREADS -I stan/lib/stan_math/lib/tbb_2020.3/include    -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials    -DBOOST_DISABLE_ASSERTS          -c -Wno-ignored-attributes   -x c++ -o /tmp/RtmpXo4x9z/model-10b631f469e0.o /tmp/RtmpXo4x9z/model-10b631f469e0.hpp
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes     -DSTAN_THREADS -I stan/lib/stan_math/lib/tbb_2020.3/include    -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials    -DBOOST_DISABLE_ASSERTS                -Wl,-L,"/home/karim/.cmdstan/cmdstan-2.29.0/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/home/karim/.cmdstan/cmdstan-2.29.0/stan/lib/stan_math/lib/tbb"      /tmp/RtmpXo4x9z/model-10b631f469e0.o src/cmdstan/main_threads.o        -Wl,-L,"/home/karim/.cmdstan/cmdstan-2.29.0/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/home/karim/.cmdstan/cmdstan-2.29.0/stan/lib/stan_math/lib/tbb"   stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_kinsol.a  stan/lib/stan_math/lib/tbb/libtbb.so.2 -o /tmp/RtmpXo4x9z/model-10b631f469e0
rm -f /tmp/RtmpXo4x9z/model-10b631f469e0.o

The very same configuration and model compiled and runs normally on my Mac laptop.

2

Does the same thing happen if you run:

library(cmdstanr)

rebuild_cmdstan(cores = 4)

file <- file.path(cmdstan_path(), "examples", "bernoulli", "bernoulli.stan")
mod <- cmdstan_model(file)
mod <- cmdstan_model(file, cpp_options = list(STAN_THREADS = TRUE), force_recompile = TRUE)
3

It works fine with that model. I ran both using mod$sample(data = list(N = 10, y = rbinom(10, 1, 0.5))).

4

I noticed that if I comment out this line of code in my model

    target += reduce_sum(partial_true_seller_log_lik_sum, rep_array(0, prod(dims(n_true_seller))), 1, n_true_seller, n_ts_lead, toplevel_ts_gp, offer_channel_ts_gp, market_ts_gp);

Things work fine.

5

Not at my computer anymore, but it looks as if its grabbing system installed tbb.

Can you try setting TBB_INC and TBB_LIB variables to point to your cmdstan install?

TBB_INC="$TBB/include"
TBB_LIB="$TBB/lib/intel64/gcc4.8

Can provide more details in the morning.

6

I found “stan/lib/stan_math/lib/tbb_2020.3/include” for the include env var, but I’m not finding anything with gcc4.8 in my .cmdstanr directory.

7

Hi @rok_cesnovar, do you know what I should set the TBB env vars to?

Thanks,
Karim

8

Can anyone please help me with this problem? I’m totally stuck and I need to make this work for my model to run. I’ve never had problems with parallelization before running on an Ubuntu virtual machine. My work is essentially blocked. I would really appreciate any suggestions!

9

does this directory exist and what does it contain?

10

Yes, it exists:

drwxr-xr-x 3 karim karim 4.0K Feb 14 20:05 build
drwxr-xr-x 3 karim karim 4.0K Feb 14 20:05 cmake
drwxr-xr-x 2 karim karim 4.0K Feb 14 20:05 doc
drwxr-xr-x 4 karim karim 4.0K Feb 14 20:05 include
drwxr-xr-x 2 karim karim 4.0K Feb 14 20:05 jni
drwxr-xr-x 4 karim karim 4.0K Feb 14 20:05 python
drwxr-xr-x 9 karim karim 4.0K Feb 14 20:05 src
-rw-r--r-- 1 karim karim 113K Feb 14 14:48 CHANGES
-rw-r--r-- 1 karim karim  55K Feb 14 14:48 Doxyfile
-rw-r--r-- 1 karim karim  12K Feb 14 14:48 LICENSE
-rw-r--r-- 1 karim karim 2.6K Feb 14 14:48 Makefile
-rw-r--r-- 1 karim karim  353 Feb 14 14:48 README
-rw-r--r-- 1 karim karim 1.9K Feb 14 14:48 README.md
-rw-r--r-- 1 karim karim  143 Feb 14 14:48 STAN_CHANGES
-rw-r--r-- 1 karim karim 1.8K Feb 14 14:48 index.html
-rw-r--r-- 1 karim karim  16K Feb 14 14:48 third-party-programs.txt
11

I tried to run like this, even though I’m not sure what the right values to use here:

 ts_model <- withr::with_envvar(
  c(
    "TBB_INC" = file.path(cmdstan_path(), "stan/lib/stan_math/lib/tbb_2020.3/include"),
    "TBB_LIB" = file.path(cmdstan_path(), "stan/lib/stan_math/lib/tbb")
  ),
  cmdstan_model("ts_model.stan", include_paths = here::here(),  force = T, quiet = FALSE, cpp_options = list(stan_threads = TRUE))
 )

I got these errors now:

Compiling Stan program...

--- Translating Stan model to C++ code ---
bin/stanc --include-paths=/home/karim/Code/science/cont-net-cash-elasticity --name='ts_model_model' --o=/tmp/RtmptOnD3w/model-73c712647f85.hpp /tmp/RtmptOnD3w/model-73c712647f85.stan

--- Compiling, linking C++ code ---
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes     -DSTAN_THREADS -I /home/karim/.cmdstanr/cmdstan-2.29.0/stan/lib/stan_math/lib/tbb_2020.3/include    -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials    -DBOOST_DISABLE_ASSERTS          -c -Wno-ignored-attributes   -x c++ -o /tmp/RtmptOnD3w/model-73c712647f85.o /tmp/RtmptOnD3w/model-73c712647f85.hpp
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes     -DSTAN_THREADS -I /home/karim/.cmdstanr/cmdstan-2.29.0/stan/lib/stan_math/lib/tbb_2020.3/include    -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials    -DBOOST_DISABLE_ASSERTS                -Wl,-L,"/home/karim/.cmdstanr/cmdstan-2.29.0/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/home/karim/.cmdstanr/cmdstan-2.29.0/stan/lib/stan_math/lib/tbb" -Wl,--disable-new-dtags -ltbb    /tmp/RtmptOnD3w/model-73c712647f85.o src/cmdstan/main_threads.o        -Wl,-L,"/home/karim/.cmdstanr/cmdstan-2.29.0/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/home/karim/.cmdstanr/cmdstan-2.29.0/stan/lib/stan_math/lib/tbb" -Wl,--disable-new-dtags -ltbb stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_kinsol.a   -o /tmp/RtmptOnD3w/model-73c712647f85
/tmp/RtmptOnD3w/model-73c712647f85.o: In function `stan::math::internal::reduce_sum_impl<ts_model_model_namespace::partial_true_seller_log_lik_sum_rsfunctor__, void, stan::math::var_value<double, void>, std::vector<int, std::allocator<int> >, std::vector<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > >, std::allocator<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > > > const&, std::vector<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > >, std::allocator<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > > > const&, Eigen::Matrix<stan::math::var_value<double, void>, -1, 1, 0, -1, 1>&, Eigen::Matrix<stan::math::var_value<double, void>, -1, -1, 0, -1, -1>&, Eigen::Matrix<stan::math::var_value<double, void>, -1, -1, 0, -1, -1>&>::operator()(std::vector<int, std::allocator<int> >&&, bool, int, std::ostream*, std::vector<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > >, std::allocator<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > > > const&, std::vector<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > >, std::allocator<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > > > const&, Eigen::Matrix<stan::math::var_value<double, void>, -1, 1, 0, -1, 1>&, Eigen::Matrix<stan::math::var_value<double, void>, -1, -1, 0, -1, -1>&, Eigen::Matrix<stan::math::var_value<double, void>, -1, -1, 0, -1, -1>&) const':
model-73c712647f85.hpp:(.text._ZNK4stan4math8internal15reduce_sum_implIN24ts_model_model_namespace43partial_true_seller_log_lik_sum_rsfunctor__EvNS0_9var_valueIdvEESt6vectorIiSaIiEEJRKS7_IS7_IS9_SaIS9_EESaISB_EESF_RN5Eigen6MatrixIS6_Lin1ELi1ELi0ELin1ELi1EEERNSH_IS6_Lin1ELin1ELi0ELin1ELin1EEESL_EEclEOS9_biPSoSF_SF_SJ_SL_SL_[_ZNK4stan4math8internal15reduce_sum_implIN24ts_model_model_namespace43partial_true_seller_log_lik_sum_rsfunctor__EvNS0_9var_valueIdvEESt6vectorIiSaIiEEJRKS7_IS7_IS9_SaIS9_EESaISB_EESF_RN5Eigen6MatrixIS6_Lin1ELi1ELi0ELin1ELi1EEERNSH_IS6_Lin1ELin1ELi0ELin1ELin1EEESL_EEclEOS9_biPSoSF_SF_SJ_SL_SL_]+0x6a0): undefined reference to `tbb::interface7::internal::isolate_within_arena(tbb::interface7::internal::delegate_base&, long)'
collect2: error: ld returned 1 exit status
/tmp/RtmptOnD3w/model-73c712647f85.o: In function `stan::math::internal::reduce_sum_impl<ts_model_model_namespace::partial_true_seller_log_lik_sum_rsfunctor__, void, stan::math::var_value<double, void>, std::vector<int, std::allocator<int> >, std::vector<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > >, std::allocator<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > > > const&, std::vector<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > >, std::allocator<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > > > const&, Eigen::Matrix<stan::math::var_value<double, void>, -1, 1, 0, -1, 1>&, Eigen::Matrix<stan::math::var_value<double, void>, -1, -1, 0, -1, -1>&, Eigen::Matrix<stan::math::var_value<double, void>, -1, -1, 0, -1, -1>&>::operator()(std::vector<int, std::allocator<int> >&&, bool, int, std::ostream*, std::vector<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > >, std::allocator<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > > > const&, std::vector<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > >, std::allocator<std::vector<std::vector<int, std::allocator<int> >, std::allocator<std::vector<int, std::allocator<int> > > > > > const&, Eigen::Matrix<stan::math::var_value<double, void>, -1, 1, 0, -1, 1>&, Eigen::Matrix<stan::math::var_value<double, void>, -1, -1, 0, -1, -1>&, Eigen::Matrix<stan::math::var_value<double, void>, -1, -1, 0, -1, -1>&) const':
model-73c712647f85.hpp:(.text._ZNK4stan4math8internal15reduce_sum_implIN24ts_model_model_namespace43partial_true_seller_log_lik_sum_rsfunctor__EvNS0_9var_valueIdvEESt6vectorIiSaIiEEJRKS7_IS7_IS9_SaIS9_EESaISB_EESF_RN5Eigen6MatrixIS6_Lin1ELi1ELi0ELin1ELi1EEERNSH_IS6_Lin1ELin1ELi0ELin1ELin1EEESL_EEclEOS9_biPSoSF_SF_SJ_SL_SL_[_ZNK4stan4math8internal15reduce_sum_implIN24ts_model_model_namespace43partial_true_seller_log_lik_sum_rsfunctor__EvNS0_9var_valueIdvEESt6vectorIiSaIiEEJRKS7_IS7_IS9_SaIS9_EESaISB_EESF_RN5Eigen6MatrixIS6_Lin1ELi1ELi0ELin1ELi1EEERNSH_IS6_Lin1ELin1ELi0ELin1ELin1EEESL_EEclEOS9_biPSoSF_SF_SJ_SL_SL_]+0x6a0): undefined reference to `tbb::interface7::internal::isolate_within_arena(tbb::interface7::internal::delegate_base&, long)'
collect2: error: ld returned 1 exit status
make/program:56: recipe for target '/tmp/RtmptOnD3w/model-73c712647f85' failed
make: *** [/tmp/RtmptOnD3w/model-73c712647f85] Error 1
make: *** [/tmp/RtmptOnD3w/model-73c712647f85] Error 1
Error: An error occured during compilation! See the message above for more information.
>
12

you’re asking the right question but I’m really not sure what further load paths are needed here for Ubuntu. @rok_cesnovar ?

13

Sorry to keep badgering you about this. Is there no path out of this problem? Happy to try anything to debug this. Anyone else tried this on AWS ec2? Truly desperate here.

14

you could try asking on the Stan slack - Join Stan on Slack | Slack

there is a problem with TBB lib not building properly on Windows - discussion starts here - Problems with installing cmdstanpy on win10 - #9 by ahartikainen

15

Thanks. I posted a message on Slack. The TBB problem you mentioned, that’s on Windows only or could be causing the issues I’m seeing on Ubuntu? Is the workaround on that thread something I should try? I don’t know much about conda.

16

I was able to fix it by removing the library libtbb2 from my system. I just ran

sudo apt list --installed | grep tbb

and removed anything that came up!

2 Likes
17

Thanks Karim and sorry for all the inconveniences.

Now that you found the cause that will also give us a chance to replicate your issue and find a simpler and less involved fix.

1 Like
18

No problem at all. Thanks for helping out on Slack.

19

Ran into the same issue (with 2.30.1) and ended up with removing system tbb lib from LD_LIBRARY_PATH.