Error while loading shared libraries: libtbb.so.12: cannot open shared object file: No such file or directory

Hi,
I was trying to test out our stan model in a new conda env and got an error message (can be found below). This used to worked fine in the old env. I’ve tried install_cmdstan and source install, but wasn’t able to resolve the issue. The stan code is here, the model is here and a unittest can be found here. Any help would be much appreciated.

(birdman) [djin@worker1003 tests]$ python test_interactive.py 
07:30:37 - cmdstanpy - INFO - Chain [1] start processing
07:30:37 - cmdstanpy - INFO - Chain [2] start processing
07:30:37 - cmdstanpy - INFO - Chain [3] start processing
07:30:37 - cmdstanpy - INFO - Chain [4] start processing
Chain [3] /mnt/ceph/users/djin/git/q2-differential/q2_differential/assets/disease_single: error while loading shared libraries: libtbb.so.12: cannot open shared object file: No such file or directory
Chain [4] /mnt/ceph/users/djin/git/q2-differential/q2_differential/assets/disease_single: error while loading shared libraries: libtbb.so.12: cannot open shared object file: No such file or directory
Chain [1] /mnt/ceph/users/djin/git/q2-differential/q2_differential/assets/disease_single: error while loading shared libraries: libtbb.so.12: cannot open shared object file: No such file or directory
Chain [1] 
07:30:37 - cmdstanpy - INFO - Chain [3] done processing
Chain [2] /mnt/ceph/users/djin/git/q2-differential/q2_differential/assets/disease_single: error while loading shared libraries: libtbb.so.12: cannot open shared object file: No such file or directory
07:30:37 - cmdstanpy - INFO - Chain [1] done processing
07:30:37 - cmdstanpy - ERROR - Chain [3] error: program not found Key has expired
07:30:37 - cmdstanpy - INFO - Chain [4] done processing
07:30:37 - cmdstanpy - ERROR - Chain [1] error: program not found Key has expired
07:30:37 - cmdstanpy - ERROR - Chain [4] error: program not found Key has expired
07:30:37 - cmdstanpy - INFO - Chain [2] done processing
07:30:37 - cmdstanpy - ERROR - Chain [2] error: program not found Key has expired
Traceback (most recent call last):
  File "/mnt/ceph/users/djin/git/q2-differential/q2_differential/tests/test_interactive.py", line 79, in <module>
    samples.append(_single_func(m))
  File "/mnt/ceph/users/djin/git/q2-differential/q2_differential/_model.py", line 270, in _single_func
    m.fit_model({'adapt_delta': 0.95, 'max_treedepth': 20})
  File "/mnt/home/djin/miniconda3/envs/birdman/lib/python3.10/site-packages/birdman/model_base.py", line 158, in fit_model
    _fit = self.sm.sample(
  File "/mnt/home/djin/miniconda3/envs/birdman/lib/python3.10/site-packages/cmdstanpy/model.py", line 1190, in sample
    raise RuntimeError(msg)
RuntimeError: Error during sampling:

Command and output files:
RunSet: chains=4, chain_ids=[1, 2, 3, 4], num_processes=4
 cmd (chain 1):
	['/mnt/ceph/users/djin/git/q2-differential/q2_differential/assets/disease_single', 'id=1', 'random', 'seed=42', 'data', 'file=/tmp/tmpadr_spag/2y9m4tqe.json', 'output', 'file=/tmp/tmpadr_spag/disease_single3urpgqmz/disease_single-20220928073037_1.csv', 'method=sample', 'num_samples=100', 'num_warmup=1000', 'algorithm=hmc', 'engine=nuts', 'max_depth=20', 'adapt', 'engaged=1', 'delta=0.95']
 retcodes=[127, 127, 127, 127]
 per-chain output files (showing chain 1 only):
 csv_file:
	/tmp/tmpadr_spag/disease_single3urpgqmz/disease_single-20220928073037_1.csv
 console_msgs (if any):
	/tmp/tmpadr_spag/disease_single3urpgqmz/disease_single-20220928073037_0-stdout.txt
Consider re-running with show_console=True if the above output is unclear!

Can you share the results of conda info and conda list?

Did you install cmdstan from conda, or is this a custom installation?

I installed cmdstan from source and it’s cmdstan v2.30.1.
And here’s the conda info and conda list

 active environment : q2_matchmaker
    active env location : /mnt/home/djin/miniconda3/envs/q2_matchmaker
            shell level : 1
       user config file : /mnt/home/djin/.condarc
 populated config files : /mnt/home/djin/.condarc
          conda version : 4.12.0
    conda-build version : not installed
         python version : 3.9.12.final.0
       virtual packages : __linux=5.4.203.1=0
                          __glibc=2.28=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /mnt/home/djin/miniconda3  (writable)
      conda av data dir : /mnt/home/djin/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/bioconda/linux-64
                          https://conda.anaconda.org/bioconda/noarch
                          https://mirrors.ustc.edu.cn/anaconda/cloud/menpo/linux-64
                          https://mirrors.ustc.edu.cn/anaconda/cloud/menpo/noarch
                          https://mirrors.ustc.edu.cn/anaconda/cloud/bioconda/linux-64
                          https://mirrors.ustc.edu.cn/anaconda/cloud/bioconda/noarch
                          https://mirrors.ustc.edu.cn/anaconda/cloud/msys2/linux-64
                          https://mirrors.ustc.edu.cn/anaconda/cloud/msys2/noarch
                          https://mirrors.ustc.edu.cn/anaconda/cloud/conda-forge/linux-64
                          https://mirrors.ustc.edu.cn/anaconda/cloud/conda-forge/noarch
                          https://mirrors.ustc.edu.cn/anaconda/pkgs/free/linux-64
                          https://mirrors.ustc.edu.cn/anaconda/pkgs/free/noarch
                          https://mirrors.ustc.edu.cn/anaconda/pkgs/main/linux-64
                          https://mirrors.ustc.edu.cn/anaconda/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /mnt/home/djin/miniconda3/pkgs
                          /mnt/home/djin/.conda/pkgs
       envs directories : /mnt/home/djin/miniconda3/envs
                          /mnt/home/djin/.conda/envs
               platform : linux-64
             user-agent : conda/4.12.0 requests/2.27.1 CPython/3.9.12 Linux/5.4.203.1.fi rocky/8.6 glibc/2.28
                UID:GID : 1783:1783
             netrc file : None
           offline mode : False
# packages in environment at /mnt/home/djin/miniconda3/envs/q2_matchmaker:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
alsa-lib                  1.2.3.2              h166bdaf_0    conda-forge
argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0          py310h5764c6d_2    conda-forge
arviz                     0.12.1                   pypi_0    pypi
asttokens                 2.0.8              pyhd8ed1ab_0    conda-forge
attrs                     22.1.0             pyh71513ae_1    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.11.1             pyha770c72_0    conda-forge
biom-format               2.1.12          py310hde88566_1    conda-forge
birdman                   0.1.0                    pypi_0    pypi
bleach                    5.0.1              pyhd8ed1ab_0    conda-forge
brotli                    1.0.9                h166bdaf_7    conda-forge
brotli-bin                1.0.9                h166bdaf_7    conda-forge
brotlipy                  0.7.0           py310h5764c6d_1004    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.9.24            ha878542_0    conda-forge
cachecontrol              0.12.11            pyhd8ed1ab_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
certifi                   2022.9.24          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h255011f_0    conda-forge
cftime                    1.6.2                    pypi_0    pypi
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
click                     8.1.3           py310hff52083_0    conda-forge
cmdstanpy                 1.0.7                    pypi_0    pypi
cryptography              37.0.1          py310h9ce1e76_0  
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cython                    0.29.32         py310hd8f1fbe_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
debugpy                   1.6.3           py310hd8f1fbe_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
executing                 0.10.0             pyhd8ed1ab_0    conda-forge
expat                     2.4.8                h27087fc_0    conda-forge
flit-core                 3.7.1              pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.0               h8e229c2_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.36.0          py310h5764c6d_0    conda-forge
freetype                  2.12.1               hca18f0e_0    conda-forge
future                    0.18.2          py310hff52083_5    conda-forge
gettext                   0.19.8.1          h73d1719_1008    conda-forge
glib                      2.72.1               h6239696_0    conda-forge
glib-tools                2.72.1               h6239696_0    conda-forge
gst-plugins-base          1.20.2               hcf0ee16_0    conda-forge
gstreamer                 1.20.3               hd4edc92_0    conda-forge
h5py                      3.7.0           nompi_py310h416281c_101    conda-forge
hdf5                      1.12.2          nompi_h4df4325_100    conda-forge
hdmedians                 0.14.2          py310hde88566_2    conda-forge
icu                       69.1                 h9c3ff4c_0    conda-forge
idna                      3.3                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.11.4          py310hff52083_0    conda-forge
importlib_resources       5.9.0              pyhd8ed1ab_0    conda-forge
iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
ipykernel                 6.15.1             pyh210e3f2_0    conda-forge
ipython                   8.4.0           py310hff52083_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                8.0.1              pyhd8ed1ab_0    conda-forge
jedi                      0.18.1             pyhd8ed1ab_2    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
joblib                    1.1.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h166bdaf_2    conda-forge
jsonschema                4.14.0             pyhd8ed1ab_0    conda-forge
jupyter                   1.0.0           py310hff52083_7    conda-forge
jupyter_client            7.3.4              pyhd8ed1ab_0    conda-forge
jupyter_console           6.4.4              pyhd8ed1ab_0    conda-forge
jupyter_core              4.11.1          py310hff52083_0    conda-forge
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        3.0.2              pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4           py310hbf28c38_0    conda-forge
krb5                      1.19.3               h08a2579_0    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libblas                   3.9.0           16_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_7    conda-forge
libbrotlidec              1.0.9                h166bdaf_7    conda-forge
libbrotlienc              1.0.9                h166bdaf_7    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libclang                  13.0.1          default_hc23dcda_0    conda-forge
libcurl                   7.83.1               h2283fc2_0    conda-forge
libdeflate                1.13                 h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               h28343ad_4    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.1.0              h8d9b700_16    conda-forge
libgfortran-ng            12.1.0              h69a702a_16    conda-forge
libgfortran5              12.1.0              hdcd56e2_16    conda-forge
libglib                   2.72.1               h2d90d5f_0    conda-forge
libgomp                   12.1.0              h8d9b700_16    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblapack                 3.9.0           16_linux64_openblas    conda-forge
libllvm13                 13.0.1               hf817b99_2    conda-forge
libnghttp2                1.47.0               hff17c54_1    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.21          pthreads_h78a6416_2    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.37               h753d276_4    conda-forge
libpq                     14.5                 he2d8382_0    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libsqlite                 3.39.2               h753d276_1    conda-forge
libssh2                   1.10.0               hf14f497_3    conda-forge
libstdcxx-ng              12.1.0              ha89aaad_16    conda-forge
libtiff                   4.4.0                h0e0dad5_3    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp-base              1.2.4                h166bdaf_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.9.12               h885dcf4_1    conda-forge
libxslt                   1.1.33               h0ef7038_3    conda-forge
libzlib                   1.2.12               h166bdaf_2    conda-forge
lockfile                  0.12.2                     py_1    conda-forge
lxml                      4.8.0           py310h5764c6d_2    conda-forge
markupsafe                2.1.1           py310h5764c6d_1    conda-forge
matplotlib-base           3.5.3           py310h5701ce4_1    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mistune                   0.8.4           py310h6acc77f_1005    conda-forge
msgpack-python            1.0.4           py310hbf28c38_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.30               h26416b9_0    conda-forge
mysql-libs                8.0.30               hbc51c84_0    conda-forge
natsort                   8.1.0              pyhd8ed1ab_0    conda-forge
nbclient                  0.6.6              pyhd8ed1ab_0    conda-forge
nbconvert                 6.5.3              pyhd8ed1ab_0    conda-forge
nbconvert-core            6.5.3              pyhd8ed1ab_0    conda-forge
nbconvert-pandoc          6.5.3              pyhd8ed1ab_0    conda-forge
nbformat                  5.4.0              pyhd8ed1ab_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
nest-asyncio              1.5.5              pyhd8ed1ab_0    conda-forge
netcdf4                   1.6.1                    pypi_0    pypi
notebook                  6.4.12             pyha770c72_0    conda-forge
nspr                      4.32                 h9c3ff4c_1    conda-forge
nss                       3.78                 h2350873_0    conda-forge
numpy                     1.23.2          py310h53a5b5f_0    conda-forge
openjpeg                  2.5.0                h7d73246_1    conda-forge
openssl                   3.0.5                h166bdaf_2    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.4.3           py310h769672d_0    conda-forge
pandoc                    2.19.1               ha770c72_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
patsy                     0.5.2              pyhd8ed1ab_0    conda-forge
pcre                      8.45                 h9c3ff4c_0    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.2.0           py310hbd86126_2    conda-forge
pip                       22.2.2             pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_0    conda-forge
pluggy                    1.0.0           py310hff52083_3    conda-forge
prometheus_client         0.14.1             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.30             pyha770c72_0    conda-forge
prompt_toolkit            3.0.30               hd8ed1ab_0    conda-forge
psutil                    5.9.1           py310h5764c6d_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
py                        1.11.0             pyh6c4a22f_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.13.0             pyhd8ed1ab_0    conda-forge
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyqt                      5.12.3          py310hff52083_8    conda-forge
pyqt-impl                 5.12.3          py310h1f8e252_8    conda-forge
pyqt5-sip                 4.19.18         py310h122e73d_8    conda-forge
pyqtchart                 5.12            py310hfcd6d55_8    conda-forge
pyqtwebengine             5.12.1          py310hfcd6d55_8    conda-forge
pyrsistent                0.18.1          py310h5764c6d_1    conda-forge
pysocks                   1.7.1           py310hff52083_5    conda-forge
pytest                    7.1.2           py310hff52083_0    conda-forge
python                    3.10.5          ha86cf86_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.16.1             pyhd8ed1ab_0    conda-forge
python_abi                3.10                    2_cp310    conda-forge
pytz                      2022.2.1           pyhd8ed1ab_0    conda-forge
pyzmq                     23.2.1          py310h330234f_0    conda-forge
q2-differential           0.1.0                     dev_0    <develop>
qt                        5.12.9               h1304e3e_6    conda-forge
qtconsole                 5.3.1              pyhd8ed1ab_0    conda-forge
qtconsole-base            5.3.1              pyha770c72_0    conda-forge
qtpy                      2.2.0              pyhd8ed1ab_0    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
requests                  2.28.1             pyhd8ed1ab_0    conda-forge
scikit-bio                0.5.7           py310h96516ba_0    conda-forge
scikit-learn              1.1.2           py310h0c3af53_0    conda-forge
scipy                     1.8.1                    pypi_0    pypi
seaborn                   0.11.2               hd8ed1ab_0    conda-forge
seaborn-base              0.11.2             pyhd8ed1ab_0    conda-forge
send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
setuptools                65.2.0          py310hff52083_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
sqlite                    3.39.2               h4ff8645_1    conda-forge
stack_data                0.4.0              pyhd8ed1ab_0    conda-forge
statsmodels               0.13.2          py310hde88566_0    conda-forge
terminado                 0.15.0          py310hff52083_0    conda-forge
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tinycss2                  1.1.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
torch                     1.12.1                   pypi_0    pypi
tornado                   6.2             py310h5764c6d_0    conda-forge
tqdm                      4.64.1                   pypi_0    pypi
traitlets                 5.3.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.3.0                    pypi_0    pypi
tzdata                    2022c                h191b570_0    conda-forge
ujson                     5.5.0                    pypi_0    pypi
unicodedata2              14.0.0          py310h5764c6d_1    conda-forge
urllib3                   1.26.11            pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
widgetsnbextension        4.0.2              pyhd8ed1ab_0    conda-forge
xarray                    0.21.0                   pypi_0    pypi
xarray-einstats           0.3.0                    pypi_0    pypi
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zeromq                    4.3.4                h9c3ff4c_1    conda-forge
zipp                      3.8.1              pyhd8ed1ab_0    conda-forge
zlib                      1.2.12               h166bdaf_2    conda-forge
zstd                      1.5.2                h6239696_4    conda-forge

Did you re-compile the model after switching to the new environment?

I reran the unittest and not sure if I have re-compiled the model. How do I make sure to re-compile it? Thanks!

The easiest thing is to go to the folder where the model is and delete the executable (this has the same name as the .stan file, but no extension)

Re-compile solved the issue. Thanks a lot!