Cmdstanpy Compilation Error - No such File or Directory

Hey all,

I recently started having issues that any time I try to compile a Stan model using cmdstanpy in a linux environment I get an error a “No such file or directory” error.

I am trying to initialize a cmdstanpy model via stan_model_compiled = cmdstanpy.CmdStanModel(stan_file=model_file_path) and have confirmed that I have a .stan file at the model_file_path prior to initializing the model (for example, I have a models/stan_models/model.stan file).

When initializing, I get this following error:

--- Translating Stan model to C++ code ---
bin/stanc  --o=/models/stan_models/model.hpp /models/stan_models/model.stan
Warning in '/models/stan_models/model.stan', line 4, column 4: Declaration
    of arrays by placing brackets after a variable name is deprecated and
    will be removed in Stan 2.32.0. Instead use the array keyword before the
    type. This can be changed automatically using the auto-format flag to
    stanc
Warning in '/models/stan_models/model.stan', line 8, column 4: Declaration
    of arrays by placing brackets after a variable name is deprecated and
    will be removed in Stan 2.32.0. Instead use the array keyword before the
    type. This can be changed automatically using the auto-format flag to
    stanc
Warning in '/models/stan_models/model.stan', line 10, column 4: Declaration
    of arrays by placing brackets after a variable name is deprecated and
    will be removed in Stan 2.32.0. Instead use the array keyword before the
    type. This can be changed automatically using the auto-format flag to
    stanc

--- Compiling, linking C++ code ---
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes      -I /opt/conda/envs/mlaasflow/include/    -O3 -I src -I stan/src -I stan/lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.78.0 -I stan/lib/stan_math/lib/sundials_6.1.1/include -I stan/lib/stan_math/lib/sundials_6.1.1/src/sundials    -DBOOST_DISABLE_ASSERTS   -DTBB_INTERFACE_NEW  -DTBB_INTERFACE_NEW     -c  -x c++ -o /models/stan_models/model.o /models/stan_models/model.hpp
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes      -I /opt/conda/envs/mlaasflow/include/    -O3 -I src -I stan/src -I stan/lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.78.0 -I stan/lib/stan_math/lib/sundials_6.1.1/include -I stan/lib/stan_math/lib/sundials_6.1.1/src/sundials    -DBOOST_DISABLE_ASSERTS   -DTBB_INTERFACE_NEW  -DTBB_INTERFACE_NEW           -Wl,-L,"/opt/conda/envs/mlaasflow/lib/" -Wl,-rpath,"/opt/conda/envs/mlaasflow/lib/" -Wl,--disable-new-dtags -ltbb    /models/stan_models/model.o src/cmdstan/main.o        -Wl,-L,"/opt/conda/envs/mlaasflow/lib/" -Wl,-rpath,"/opt/conda/envs/mlaasflow/lib/" -Wl,--disable-new-dtags -ltbb stan/lib/stan_math/lib/sundials_6.1.1/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_6.1.1/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_6.1.1/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_6.1.1/lib/libsundials_kinsol.a   -o /models/stan_models/model
/opt/conda/envs/mlaasflow/lib//libtbb.so: undefined reference to `std::__exception_ptr::exception_ptr::_M_release()@CXXABI_1.3.13'
/opt/conda/envs/mlaasflow/lib//libtbb.so: undefined reference to `std::__exception_ptr::exception_ptr::_M_addref()@CXXABI_1.3.13'
collect2: error: ld returned 1 exit status
make: *** [make/program:59: /models/stan_models/model] Error 1

Command ['make', '/models/stan_models/model']
	error during processing No such file or directory

If I handle the error and then do a listdir, then I can see that some new files have been created 'model.stan', 'model.hpp', 'model.o', but as the error suggests, there is no model (without extension) file.

Can someone help me out with this error, I’ve been looking at it for a while now and am at a loss for what is going wrong.

GCC Version: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Make Version: GNU Make 4.3
Cmdstanpy Version: 1.1.0
Cmdstan Version: 2.31.0

The errors here are related to the TBB:

/opt/conda/envs/mlaasflow/lib//libtbb.so: undefined reference to `std::__exception_ptr::exception_ptr::_M_release()@CXXABI_1.3.13'
/opt/conda/envs/mlaasflow/lib//libtbb.so: undefined reference to `std::__exception_ptr::exception_ptr::_M_addref()@CXXABI_1.3.13'

It looks you are using an external TBB installation, rather than the one bundled with cmdstan. Unfortunately we cannot guarantee compatibility with all versions of the TBB and recommend using the bundled sources where possible

What exactly am I supposed to use / how? I just installed cmdstanpy via conda and that’s what I got.

I was able to get things working by replicating an old version of my dependencies where I am specifically using cmdstanpy==0.9.77, libhwloc==2.8.0, and tbb/tbb-devel==2021.7.0 but using the newest version of any of these packages caused my compiles to fail.

Could you post the output of conda list for the failing setup?

Sure thing, here is the conda list from one of the failing environments I have that uses cmdstanpy==1.1.0:

Failing environment conda list output
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
aenum                     3.1.0                    pypi_0    pypi
aiohttp                   3.8.4                    pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
alabaster                 0.7.13                   pypi_0    pypi
anyio                     3.6.2                    pypi_0    pypi
anytree                   2.8.0                    pypi_0    pypi
argon2-cffi               21.3.0                   pypi_0    pypi
argon2-cffi-bindings      21.2.0                   pypi_0    pypi
arviz                     0.15.0                   pypi_0    pypi
asn1crypto                1.5.1                    pypi_0    pypi
astroid                   2.14.2                   pypi_0    pypi
asttokens                 2.2.1                    pypi_0    pypi
async-timeout             4.0.2                    pypi_0    pypi
attrs                     22.2.0                   pypi_0    pypi
aws-encryption-sdk        3.1.1                    pypi_0    pypi
babel                     2.11.0                   pypi_0    pypi
backcall                  0.2.0                    pypi_0    pypi
backports-shutil-get-terminal-size 1.0.0                    pypi_0    pypi
beautifulsoup4            4.11.2                   pypi_0    pypi
binutils_impl_linux-64    2.38                 h2a08ee3_1
binutils_linux-64         2.38.0               hc2dff05_0
black                     21.12b0                  pypi_0    pypi
bleach                    6.0.0                    pypi_0    pypi
bokeh                     2.4.3                    pypi_0    pypi
boto3                     1.17.52                  pypi_0    pypi
botocore                  1.20.52                  pypi_0    pypi
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.12.7            ha878542_0    conda-forge
cachetools                5.3.0                    pypi_0    pypi
certifi                   2022.12.7                pypi_0    pypi
cffi                      1.15.1                   pypi_0    pypi
cfgv                      3.3.1                    pypi_0    pypi
charset-normalizer        2.1.1                    pypi_0    pypi
click                     8.1.3                    pypi_0    pypi
cloudpickle               2.2.1                    pypi_0    pypi
cmdstan                   2.31.0               h1ffcc5c_0    conda-forge
cmdstanpy                 1.1.0              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
comm                      0.1.2                    pypi_0    pypi
commonmark                0.9.1                    pypi_0    pypi
confluent-kafka           2.0.2                    pypi_0    pypi
cons                      0.4.5                    pypi_0    pypi
contourpy                 1.0.7                    pypi_0    pypi
coverage                  7.1.0                    pypi_0    pypi
cramjam                   2.6.2                    pypi_0    pypi
croniter                  1.3.8                    pypi_0    pypi
cryptography              38.0.4                   pypi_0    pypi
cycler                    0.11.0                   pypi_0    pypi
cython                    0.29.33                  pypi_0    pypi
dacite                    1.8.0                    pypi_0    pypi
dask                      2023.2.0                 pypi_0    pypi
dask-kubernetes           2023.1.1                 pypi_0    pypi
debugpy                   1.6.6                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
defusedxml                0.7.1                    pypi_0    pypi
dependency-injector       4.41.0                   pypi_0    pypi
deprecation               2.0.7                    pypi_0    pypi
dill                      0.3.6                    pypi_0    pypi
diskcache                 5.4.0                    pypi_0    pypi
distlib                   0.3.6                    pypi_0    pypi
distributed               2023.2.0                 pypi_0    pypi
docker                    4.4.4                    pypi_0    pypi
docutils                  0.17.1                   pypi_0    pypi
entrypoints               0.4                      pypi_0    pypi
etuples                   0.3.8                    pypi_0    pypi
exceptiongroup            1.1.0                    pypi_0    pypi
execnet                   1.9.0                    pypi_0    pypi
executing                 1.2.0                    pypi_0    pypi
factory-boy               3.2.1                    pypi_0    pypi
faker                     13.7.0                   pypi_0    pypi
fastavro                  1.7.2                    pypi_0    pypi
fastjsonschema            2.16.2                   pypi_0    pypi
fastparquet               2023.2.0                 pypi_0    pypi
fastprogress              1.0.3                    pypi_0    pypi
filelock                  3.9.0                    pypi_0    pypi
flake8                    4.0.1                    pypi_0    pypi
fonttools                 4.38.0                   pypi_0    pypi
freezegun                 1.2.2                    pypi_0    pypi
frozenlist                1.3.3                    pypi_0    pypi
fsspec                    2022.7.1                 pypi_0    pypi
gcc_impl_linux-64         11.2.0               h1234567_1
gcc_linux-64              11.2.0               h5c386dc_0
google-auth               2.16.1                   pypi_0    pypi
grpcio                    1.51.3                   pypi_0    pypi
grpcio-tools              1.48.2                   pypi_0    pypi
gxx_impl_linux-64         11.2.0               h1234567_1
gxx_linux-64              11.2.0               hc2dff05_0
h11                       0.14.0                   pypi_0    pypi
h5netcdf                  1.1.0                    pypi_0    pypi
h5py                      3.8.0                    pypi_0    pypi
heapdict                  1.0.1                    pypi_0    pypi
httpcore                  0.16.3                   pypi_0    pypi
httpx                     0.23.3                   pypi_0    pypi
icu                       70.1                 h27087fc_0    conda-forge
identify                  2.5.18                   pypi_0    pypi
idna                      3.4                      pypi_0    pypi
imagesize                 1.4.1                    pypi_0    pypi
importlib-metadata        3.10.1                   pypi_0    pypi
importlib-resources       5.12.0                   pypi_0    pypi
infinity                  1.5                      pypi_0    pypi
iniconfig                 2.0.0                    pypi_0    pypi
ipykernel                 6.21.2                   pypi_0    pypi
ipython                   8.10.0                   pypi_0    pypi
ipython-genutils          0.2.0                    pypi_0    pypi
ipywidgets                8.0.4                    pypi_0    pypi
iso8601                   1.1.0                    pypi_0    pypi
isort                     5.12.0                   pypi_0    pypi
jedi                      0.18.2                   pypi_0    pypi
jinja2                    3.0.3                    pypi_0    pypi
jmespath                  0.10.0                   pypi_0    pypi
joblib                    1.2.0                    pypi_0    pypi
jsonschema                4.17.3                   pypi_0    pypi
jupyter                   1.0.0                    pypi_0    pypi
jupyter-client            7.4.9                    pypi_0    pypi
jupyter-console           6.6.1                    pypi_0    pypi
jupyter-core              5.2.0                    pypi_0    pypi
jupyter-server            1.23.6                   pypi_0    pypi
jupyterlab-pygments       0.2.2                    pypi_0    pypi
jupyterlab-widgets        3.0.5                    pypi_0    pypi
kernel-headers_linux-64   2.6.32              he073ed8_15    conda-forge
kiwisolver                1.4.4                    pypi_0    pypi
kopf                      1.36.0                   pypi_0    pypi
kubernetes                25.3.0                   pypi_0    pypi
kubernetes-asyncio        24.2.2                   pypi_0    pypi
lazy-object-proxy         1.9.0                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1
ldap3                     2.9                      pypi_0    pypi
libblas                   3.9.0           16_linux64_openblas    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-devel_linux-64     11.2.0               h1234567_1
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgfortran-ng            12.2.0              h69a702a_19    conda-forge
libgfortran5              12.2.0              h337968e_19    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libhwloc                  2.9.0                hd6dc26d_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
liblapack                 3.9.0           16_linux64_openblas    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
libsqlite                 3.40.0               h753d276_0    conda-forge
libstdcxx-devel_linux-64  11.2.0               h1234567_1
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libxml2                   2.10.3               h7463322_0    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
livereload                2.6.3                    pypi_0    pypi
locket                    1.0.0                    pypi_0    pypi
logical-unification       0.4.5                    pypi_0    pypi
make                      4.3                  hd18ef5c_1    conda-forge
markupsafe                2.1.2                    pypi_0    pypi
marshmallow               3.19.0                   pypi_0    pypi
marshmallow-oneofschema   3.0.1                    pypi_0    pypi
matplotlib                3.7.0                    pypi_0    pypi
matplotlib-inline         0.1.6                    pypi_0    pypi
matrixprofile-ts          0.0.9                    pypi_0    pypi
mccabe                    0.6.1                    pypi_0    pypi
minikanren                1.0.3                    pypi_0    pypi
mistune                   2.0.5                    pypi_0    pypi
msgpack                   1.0.4                    pypi_0    pypi
multidict                 6.0.4                    pypi_0    pypi
multipledispatch          0.6.0                    pypi_0    pypi
mypy                      1.0.0                    pypi_0    pypi
mypy-extensions           1.0.0                    pypi_0    pypi
nbclassic                 0.5.2                    pypi_0    pypi
nbclient                  0.7.2                    pypi_0    pypi
nbconvert                 7.2.9                    pypi_0    pypi
nbformat                  5.7.3                    pypi_0    pypi
nbsphinx                  0.8.12                   pypi_0    pypi
ncurses                   6.3                  h27087fc_1    conda-forge
nest-asyncio              1.5.6                    pypi_0    pypi
networkx                  3.0                      pypi_0    pypi
nodeenv                   1.7.0                    pypi_0    pypi
notebook                  6.5.2                    pypi_0    pypi
notebook-shim             0.2.2                    pypi_0    pypi
numpy                     1.24.2           py38h10c12cc_0    conda-forge
oauthlib                  3.2.2                    pypi_0    pypi
onelake-credentials-provider 0.6.5                    pypi_0    pypi
openssl                   3.0.8                h0b41bf4_0    conda-forge
oscrypto                  1.3.0                    pypi_0    pypi
packaging                 23.0                     pypi_0    pypi
pandas                    1.5.3                    pypi_0    pypi
pandoc                    2.3                      pypi_0    pypi
pandocfilters             1.5.0                    pypi_0    pypi
parso                     0.8.3                    pypi_0    pypi
partd                     1.3.0                    pypi_0    pypi
pathspec                  0.11.0                   pypi_0    pypi
patsy                     0.5.3                    pypi_0    pypi
pendulum                  2.1.2                    pypi_0    pypi
pexpect                   4.8.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pillow                    9.4.0                    pypi_0    pypi
pip                       23.0.1             pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10                   pypi_0    pypi
platformdirs              3.0.0                    pypi_0    pypi
plotly                    5.13.0                   pypi_0    pypi
pluggy                    1.0.0                    pypi_0    pypi
plumbum                   1.8.1                    pypi_0    pypi
ply                       3.11                     pypi_0    pypi
pre-commit                3.0.4                    pypi_0    pypi
prefect                   0.15.13                  pypi_0    pypi
prometheus-client         0.16.0                   pypi_0    pypi
prompt-toolkit            3.0.37                   pypi_0    pypi
protobuf                  3.20.2                   pypi_0    pypi
psutil                    5.9.4                    pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
py                        1.11.0                   pypi_0    pypi
pyarrow                   8.0.0                    pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pycodestyle               2.8.0                    pypi_0    pypi
pycparser                 2.21                     pypi_0    pypi
pycryptodomex             3.17                     pypi_0    pypi
pydantic                  1.8.2                    pypi_0    pypi
pyflakes                  2.4.0                    pypi_0    pypi
pygments                  2.14.0                   pypi_0    pypi
pyjwt                     2.4.0                    pypi_0    pypi
pykube-ng                 22.9.0                   pypi_0    pypi
pylint                    2.16.2                   pypi_0    pypi
pylint-fail-under         0.3.0                    pypi_0    pypi
pymc                      5.0.2                    pypi_0    pypi
pyopenssl                 23.0.0                   pypi_0    pypi
pyparsing                 3.0.9                    pypi_0    pypi
pyrsistent                0.19.3                   pypi_0    pypi
pytensor                  2.9.1                    pypi_0    pypi
pytest                    7.2.1                    pypi_0    pypi
pytest-asyncio            0.18.3                   pypi_0    pypi
pytest-cov                4.0.0                    pypi_0    pypi
pytest-forked             1.6.0                    pypi_0    pypi
pytest-mock               3.10.0                   pypi_0    pypi
pytest-timeout            2.1.0                    pypi_0    pypi
pytest-xdist              2.5.0                    pypi_0    pypi
python                    3.8.16          he550d4f_1_cpython    conda-forge
python-box                7.0.1                    pypi_0    pypi
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-graphviz           0.20.1                   pypi_0    pypi
python-json-logger        2.0.7                    pypi_0    pypi
python-slugify            8.0.0                    pypi_0    pypi
python-snappy             0.6.0                    pypi_0    pypi
python_abi                3.8                      3_cp38    conda-forge
pytz                      2022.7.1           pyhd8ed1ab_0    conda-forge
pytzdata                  2020.1                   pypi_0    pypi
pyyaml                    5.4.1                    pypi_0    pypi
pyzmq                     23.2.1                   pypi_0    pypi
qtconsole                 5.4.0                    pypi_0    pypi
qtpy                      2.3.0                    pypi_0    pypi
readline                  8.1.2                h0f457ee_0    conda-forge
recommonmark              0.7.1                    pypi_0    pypi
redis                     4.5.1                    pypi_0    pypi
reprint                   0.6.0                    pypi_0    pypi
requests                  2.28.2                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
reretry                   0.11.8                   pypi_0    pypi
rfc3986                   1.5.0                    pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
s3fs                      0.4.1                    pypi_0    pypi
s3transfer                0.3.7                    pypi_0    pypi
scikit-learn              0.24.1                   pypi_0    pypi
scipy                     1.10.1                   pypi_0    pypi
seaborn                   0.12.2                   pypi_0    pypi
send2trash                1.8.0                    pypi_0    pypi
setuptools                67.4.0                   pypi_0    pypi
six                       1.16.0             pyh6c4a22f_0    conda-forge
sniffio                   1.3.0                    pypi_0    pypi
snowballstemmer           2.2.0                    pypi_0    pypi
snowflake-connector-python 3.0.0                    pypi_0    pypi
sortedcontainers          2.4.0                    pypi_0    pypi
soupsieve                 2.4                      pypi_0    pypi
sphinx                    4.3.2                    pypi_0    pypi
sphinx-autobuild          2021.3.14                pypi_0    pypi
sphinx-rtd-theme          1.2.0                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.4                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
sphinxcontrib-htmlhelp    2.0.1                    pypi_0    pypi
sphinxcontrib-jquery      2.0.0                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.5                    pypi_0    pypi
stack-data                0.6.2                    pypi_0    pypi
statsmodels               0.13.5                   pypi_0    pypi
streamz                   0.6.4                    pypi_0    pypi
structlog                 22.3.0                   pypi_0    pypi
sysroot_linux-64          2.12                he073ed8_15    conda-forge
tabulate                  0.8.9                    pypi_0    pypi
tbb                       2021.8.0             hf52228f_0    conda-forge
tbb-devel                 2021.8.0             hf52228f_0    conda-forge
tblib                     1.7.0                    pypi_0    pypi
tenacity                  8.2.1                    pypi_0    pypi
terminado                 0.17.1                   pypi_0    pypi
text-unidecode            1.3                      pypi_0    pypi
threadpoolctl             3.1.0                    pypi_0    pypi
tinycss2                  1.2.1                    pypi_0    pypi
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2                   pypi_0    pypi
tomli                     1.2.3                    pypi_0    pypi
tomlkit                   0.11.6                   pypi_0    pypi
toolz                     0.12.0                   pypi_0    pypi
tornado                   6.2                      pypi_0    pypi
tox                       3.25.1                   pypi_0    pypi
tqdm                      4.64.1             pyhd8ed1ab_0    conda-forge
traces                    0.6.0                    pypi_0    pypi
traitlets                 5.9.0                    pypi_0    pypi
typeguard                 2.13.3                   pypi_0    pypi
typing-extensions         4.5.0                    pypi_0    pypi
urllib3                   1.26.14                  pypi_0    pypi
virtualenv                20.19.0                  pypi_0    pypi
wcwidth                   0.2.6                    pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
websocket-client          1.5.1                    pypi_0    pypi
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
widgetsnbextension        4.0.5                    pypi_0    pypi
wrapt                     1.14.1                   pypi_0    pypi
xarray                    2023.1.0                 pypi_0    pypi
xarray-einstats           0.5.1                    pypi_0    pypi
xz                        5.2.6                h166bdaf_0    conda-forge
yarl                      1.8.2                    pypi_0    pypi
zict                      2.2.0                    pypi_0    pypi
zipp                      3.14.0                   pypi_0    pypi

In case it’s helpful, here is a the output from an environment where I was able to run successfully

Successful environment conda list output
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
aenum                     3.1.0                    pypi_0    pypi
aiohttp                   3.8.4                    pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
alabaster                 0.7.13                   pypi_0    pypi
anyio                     3.6.2                    pypi_0    pypi
anytree                   2.8.0                    pypi_0    pypi
argon2-cffi               21.3.0                   pypi_0    pypi
argon2-cffi-bindings      21.2.0                   pypi_0    pypi
arviz                     0.15.0                   pypi_0    pypi
asn1crypto                1.5.1                    pypi_0    pypi
astroid                   2.14.2                   pypi_0    pypi
asttokens                 2.2.1                    pypi_0    pypi
async-timeout             4.0.2                    pypi_0    pypi
attrs                     22.2.0                   pypi_0    pypi
aws-encryption-sdk        3.1.1                    pypi_0    pypi
babel                     2.11.0                   pypi_0    pypi
backcall                  0.2.0                    pypi_0    pypi
backports-shutil-get-terminal-size 1.0.0                    pypi_0    pypi
beautifulsoup4            4.11.2                   pypi_0    pypi
binutils_impl_linux-64    2.39                 he00db2b_1    conda-forge
binutils_linux-64         2.39                h5fc0e48_11    conda-forge
black                     21.12b0                  pypi_0    pypi
bleach                    6.0.0                    pypi_0    pypi
bokeh                     2.4.3                    pypi_0    pypi
boto3                     1.17.52                  pypi_0    pypi
botocore                  1.20.52                  pypi_0    pypi
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.12.7            ha878542_0    conda-forge
cachetools                5.3.0                    pypi_0    pypi
certifi                   2022.12.7                pypi_0    pypi
cffi                      1.15.1                   pypi_0    pypi
cfgv                      3.3.1                    pypi_0    pypi
charset-normalizer        2.1.1                    pypi_0    pypi
click                     8.1.3                    pypi_0    pypi
cloudpickle               2.2.1                    pypi_0    pypi
cmdstan                   2.31.0               h1ffcc5c_0    conda-forge
cmdstanpy                 0.9.77             pyhd8ed1ab_0    conda-forge
colorama                  0.4.6                    pypi_0    pypi
comm                      0.1.2                    pypi_0    pypi
commonmark                0.9.1                    pypi_0    pypi
confluent-kafka           2.0.2                    pypi_0    pypi
cons                      0.4.5                    pypi_0    pypi
contourpy                 1.0.7                    pypi_0    pypi
coverage                  7.1.0                    pypi_0    pypi
cramjam                   2.6.2                    pypi_0    pypi
croniter                  1.3.8                    pypi_0    pypi
cryptography              38.0.4                   pypi_0    pypi
cudatoolkit               11.2.2              hbe64b41_11    conda-forge
cudnn                     8.1.0.77             h90431f1_0    conda-forge
cycler                    0.11.0                   pypi_0    pypi
cython                    0.29.33                  pypi_0    pypi
dacite                    1.8.0                    pypi_0    pypi
dask                      2023.2.0                 pypi_0    pypi
dask-kubernetes           2023.1.1                 pypi_0    pypi
debugpy                   1.6.6                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
defusedxml                0.7.1                    pypi_0    pypi
dependency-injector       4.41.0                   pypi_0    pypi
deprecation               2.0.7                    pypi_0    pypi
dill                      0.3.6                    pypi_0    pypi
diskcache                 5.4.0                    pypi_0    pypi
distlib                   0.3.6                    pypi_0    pypi
distributed               2023.2.0                 pypi_0    pypi
docker                    4.4.4                    pypi_0    pypi
docutils                  0.17.1                   pypi_0    pypi
entrypoints               0.4                      pypi_0    pypi
etuples                   0.3.8                    pypi_0    pypi
exceptiongroup            1.1.0                    pypi_0    pypi
execnet                   1.9.0                    pypi_0    pypi
executing                 1.2.0                    pypi_0    pypi
factory-boy               3.2.1                    pypi_0    pypi
faker                     13.7.0                   pypi_0    pypi
fastavro                  1.7.2                    pypi_0    pypi
fastjsonschema            2.16.2                   pypi_0    pypi
fastparquet               2023.2.0                 pypi_0    pypi
fastprogress              1.0.3                    pypi_0    pypi
filelock                  3.9.0                    pypi_0    pypi
flake8                    4.0.1                    pypi_0    pypi
fonttools                 4.38.0                   pypi_0    pypi
freezegun                 1.2.2                    pypi_0    pypi
frozenlist                1.3.3                    pypi_0    pypi
fsspec                    2022.7.1                 pypi_0    pypi
gcc_impl_linux-64         11.3.0              hab1b70f_19    conda-forge
gcc_linux-64              11.3.0              he6f903b_11    conda-forge
google-auth               2.16.1                   pypi_0    pypi
grpcio                    1.51.3                   pypi_0    pypi
grpcio-tools              1.48.2                   pypi_0    pypi
gxx_impl_linux-64         11.3.0              hab1b70f_19    conda-forge
gxx_linux-64              11.3.0              hc203a17_11    conda-forge
h11                       0.14.0                   pypi_0    pypi
h5netcdf                  1.1.0                    pypi_0    pypi
h5py                      3.8.0                    pypi_0    pypi
heapdict                  1.0.1                    pypi_0    pypi
httpcore                  0.16.3                   pypi_0    pypi
httpx                     0.23.3                   pypi_0    pypi
icu                       70.1                 h27087fc_0    conda-forge
identify                  2.5.18                   pypi_0    pypi
idna                      3.4                      pypi_0    pypi
imagesize                 1.4.1                    pypi_0    pypi
importlib-metadata        3.10.1                   pypi_0    pypi
importlib-resources       5.12.0                   pypi_0    pypi
infinity                  1.5                      pypi_0    pypi
iniconfig                 2.0.0                    pypi_0    pypi
ipykernel                 6.21.2                   pypi_0    pypi
ipython                   8.10.0                   pypi_0    pypi
ipython-genutils          0.2.0                    pypi_0    pypi
ipywidgets                8.0.4                    pypi_0    pypi
iso8601                   1.1.0                    pypi_0    pypi
isort                     5.12.0                   pypi_0    pypi
jedi                      0.18.2                   pypi_0    pypi
jinja2                    3.0.3                    pypi_0    pypi
jmespath                  0.10.0                   pypi_0    pypi
joblib                    1.2.0                    pypi_0    pypi
jsonschema                4.17.3                   pypi_0    pypi
jupyter                   1.0.0                    pypi_0    pypi
jupyter-client            7.4.9                    pypi_0    pypi
jupyter-console           6.6.1                    pypi_0    pypi
jupyter-core              5.2.0                    pypi_0    pypi
jupyter-server            1.23.6                   pypi_0    pypi
jupyterlab-pygments       0.2.2                    pypi_0    pypi
jupyterlab-widgets        3.0.5                    pypi_0    pypi
kernel-headers_linux-64   2.6.32              he073ed8_15    conda-forge
kiwisolver                1.4.4                    pypi_0    pypi
kopf                      1.36.0                   pypi_0    pypi
kubernetes                25.3.0                   pypi_0    pypi
kubernetes-asyncio        24.2.2                   pypi_0    pypi
lazy-object-proxy         1.9.0                    pypi_0    pypi
ld_impl_linux-64          2.39                 hcc3a1bd_1    conda-forge
ldap3                     2.9                      pypi_0    pypi
libblas                   3.9.0           16_linux64_openblas    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-devel_linux-64     11.3.0              h210ce93_19    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgfortran-ng            12.2.0              h69a702a_19    conda-forge
libgfortran5              12.2.0              h337968e_19    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libhwloc                  2.8.0                h32351e8_1    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
liblapack                 3.9.0           16_linux64_openblas    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
libsanitizer              11.3.0              h239ccf8_19    conda-forge
libsqlite                 3.40.0               h753d276_0    conda-forge
libstdcxx-devel_linux-64  11.3.0              h210ce93_19    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libxml2                   2.10.3               h7463322_0    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
livereload                2.6.3                    pypi_0    pypi
locket                    1.0.0                    pypi_0    pypi
logical-unification       0.4.5                    pypi_0    pypi
make                      4.3                  hd18ef5c_1    conda-forge
markupsafe                2.1.2                    pypi_0    pypi
marshmallow               3.19.0                   pypi_0    pypi
marshmallow-oneofschema   3.0.1                    pypi_0    pypi
matplotlib                3.7.0                    pypi_0    pypi
matplotlib-inline         0.1.6                    pypi_0    pypi
matrixprofile-ts          0.0.9                    pypi_0    pypi
mccabe                    0.6.1                    pypi_0    pypi
minikanren                1.0.3                    pypi_0    pypi
mistune                   2.0.5                    pypi_0    pypi
msgpack                   1.0.4                    pypi_0    pypi
multidict                 6.0.4                    pypi_0    pypi
multipledispatch          0.6.0                    pypi_0    pypi
mypy                      1.0.0                    pypi_0    pypi
mypy-extensions           1.0.0                    pypi_0    pypi
nbclassic                 0.5.2                    pypi_0    pypi
nbclient                  0.7.2                    pypi_0    pypi
nbconvert                 7.2.9                    pypi_0    pypi
nbformat                  5.7.3                    pypi_0    pypi
nbsphinx                  0.8.12                   pypi_0    pypi
ncurses                   6.3                  h27087fc_1    conda-forge
nest-asyncio              1.5.6                    pypi_0    pypi
networkx                  3.0                      pypi_0    pypi
nodeenv                   1.7.0                    pypi_0    pypi
notebook                  6.5.2                    pypi_0    pypi
notebook-shim             0.2.2                    pypi_0    pypi
numpy                     1.24.2           py38h10c12cc_0    conda-forge
oauthlib                  3.2.2                    pypi_0    pypi
onelake-credentials-provider 0.6.5                    pypi_0    pypi
openssl                   3.0.8                h0b41bf4_0    conda-forge
oscrypto                  1.3.0                    pypi_0    pypi
packaging                 23.0                     pypi_0    pypi
pandas                    1.5.3                    pypi_0    pypi
pandoc                    2.3                      pypi_0    pypi
pandocfilters             1.5.0                    pypi_0    pypi
parso                     0.8.3                    pypi_0    pypi
partd                     1.3.0                    pypi_0    pypi
pathspec                  0.11.0                   pypi_0    pypi
patsy                     0.5.3                    pypi_0    pypi
pendulum                  2.1.2                    pypi_0    pypi
pexpect                   4.8.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pillow                    9.4.0                    pypi_0    pypi
pip                       23.0.1             pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10                   pypi_0    pypi
platformdirs              3.0.0                    pypi_0    pypi
plotly                    5.13.0                   pypi_0    pypi
pluggy                    1.0.0                    pypi_0    pypi
plumbum                   1.8.1                    pypi_0    pypi
ply                       3.11                     pypi_0    pypi
pre-commit                3.0.4                    pypi_0    pypi
prefect                   0.15.13                  pypi_0    pypi
prometheus-client         0.16.0                   pypi_0    pypi
prompt-toolkit            3.0.37                   pypi_0    pypi
protobuf                  3.20.2                   pypi_0    pypi
psutil                    5.9.4                    pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
py                        1.11.0                   pypi_0    pypi
pyarrow                   8.0.0                    pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pycodestyle               2.8.0                    pypi_0    pypi
pycparser                 2.21                     pypi_0    pypi
pycryptodomex             3.17                     pypi_0    pypi
pydantic                  1.8.2                    pypi_0    pypi
pyflakes                  2.4.0                    pypi_0    pypi
pygments                  2.14.0                   pypi_0    pypi
pyjwt                     2.4.0                    pypi_0    pypi
pykube-ng                 22.9.0                   pypi_0    pypi
pylint                    2.16.2                   pypi_0    pypi
pylint-fail-under         0.3.0                    pypi_0    pypi
pymc                      5.0.2                    pypi_0    pypi
pyopenssl                 23.0.0                   pypi_0    pypi
pyparsing                 3.0.9                    pypi_0    pypi
pyrsistent                0.19.3                   pypi_0    pypi
pytensor                  2.9.1                    pypi_0    pypi
pytest                    7.2.1                    pypi_0    pypi
pytest-asyncio            0.18.3                   pypi_0    pypi
pytest-cov                4.0.0                    pypi_0    pypi
pytest-forked             1.6.0                    pypi_0    pypi
pytest-mock               3.10.0                   pypi_0    pypi
pytest-timeout            2.1.0                    pypi_0    pypi
pytest-xdist              2.5.0                    pypi_0    pypi
python                    3.8.16          he550d4f_1_cpython    conda-forge
python-box                7.0.1                    pypi_0    pypi
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-graphviz           0.20.1                   pypi_0    pypi
python-json-logger        2.0.7                    pypi_0    pypi
python-slugify            8.0.0                    pypi_0    pypi
python-snappy             0.6.0                    pypi_0    pypi
python_abi                3.8                      3_cp38    conda-forge
pytz                      2022.7.1           pyhd8ed1ab_0    conda-forge
pytzdata                  2020.1                   pypi_0    pypi
pyyaml                    5.4.1                    pypi_0    pypi
pyzmq                     23.2.1                   pypi_0    pypi
qtconsole                 5.4.0                    pypi_0    pypi
qtpy                      2.3.0                    pypi_0    pypi
readline                  8.1.2                h0f457ee_0    conda-forge
recommonmark              0.7.1                    pypi_0    pypi
redis                     4.5.1                    pypi_0    pypi
reprint                   0.6.0                    pypi_0    pypi
requests                  2.28.2                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
reretry                   0.11.8                   pypi_0    pypi
rfc3986                   1.5.0                    pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
s3fs                      0.4.1                    pypi_0    pypi
s3transfer                0.3.7                    pypi_0    pypi
scikit-learn              0.24.1                   pypi_0    pypi
scipy                     1.10.1                   pypi_0    pypi
seaborn                   0.12.2                   pypi_0    pypi
send2trash                1.8.0                    pypi_0    pypi
setuptools                67.4.0                   pypi_0    pypi
six                       1.16.0             pyh6c4a22f_0    conda-forge
sniffio                   1.3.0                    pypi_0    pypi
snowballstemmer           2.2.0                    pypi_0    pypi
snowflake-connector-python 3.0.0                    pypi_0    pypi
sortedcontainers          2.4.0                    pypi_0    pypi
soupsieve                 2.4                      pypi_0    pypi
sphinx                    4.3.2                    pypi_0    pypi
sphinx-autobuild          2021.3.14                pypi_0    pypi
sphinx-rtd-theme          1.2.0                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.4                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
sphinxcontrib-htmlhelp    2.0.1                    pypi_0    pypi
sphinxcontrib-jquery      2.0.0                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.5                    pypi_0    pypi
stack-data                0.6.2                    pypi_0    pypi
statsmodels               0.13.5                   pypi_0    pypi
streamz                   0.6.4                    pypi_0    pypi
structlog                 22.3.0                   pypi_0    pypi
sysroot_linux-64          2.12                he073ed8_15    conda-forge
tabulate                  0.8.9                    pypi_0    pypi
tbb                       2021.7.0             h924138e_1    conda-forge
tbb-devel                 2021.7.0             h924138e_1    conda-forge
tblib                     1.7.0                    pypi_0    pypi
tenacity                  8.2.1                    pypi_0    pypi
terminado                 0.17.1                   pypi_0    pypi
text-unidecode            1.3                      pypi_0    pypi
threadpoolctl             3.1.0                    pypi_0    pypi
tinycss2                  1.2.1                    pypi_0    pypi
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2                   pypi_0    pypi
tomli                     1.2.3                    pypi_0    pypi
tomlkit                   0.11.6                   pypi_0    pypi
toolz                     0.12.0                   pypi_0    pypi
tornado                   6.2                      pypi_0    pypi
tox                       3.25.1                   pypi_0    pypi
tqdm                      4.64.1                   pypi_0    pypi
traces                    0.6.0                    pypi_0    pypi
traitlets                 5.9.0                    pypi_0    pypi
typeguard                 2.13.3                   pypi_0    pypi
typing-extensions         4.5.0                    pypi_0    pypi
ujson                     5.7.0            py38h8dc9893_0    conda-forge
urllib3                   1.26.14                  pypi_0    pypi
virtualenv                20.19.0                  pypi_0    pypi
wcwidth                   0.2.6                    pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
websocket-client          1.5.1                    pypi_0    pypi
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
widgetsnbextension        4.0.5                    pypi_0    pypi
wrapt                     1.14.1                   pypi_0    pypi
xarray                    2023.1.0                 pypi_0    pypi
xarray-einstats           0.5.1                    pypi_0    pypi
xz                        5.2.6                h166bdaf_0    conda-forge
yarl                      1.8.2                    pypi_0    pypi
zict                      2.2.0                    pypi_0    pypi
zipp                      3.14.0                   pypi_0    pypi

I’m on Ubuntu 22.04 so I can’t recreate your exact environment but the one you list as failing works for me. I suspect the issue (/opt/conda/envs/mlaasflow/lib//libtbb.so: undefined reference to std::__exception_ptr::exception_ptr::_M_release()@CXXABI_1.3.13) could be because your system gcc or glibc are older than the most recent TBB versions support. I would either pin 2021.7, or build from source as @andrjohns suggests