I have been fitting stan models on our HPC successfully over the past few weeks, but all of a sudden, I am getting an error message when running cmdstan_model():
/usr/bin/ld: cannot find -l/cm/shared/apps/hdf5/1.10.1/include
collect2: error: ld returned 1 exit status
make: *** [make/program:66: /tmp/RtmpfgFnTA/model-25e5a76af44a6d] Error 1
Error: An error occured during compilation! See the message above for more information.
Execution halted
We are using cmdstan version 2.34.1 and cmdstanr version 0.7.1, and R version 4.3.2. We can successfully compile a stan file outside of R (using make), so it does not seem to be an issue with cmdstan. It is not an issue related to permissions, since the admin is getting the same errors. Again, this was all working just fine a few days ago. We did reinstall cmdstanr (though not cmdstan) to see if that helped.
If anyone has seen anything like this before, any insight would be welcome. Also, let me know if any more information would be helpful.
It seems like something in your R environment might be touching LDFLAGS/LDLIBS - Stan doesn’t depend on hdf5. Could you share sessionInfo()
Here is the session environment:
R version 4.3.2 (2023-10-31)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux 8.6 (Ootpa)
Matrix products: default
BLAS: /gpfs/share/apps/R/4.3.2/lib64/R/lib/libRblas.so
LAPACK: /gpfs/share/apps/R/4.3.2/lib64/R/lib/libRlapack.so; LAPACK version 3.11.0
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
time zone: America/New_York
tzcode source: system (glibc)
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] cmdstanr_0.7.1
loaded via a namespace (and not attached):
[1] tensorA_0.36.2.1 backports_1.4.1 utf8_1.2.4
[4] R6_2.5.1 xfun_0.43 magrittr_2.0.3
[7] posterior_1.5.0 glue_1.7.0 abind_1.4-5
[10] tibble_3.2.1 knitr_1.46 pkgconfig_2.0.3
[13] generics_0.1.3 lifecycle_1.0.4 cli_3.6.2
[16] fansi_1.0.6 vctrs_0.6.5 compiler_4.3.2
[19] tools_4.3.2 distributional_0.4.0 checkmate_2.3.1
[22] pillar_1.9.0 jsonlite_1.8.8 rlang_1.1.3
We’ve tried numerous things, including re-installing cmdstan. We are now getting an error message after the model appears to have estimated:
Chain 2 finished in 0.1 seconds.
Chain 3 finished in 0.1 seconds.
Chain 4 finished in 0.1 seconds.
All 4 chains finished successfully.
Mean chain execution time: 0.1 seconds.
Total execution time: 0.4 seconds.
Error in !is.null(csv_contents$metadata$save_warmup) && csv_contents$metadata$save_warmup :
invalid 'y' type in 'x && y'
Calls: <Anonymous> -> <Anonymous> -> initialize -> <Anonymous>
Execution halted
Any suggestions? I was also wondering when version 2.35.0 might be released - hoping that will solve our problems.
That message is happening inside cmdstanr, so it appears you’ve resolved whatever linker issue was occuring, and now it’s a plain old R issue (which sadly means I can’t help all that much)
@rok_cesnovar or @jonah might be able to
Thanks - do you know when cmdstan 2.35.0 is slated to be released? Summer, fall, winter?
The release candidates for 2.35.0 were posted earlier today. If no major new issues arise, the release will happen next week
We installed 2.35.0-rc2. Unfortunately, the admin could not compile the Bernoulli example:
$ make examples/bernoulli/bernoulli
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -I stan/lib/stan_math/lib/tbb_2020.3/include -O3 -I src -I stan/src -I stan/lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.4.0 -I stan/lib/stan_math/lib/boost_1.84.0 -I stan/lib/stan_math/lib/sundials_6.1.1/include -I stan/lib/stan_math/lib/sundials_6.1.1/src/sundials -DBOOST_DISABLE_ASSERTS -c -MT src/cmdstan/main.o -MM -E -MG -MP -MF src/cmdstan/main.d src/cmdstan/main.cpp
--- Translating Stan model to C++ code ---
bin/stanc --o=examples/bernoulli/bernoulli.hpp examples/bernoulli/bernoulli.stan
bash: bin/stanc: cannot execute binary file: Exec format error
make: *** [make/program:64: examples/bernoulli/bernoulli.hpp] Error 126
Are you running on a non-x86-64 architecture?
This is running on a high performance cluster:
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux 8.6 (Ootpa)
That error sounds like what I would expect if you downloaded a cmdstan meant for a non-x86 architecture, is that possible?
It is possible. They used cmdstan-2.35.0-rc2-linux-ppc64el.tar.gz. Is that correct?
OK - we installed using the source code, and everything compiled fine. We were able to compile the example without any errors. In my test case, I believe the stan file is compiling and the sampling is working - there is just something going on at the end of the sampling that causes an error, which I told you about before:
Error in !is.null(csv_contents$metadata$save_warmup) && csv_contents$metadata$save_warmup :
invalid 'y' type in 'x && y'
Calls: <Anonymous> -> <Anonymous> -> initialize -> <Anonymous>
Execution halted
I know you said this is not “your” error anymore - how can I get this question to the correct folks?
And in case this helpful - here is the detailed output:
Loading r/4.3.2
Loading requirement: hdf5/1.10.1 gcc/10.2.0 lapack/gcc/64/3.8.0 jags/4.3.0
jdk/8u181 gsl/2.5 udunits/2.2.28 imagemagick/6.9.10 icu/70.1 tiff/4.1.0
geos/3.9.1 pcre2/10.35 gcc/8.1.0 proj/6.1.0-gcc8.1.0 sqlite/3.28.0.1
gdal/3.8.4 python/cpu/3.7.2 bzip2/1.1.0 curl/7.60.0 libjpeg/8 jbig/2.1
webp/1.2.1 gmp/6.1.2 glpk/4.55 libpng/1.6.37
This is cmdstanr version 0.7.1
- CmdStanR documentation and vignettes: mc-stan.org/cmdstanr
- Use set_cmdstan_path() to set the path to CmdStan
- Use install_cmdstan() to install CmdStan
R version 4.3.2 (2023-10-31)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux 8.6 (Ootpa)
Matrix products: default
BLAS: /gpfs/share/apps/R/4.3.2/lib64/R/lib/libRblas.so
LAPACK: /gpfs/share/apps/R/4.3.2/lib64/R/lib/libRlapack.so; LAPACK version 3.11.0
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
time zone: America/New_York
tzcode source: system (glibc)
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] cmdstanr_0.7.1
loaded via a namespace (and not attached):
[1] tensorA_0.36.2.1 backports_1.4.1 utf8_1.2.4
[4] R6_2.5.1 xfun_0.43 magrittr_2.0.3
[7] posterior_1.5.0 glue_1.7.0 abind_1.4-5
[10] tibble_3.2.1 knitr_1.46 pkgconfig_2.0.3
[13] generics_0.1.3 lifecycle_1.0.4 ps_1.7.6
[16] cli_3.6.2 processx_3.8.4 fansi_1.0.6
[19] vctrs_0.6.5 compiler_4.3.2 tools_4.3.2
[22] distributional_0.4.0 checkmate_2.3.1 pillar_1.9.0
[25] jsonlite_1.8.8 rlang_1.1.3
[1] 53.09484
[1] "compling stan"
CmdStan path set to: /gpfs/share/apps/cmdstan/2.35.0-rc2
[1] "fitting model"
Running MCMC with 4 parallel chains...
Chain 1 Iteration: 1 / 3000 [ 0%] (Warmup)
Chain 1 Iteration: 500 / 3000 [ 16%] (Warmup)
Chain 1 Iteration: 501 / 3000 [ 16%] (Sampling)
Chain 1 Iteration: 1000 / 3000 [ 33%] (Sampling)
Chain 1 Iteration: 1500 / 3000 [ 50%] (Sampling)
Chain 1 Iteration: 2000 / 3000 [ 66%] (Sampling)
Chain 1 Iteration: 2500 / 3000 [ 83%] (Sampling)
Chain 1 Iteration: 3000 / 3000 [100%] (Sampling)
Chain 2 Iteration: 1 / 3000 [ 0%] (Warmup)
Chain 2 Iteration: 500 / 3000 [ 16%] (Warmup)
Chain 2 Iteration: 501 / 3000 [ 16%] (Sampling)
Chain 2 Iteration: 1000 / 3000 [ 33%] (Sampling)
Chain 2 Iteration: 1500 / 3000 [ 50%] (Sampling)
Chain 2 Iteration: 2000 / 3000 [ 66%] (Sampling)
Chain 2 Iteration: 2500 / 3000 [ 83%] (Sampling)
Chain 2 Iteration: 3000 / 3000 [100%] (Sampling)
Chain 3 Iteration: 1 / 3000 [ 0%] (Warmup)
Chain 3 Iteration: 500 / 3000 [ 16%] (Warmup)
Chain 3 Iteration: 501 / 3000 [ 16%] (Sampling)
Chain 3 Iteration: 1000 / 3000 [ 33%] (Sampling)
Chain 3 Iteration: 1500 / 3000 [ 50%] (Sampling)
Chain 3 Iteration: 2000 / 3000 [ 66%] (Sampling)
Chain 3 Iteration: 2500 / 3000 [ 83%] (Sampling)
Chain 3 Iteration: 3000 / 3000 [100%] (Sampling)
Chain 4 Iteration: 1 / 3000 [ 0%] (Warmup)
Chain 4 Iteration: 500 / 3000 [ 16%] (Warmup)
Chain 4 Iteration: 501 / 3000 [ 16%] (Sampling)
Chain 4 Iteration: 1000 / 3000 [ 33%] (Sampling)
Chain 4 Iteration: 1500 / 3000 [ 50%] (Sampling)
Chain 4 Iteration: 2000 / 3000 [ 66%] (Sampling)
Chain 4 Iteration: 2500 / 3000 [ 83%] (Sampling)
Chain 4 Iteration: 3000 / 3000 [100%] (Sampling)
Chain 1 finished in 0.2 seconds.
Chain 2 finished in 0.2 seconds.
Chain 3 finished in 0.2 seconds.
Chain 4 finished in 0.1 seconds.
All 4 chains finished successfully.
Mean chain execution time: 0.2 seconds.
Total execution time: 0.5 seconds.
Error in !is.null(csv_contents$metadata$save_warmup) && csv_contents$metadata$save_warmup :
invalid 'y' type in 'x && y'
Calls: <Anonymous> -> <Anonymous> -> initialize -> <Anonymous>
Execution halted
I can give you my stan and R code if that is helpful. Thanks so much.
Can you try the GitHub version of cmdstanr:
# install.packages("remotes")
remotes::install_github("stan-dev/cmdstanr")
kgoldfeld:
ppc64el
This is for the PowerPc architecture, which is why you would have gotten an error when running stanc
This new issue is 2.35 specific and I’ll be resolving today (info here .
For now you can just use v2.34:
cmdstanr::install_cmdstan(version="2.34.1")
Unfortunately, we get the same error using 2.34.1. Here is the detailed output:
Loading r/4.3.2
Loading requirement: hdf5/1.10.1 gcc/10.2.0 lapack/gcc/64/3.8.0 jags/4.3.0
jdk/8u181 gsl/2.5 udunits/2.2.28 imagemagick/6.9.10 icu/70.1 tiff/4.1.0
geos/3.9.1 pcre2/10.35 gcc/8.1.0 proj/6.1.0-gcc8.1.0 sqlite/3.28.0.1
gdal/3.8.4 python/cpu/3.7.2 bzip2/1.1.0 curl/7.60.0 libjpeg/8 jbig/2.1
webp/1.2.1 gmp/6.1.2 glpk/4.55 libpng/1.6.37
This is cmdstanr version 0.7.1
- CmdStanR documentation and vignettes: mc-stan.org/cmdstanr
- Use set_cmdstan_path() to set the path to CmdStan
- Use install_cmdstan() to install CmdStan
R version 4.3.2 (2023-10-31)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux 8.6 (Ootpa)
Matrix products: default
BLAS: /gpfs/share/apps/R/4.3.2/lib64/R/lib/libRblas.so
LAPACK: /gpfs/share/apps/R/4.3.2/lib64/R/lib/libRlapack.so; LAPACK version 3.11.0
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
time zone: America/New_York
tzcode source: system (glibc)
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] cmdstanr_0.7.1
loaded via a namespace (and not attached):
[1] tensorA_0.36.2.1 backports_1.4.1 utf8_1.2.4
[4] R6_2.5.1 xfun_0.43 magrittr_2.0.3
[7] posterior_1.5.0 glue_1.7.0 abind_1.4-5
[10] tibble_3.2.1 knitr_1.46 pkgconfig_2.0.3
[13] generics_0.1.3 lifecycle_1.0.4 ps_1.7.6
[16] cli_3.6.2 processx_3.8.4 fansi_1.0.6
[19] vctrs_0.6.5 compiler_4.3.2 tools_4.3.2
[22] distributional_0.4.0 checkmate_2.3.1 pillar_1.9.0
[25] jsonlite_1.8.8 rlang_1.1.3
[1] 54.237
[1] "compling stan"
CmdStan path set to: /gpfs/share/apps/cmdstan/2.34.1
[1] "fitting model"
Running MCMC with 4 parallel chains...
Chain 1 Iteration: 1 / 3000 [ 0%] (Warmup)
Chain 1 Iteration: 500 / 3000 [ 16%] (Warmup)
Chain 1 Iteration: 501 / 3000 [ 16%] (Sampling)
Chain 1 Iteration: 1000 / 3000 [ 33%] (Sampling)
Chain 1 Iteration: 1500 / 3000 [ 50%] (Sampling)
Chain 1 Iteration: 2000 / 3000 [ 66%] (Sampling)
Chain 1 Iteration: 2500 / 3000 [ 83%] (Sampling)
Chain 2 Iteration: 1 / 3000 [ 0%] (Warmup)
Chain 2 Iteration: 500 / 3000 [ 16%] (Warmup)
Chain 2 Iteration: 501 / 3000 [ 16%] (Sampling)
Chain 2 Iteration: 1000 / 3000 [ 33%] (Sampling)
Chain 2 Iteration: 1500 / 3000 [ 50%] (Sampling)
Chain 2 Iteration: 2000 / 3000 [ 66%] (Sampling)
Chain 2 Iteration: 2500 / 3000 [ 83%] (Sampling)
Chain 2 Iteration: 3000 / 3000 [100%] (Sampling)
Chain 3 Iteration: 1 / 3000 [ 0%] (Warmup)
Chain 3 Iteration: 500 / 3000 [ 16%] (Warmup)
Chain 3 Iteration: 501 / 3000 [ 16%] (Sampling)
Chain 3 Iteration: 1000 / 3000 [ 33%] (Sampling)
Chain 3 Iteration: 1500 / 3000 [ 50%] (Sampling)
Chain 3 Iteration: 2000 / 3000 [ 66%] (Sampling)
Chain 3 Iteration: 2500 / 3000 [ 83%] (Sampling)
Chain 3 Iteration: 3000 / 3000 [100%] (Sampling)
Chain 4 Iteration: 1 / 3000 [ 0%] (Warmup)
Chain 4 Iteration: 500 / 3000 [ 16%] (Warmup)
Chain 4 Iteration: 501 / 3000 [ 16%] (Sampling)
Chain 4 Iteration: 1000 / 3000 [ 33%] (Sampling)
Chain 4 Iteration: 1500 / 3000 [ 50%] (Sampling)
Chain 4 Iteration: 2000 / 3000 [ 66%] (Sampling)
Chain 4 Iteration: 2500 / 3000 [ 83%] (Sampling)
Chain 4 Iteration: 3000 / 3000 [100%] (Sampling)
Chain 1 Iteration: 3000 / 3000 [100%] (Sampling)
Chain 1 finished in 0.2 seconds.
Chain 2 finished in 0.2 seconds.
Chain 3 finished in 0.2 seconds.
Chain 4 finished in 0.2 seconds.
All 4 chains finished successfully.
Mean chain execution time: 0.2 seconds.
Total execution time: 0.5 seconds.
Error in !is.null(csv_contents$metadata$save_warmup) && csv_contents$metadata$save_warmup :
invalid 'y' type in 'x && y'
Calls: <Anonymous> -> <Anonymous> -> initialize -> <Anonymous>
Execution halted
Have you recompiled the model executable as well (i.e., cmdstan_model(..., force_recompile=TRUE))? Otherwise it will still have the same 2.35 behaviour
Yes - it is re-compiling. I forced it and got the same result. It is possible the admin somehow installed version 2.35.0 in the 2.34.1 directory. Is there any way to check if that is the case?
I confirmed that somehow 2.35.0 is installed in that directory - so that would explain this issue. I’ll see if he can reinstall, and will follow up here.