I fixed the compiler error from using cmdstan described here by installing the more updated gcc-4.9.4 locally in my account on a cluster that I don’t have root access.
However, bin/stanc keeps looking up the system’s older /lib64/libstdc++.so.6 and complains with the error message below even when I have set the $PATH and $LD_LIBRARY_PATH properly to include /home/city/sbbg070/gcc-4.9.4/bin and /home/city/sbbg070/gcc-4.9.4/lib64:/home/city/sbbg070/gcc-4.9.4/lib, respecitively:
Error message when compiling cmdstan's example:
[sbbg070@wrds-cloud-login1-h ~]$ cd cmdstan-2.19.1
[sbbg070@wrds-cloud-login1-h ~/cmdstan-2.19.1]$ make examples/bernoulli/bernoulli
--- Translating Stan model to C++ code ---
bin/stanc --o=examples/bernoulli/bernoulli.hpp examples/bernoulli/bernoulli.stan
bin/stanc: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by bin/stanc)
make: *** No rule to make target `examples/bernoulli/bernoulli.hpp', needed by `examples/bernoulli/bernoulli'. Stop.
Would appreciate any advice on directing bin/stanc to refer to the updated libstdc++.so.6 under the /home/city/sbbg070/gcc-4.9.4/lib64 folder of the newly installed gcc.
This is an issue on our local cluster and we have activate scripts to setup the env vars appropriately. I think you’re just missing LDFLAGS but this is what I use:
$ cat /soft/stan/activate
ver=2.17.0
echo "activating stan v$ver"
export CMDSTAN=/soft/stan/cmdstan-2.17.1
export PATH=/soft/clang/4.0.1/bin:/soft/stan/cmdstan-2.17.0/bin:$PATH
export LDFLAGS="-Wl,-rpath,/soft/gcc630/bin $LDFLAGS"
export LD_LIBRARY_PATH=/soft/gcc630/lib64:/soft/gcc492/lib64:$LD_LIBRARY_PATH
$ . /soft/stan/activate
activating stan v2.17.0
$ make -C $CMDSTAN $PWD/model
make: Entering directory `/soft/stan/cmdstan-2.17.1'
--- Translating Stan model to C++ code ---
...
Yours should be simpler, since in our case, GCC 6.3.0 was built on GCC 4.9.2, which was built on god-knows-what, was used to build CmdStan.
I tried adding the environment variables in your suggestion but the same error message appears. bin/stanc still stubbornly looks up the libstdc++.so.6 under the system installed /lib64 even though the more updated gcc-4.9.4 installation added by me seems to work fine:
[sbbg070@wrds-cloud-login1-h ~]$ gcc --version
gcc (GCC) 4.9.4
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
and even though I have manually added a link in the cmdstan folder to refer libstdc++.so.6 to the libstdc++.so.6.0.20 in the updated gcc-4.9.4 installation added by me!
[sbbg070@wrds-cloud-login1-h ~/cmdstan-2.19.1]$ ls -l
total 6652
drwx------ 3 sbbg070 city 4096 Jun 1 19:42 bin
drwx------ 3 sbbg070 city 4096 Jun 1 19:19 examples
-rw------- 1 sbbg070 city 5271 Apr 18 15:00 Jenkinsfile
-rwxr-xr-x 2 sbbg070 city 6720624 Jun 1 18:15 libstdc++.so.6
-rw------- 1 sbbg070 city 1480 Apr 18 14:57 LICENSE
drwx------ 2 sbbg070 city 4096 Jun 1 19:19 make
-rw------- 1 sbbg070 city 7521 Apr 18 15:00 makefile
-rw------- 1 sbbg070 city 1708 Apr 18 14:57 README.md
-rwx------ 1 sbbg070 city 6068 Apr 18 14:57 runCmdStanTests.py
drwx------ 5 sbbg070 city 4096 Jun 1 19:19 src
drwx------ 7 sbbg070 city 4096 Jun 1 19:19 stan
-rwx------ 1 sbbg070 city 587 Apr 18 14:57 test-all.sh
[sbbg070@wrds-cloud-login1-h lib64]$ pwd
/home/city/sbbg070/gcc-4.9.4/lib64
[sbbg070@wrds-cloud-login1-h lib64]$ ls -l libstdc++.so.6
lrwxrwxrwx 1 sbbg070 city 19 Jun 1 18:15 libstdc++.so.6 -> libstdc++.so.6.0.20
With the LDFLAGS now added, I removed cmdstan and installed it again.
Unfortunately, the same error message continues to appear when trying to compile the example stan model despite that the cmdstan was successfully compiled using the gcc-4.9.4 installed by me!
You may need to check that the LDFLAGS is being used when building CmdStan. At some point, make will show the command used and it should contain the path to your GCC. (You could do just that by removing bin/stanc and doing make build).
If it still doesn’t work, run ldd -v on the stanc binary and check the rpaths like so
Thanks for helping troubleshoot this, I know all the info on how do to it is in the various compiler-related docs but it sucks that we have to visit it on users without a detailed non-dev explanation of exactly where all the bits are found. Would be great to have some FAQ items for this so helping people with CmdStan didn’t mean dredging up these tricks.
Thank you very much for the guidance. It’s very useful. I checked accordingly and noticed LDFLAGS was not being used. This led me to suspect it was not set correctly despite echo $LDFLAGS showed the correct value.
On the cluster in concern, the setup uses a tcsh shell and does not have the export command. I thought set is its equivalent. But after a double check (with printenv not showing the LDFLAGS), I found that setenv should be used to set the environment variables. This fixes the compilation error.
I also would like to take this opportunity to make a note about running multiple chains as an array of batch jobs on different clusters. I have access to two clusters. In one cluster, the following commands in the .sh script works (explained in this post):
#PBS -t 1-4
cd ~/cmdstan-2.19.1
../SSM0.5.2dev sample algorithm=hmc metric=dense_e adapt delta=0.8 id=$PBS_ARRAYID data file=../SSM_data.dat output file=samples$PBS_ARRAYID.csv
However, in the other cluster, I need to indicate the array of jobs as an option of the qsub command:
qsub -t 1-4 [yourshellscriptname].sh
In the shell script, I need to use $SGE_TASK_ID, instead of $PBS_ARRAYID:
export PATH=$HOME/gcc-4.9.4/bin:$PATH
export LDFLAGS="-Wl,-rpath,$HOME/gcc-4.9.4/bin"
export LD_LIBRARY_PATH=$HOME/gcc-4.9.4/lib64
cd ~/cmdstan-2.19.1
../SSM0.5.2dev sample algorithm=hmc metric=dense_e adapt delta=0.8 id=$SGE_TASK_ID data file=../SSM_data.dat output file=samples$SGE_TASK_ID.csv
I also need to use the export command to set the environment variables because in the batch mode, the cluster uses a csh shell that has no such variables set as usual as in the interactive mode.
I hope these details would make another user’s life easier.
Thanks for sharing, in particular the difference with tcsh.
Under the Slurm job scheduler, an array job is done similarly, though many large Slurm clusters now use the node as the unit of resources to allocate, so one does multiple chains within the job. FWIW, my script looks like this for a hyperparameter sweep,
#!/bin/bash
#SBATCH -C gpu -t 12:00:00
#SBATCH -J vep-sd-lsp
#SBATCH --array=1-21
id=${SLURM_ARRAY_TASK_ID:-"1"}
rm -fv data/${id}.R
cp data.R data/${id}.R
echo 'log_sd_lsp <- ' $(grep "^\s*${id}\s" log_sd_lsp.txt | cut -f2) >> data/${id}.R
rm -fv sample/${id}.*.csv
for i in {1..12}
do
./model sample num_warmup=1000 num_samples=1000 \
data file=data/${id}.R output file=sample/${id}.${i}.csv refresh=50 \
&> logs/stan-${id}.${i}.txt &
done
echo `date` 'waiting for chains to finish'
wait
rm -fv summary/${id}.* diagnose/${id}.txt
# ./stan* are symlinks to binaries in cmdstan-*/bin folder
./stansummary --csv_file=summary/${id}.csv sample/${id}.*.csv &> summary/${id}.txt &
./standiagnose sample/${id}.*.csv &> diagnose/${id}.txt &
echo `date` 'waiting for stansummary & diagnose to finish'
wait
echo `date` 'cleaning up'
rm -fv sample/${id}.*
tar cvjf logs/stan-${id}.tbz logs/stan-${id}.*.txt
rm -fv logs/stan-${id}.*.txt
This is then submitted with sbatch run.sh.
I’m tempted to say that this may be an opportunity for additional tooling wrt. CmdStan, but workflows and such become so opinionated that common abstractions are difficult to identify.