Hi,
When building cmdstan2.18 with MPI enabled on Linux I had to create make/local which contains
STAN_MPI=true
CXX=mpicxx
This doesn’t work in 2.20 since TBB asks to specify non standard compiler via TBB_CXX_TYPE.
Thanks for any advice.
The mpicxx command is usually a wrapper around gcc or clang. So set the tbb cxx type variable to clang or gcc and it should work.
Why do we need extra compiler variable? Can’t it just point to CXX?
No. The CXX variable is with MPI something like mpicxx or mpic++. This does not tell you what is the underlying compiler (clang / gcc). The TBB makefiles need to know this which is why it has to be defined whenever the auto detection of the Stan makefiles file for this (we have a CXX_TYPE variable which is not detecting correctly the base compiler when you use mpi wrappers). The problem with the mpi wrappers is that they work differently for openMPI or mpich.
Openmpi let you use “mpicxx —showme” to query the compiler behind(mpich has something similar). Additional variable is just passing maintenance burden to users.
I know…but as you say - they use different conventions.
My recommendation to users is to lookup the cxxflags and the ldflags which the mpi wrappers basically add to the compilation and then add these directly to our makefiles (and then refrain from calling mpi** wrappers). This is the best way to integrate with out makefiles.
If you feel the makefiles should be improved for MPI - feel free to do so ;) .
Additional variable is just passing maintenance burden to users…and developers :|
The variables work fully automatic for all non-MPI users. MPI users must anyway be technically versed. So that’s a fine save of my time here - feel free to improve.
(I have already learned way to many makefile tricks which I never intended to learn)
Just from a user perspective - it there is some wiki how to install/compile MPI/thread/gpu then it is not a problem. Spending time and effort on making automatic is not important for users that employ tools beyond the vanilla sampler. However, what I found frustrating, and actually spent lots of time, when docs are missing.
PS. I vote for allowing inserting the user code for jacobian (instead of using in-build automatic differentiation) for system of ODEs and system of AEs.
I entered the thread for the same reason, but maybe more frustrated since one must build TBB even when it’s not needed.
Whats the problem with build the TBB once? Which is done for you automatically?
We were just talking about adding TBB_CXX_TYPE, which is the opposite of “automatically”.
MPI users - ok. This user group has a few hiccups to go through. It would be great if you guys can improve our wiki in this regard. Very much appreciated!
Thank you. Just in case somebody has a system similar to mine (actually it worked on systems from two different universities):
module load gcc/5.3.0
module load openmpi/3.1.4
git clone … as in https://github.com/stan-dev/cmdstan/wiki/Getting-Started-with-CmdStan
in make/local
STAN_MPI=true
CXX=mpicxx
TBB_CXX_TYPE=gcc
make build
3 Likes
I had to also load automake and an intel package (intel/16.1) to get it to work (not sure which one may have been helpful, I think automake). Before that I was getting a “cc: command not found” error when trying to compile TBB.