Building MPI

[seantalts]

@wds15

I always used STAN_HAS_MPI, but you both seem to favor STAN_MPI. I don’t mind either way, just would like to settle on either one and a clear definition. So far I defined that whenever STAN_HAS_MPI is defined, then the code is configured to rely on MPI; the actual value was even ignored. In short: (a) what macro name and (b) is defining enough or does the variable have to be equal to some value in addition?

Sorry, I have no preference here but I didn’t remember what you had chosen. I tried to make that clear with the caveat following each STAN_MPI, haha.

[wds15]

Hi!

Are u able to get the example up and running which is part of the mpi branch of cmdstan? So please checkout the proto-mpi branch of cmdstan somewhere and also checkout stan-math with the feature/concept-mpi-2 branch. I just got everything working straight away. So currently what sufficies is

1. in the stan-math branch do a make stan-mpi
2. follow the instructions
3. checkout the cmdstan branch proto-mpi and the do a
   make build
   make examples/mpi/oral_2cmt_oral4

When you get that running you can continue adapting to your program. This is what I would recommend. Be warned that the process will slightly change along the lines of what has been discussed in this thread. However, the problematic bit of finding the dynamic libraries for the linker is correctly working; I just verified it on Linux.

[wds15]

@seantalts

Sorry, I have no preference here but I didn’t remember what you had chosen. I tried to make that clear with the caveat following each STAN_MPI, haha.

No worries. I thought it would be a good occasion to decide on this detail. Maybe we should again align with GPU conventions (which I am not super familiar with). It is not hard to change these things, they just eat up time if one goes back and forth.

[wds15]

I add a branch which implements things as I think we discussed, see here.

This should work fine on Linux and MacOS.

Should I make it a WIP pull?

[aornugent]

I’m afraid not. make stan-mpi runs fine, then following the instructions I tried to add boost to the linker libraries using:

export CPPFLAGS="-I/root/cmdstan/stan/lib/stan_math/lib/boost_1.66.0"
export LIBS="-L/root/cmdstan/stan/lib/stan_math/lib/boost_1.66.0/stage/lib"

make build on the proto-mpi branch then fails to find the serialisation library.

Is there a better way to do this?

I see that make/mpi in math has linkers for the boost serialisation library, could it be that it’s not being included somehow?

[Bob_Carpenter]

@syclik

Maybe seeing the overall steps side-by-side would help:

Very much—thanks!

@seantalts

in the Stan language the user will just use map_rect and if they have STAN_MPI (or whatever) defined in make/local it will farm to map_rect_mpi (and if not it will call map_rect_serial).

That’s the plan.

[wds15]

Have you upgraded to the latest state of the branch?

Could you post the logs which show the failure?

While it should work without any additional definitions (besides the one in the instructions), the workaround is to define on Linux

export LD_LIBRARY_PATH=/root/cmdstan/stan/lib/stan_math/lib/boost_1.66.0/stage/lib

this will instruct the dynamic linker to look for boost mpi in these locations.

[seantalts]

About that branch, especially here: https://github.com/stan-dev/math/compare/feature/issue-778-add-mpi-build#diff-9a0429d264590ef98fb30995236a7756R29 - In CmdStan, how will the makefile add a dependency on LIBMPI?

[wds15]

Uhm… don’t know yet. I am glad I manged the tests where my suggestion is to detect tests with mpi in their name.

For cmdstan that is a different story, yes. Can we copy what the GPU efforts do here or are matter easier here as nothing needs to be linked? If not… can we ifdef this rule somehow “in”? I am really not an expert in makefiles.

BTW, I am using the mpi.so as target as a workaround here. So whenever boost mpi build boost mpi and boost serialization it also creates a mpi.so. The upside of using mpi.so is that it always has the same filename extension on Linux and on Mac while the other two have .dylib or .so.

[seantalts]

I was thinking we might need the imperative build code to be executed ifdef STAN_MPI, unless @syclik or @sakrejda knows some way to add a dependency at runtime in the makefile. I know the sort of normal makefile route to adding dependencies to targets at runtime is to use the compiler to generate those .d files, but I’m not sure if we need to do something like that, just run the build code, or do something else creative. I think this might impact what we do in the Math library, though we could always add it later. Might be nice to have a plan for CmdStan at least at this point.

[aornugent]

I’ve been starting from scratch each time, so I assume so. Complete build log attached.

Defining LD_LIBRARY_PATH didn’t help either. I’ll try a fresh linux install and make sure it’s not a quirk in my system.

EDIT: Fresh install of Ubuntu 16.04 didn’t help.

apt-get install git g++ mpi-default-dev python-dev
git clone --recursive https://github.com/stan-dev/cmdstan
git fetch
git checkout feature/proto-mpi
cd stan/lib/stan_math
git checkout feature/concept-mpi-2
make stan-mpi
# SUCCESS
cd $(CMDSTAN)
# check $(MATH) path and include $(MATH)make/mpi
make build

mpic++ -Wall -I . -isystem stan/lib/stan_math/lib/eigen_3.3.3 -isystem stan/lib/stan_math/lib/boost_1.66.0 -isystem stan/lib/stan_math/lib/cvodes_2.9.0/include -std=c++1y -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -DBOOST_PHOENIX_NO_VARIADIC_EXPRESSION -Wno-unused-function -Wno-uninitialized -I src -isystem stan/src -isystem stan/lib/stan_math/ -DFUSION_MAX_VECTOR_SIZE=12 -Wno-unused-local-typedefs -DEIGEN_NO_DEBUG -DSTAN_HAS_MPI -DNO_FPRINTF_OUTPUT -pipe -Lstan/lib/stan_math/lib/boost_1.66.0/stage/lib -lboost_serialization -lboost_mpi -Wl,-rpath,stan/lib/stan_math/lib/boost_1.66.0/stage/lib  -O0 -o bin/stansummary bin/cmdstan/stansummary.o

bin/cmdstan/stansummary.o: In function `boost::archive::detail::oserializer<boost::mpi::detail::mpi_datatype_oarchive, stan::math::mpi_stop_worker>::~oserializer()':
stansummary.cpp:(.text._ZN5boost7archive6detail11oserializerINS_3mpi6detail21mpi_datatype_oarchiveEN4stan4math15mpi_stop_workerEED2Ev[_ZN5boost7archive6detail11oserializerINS_3m
...

mpi_build_log.txt (270.1 KB)

[yizhang]

@syclik

What I’m trying to suggest:

step 0: done by the user. User installs some MPI implementation.
step 1: user specifies flags for MPI in make/local

step 2: user just goes on and does testing as normal; no need to type make stan-mpi or whatever to set up.

If no MPI installation:

step 0: nothing
step 1: user specifies whatever compiler flags (but no MPI)
step 2: user just goes on and does testing as normal; if MPI is attempted to be used, then this should break.

Is that appropriate? To do that, we use make in the right way and just specify the targets appropriately

We can also use autotools to configure, giving user options to download, install MPI, or specify prefix for MPI path. An example is PETSc’ installation. The configuration is one-time effort.

This will make users’ life a little bit easier.

[syclik]

@yizhang

We can also use autotools to configure, giving user options to download, install MPI, or specify prefix for MPI path. An example is PETSc’ installation. The configuration is one-time effort.

That looks promising. Hopefully this means you know how to use autotools well and you’re volunteering? (We’ve had multiple volunteers trying to move make to CMake that didn’t materialize.)

First off, I don’t think we need this complexity for the Math library. But if it’s really easy to set up, then that’s great.

Here are some of the things we have to do as maintainers:

• we something to work for Mac, Linux, and Windows. It doesn’t have to be the same across OS, but it has to work across multiple versions of the OS and often multiple versions of compilers
• we can get it working easily in CmdStan, but we also need to think about how this works with RStan and PyStan. Those are different build procedures and are much closer to what we have in make now.
• the maintenance burden on the developers. If no one else knows autotools, then it has to be implemented straightforward enough where someone else on the team can StackOverflow out of problems.

If you think it can cover that, then please put together a plan!

But autotools won’t help us (the devs) write makefiles in a way that’s more make-centric. That we still need to do to keep the makefiles clean.

[syclik]

@seantalts

dependency at runtime in the makefile

The dependency is known at compile time. And if we knew that, then yes, it’s straightfoward. In the makefile, we would include a file. In that file, there would be an additional target. It’d look something like:

foo.o: lib/boost_1.66.0/libboost_mpi.so
foo.o: LDFLAGS += -l ... <all the libs for mpi>

(or whatever)

That would just make sure foo.o would kick off the build to the boost mpi library if it isn’t already built and it’ll add the flags. We have all this info when we parse the model; we just need to generate some file that can be included from make. That’s all do-able from CmdStan. We could probably do it for RStan and PyStan too.

[wds15]

The boost build system does not seem to build a dynamic library libboost_serialization.so or libboost_mpi.so for you. Could you check in the boost/stage/lib directory if you have these two files there?

I don’t think so from the logs. If possible for you, could you please figure this out? Do you have dynamic libraries installed for your MPI system?

Please follow the doc from boost MPI to figure out how to get a working boost MPI:

http://www.boost.org/doc/libs/1_66_0/doc/html/mpi/getting_started.html

… and please report back what was the root-cause such that we can doc the issue for others or even avoid the issue from within stan-math. Thanks!

[seantalts]

@syclik

The dependency is known at compile time. And if we knew that, then yes, it’s straightfoward. In the makefile, we would include a file. In that file, there would be an additional target

I meant at the makefile’s runtime - it has to read in the STAN_MPI flag from make/local or the command line and then add some dependencies. I don’t think we’re planning to parse the model to add it because the model will just use map_rect (as Bob confirmed above) and that function will call out to either the serial or MPI implementation depending on the STAN_MPI flag in make/local. I’m just describing the current plan as far as I’m aware; I’m not really sure if it’s better or worse than what I think you’re referring to, where we’d use a separate map_rect_mpi function in the stan language code and then have the Stan compiler generate a makefile to be included.

[wds15]

Don’t we require for cmdstan anyway that the use triggers a make build which is a phony target?

If so, then we could probably just trigger inside the build phony target a make call which builds the boost mpi libraries whenever STAN_MPI is defined. Once the mpi libraries are there, the makefiles only need to add the right LDFLAGS which link in those libraries. These additional flags can be added using ifdef logic. Does that sound like a plan for cmdstan?

[seantalts]

Looks like we do tell users to do a make build first in the guide. So you suggest putting an ifdef STAN_MPI: <build MPI> block in the build target in the CmdStan makefile? I think that could probably work. Those flags might already get added conditionally and appropriately when CmdStan includes some of the math repo makefiles. Sounds good to me for a first pass (and maybe some autoconf or CMake one day 🌈). I think @syclik needs to sign off though.

[wds15]

Yes, this is what I mean. Users would be forced to do the make build call. Whenever the STAN_MPI thing is defined, then an if in that block will trigger the build. I thought the make build thing was anyway a requirement.

[yizhang]

@syclik

we something to work for Mac, Linux, and Windows. It doesn’t have to be the same across OS, but it has to work across multiple versions of the OS and often multiple versions of compilers
we can get it working easily in CmdStan, but we also need to think about how this works with RStan and PyStan. Those are different build procedures and are much closer to what we have in make now.
the maintenance burden on the developers. If no one else knows autotools, then it has to be implemented straightforward enough where someone else on the team can StackOverflow out of problems.

I can do 1 & 3. But not 2. Will that still worth trying? If so, I’ll set up a plan after MPI settles down.