Building MPI

so… will it not work with g++ or clang++? I’m not an expert in MPI, so you’ll have to explain or point me to references that explain this.

Great!

Yes. We can have make call the boost build system to generate the target (in make speak). Hope that makes sense.

So… how do we deal with that? Are there different libraries to link against?

Let’s take one step back. We do require a working MPI implementation to be on the target system already. These can be

  • OpenMPI
  • mpich
  • Intel MPI
  • whatever boost MPI supports as backend

These MPI installations come with their own headers and their own libraries which we need to tell the compiler to pick up (the includes and the libraries). In order to automate this, these MPI installations provide usually a command “mpic++” which is a wrapper around the actual compiler. All this wrapper does is to call g++ or clang++ with the correct compiler and linker options. So the MPI implementations usually recommend to build any MPI program by using mpic++ as you then are guaranteed that all compiler and linker settings are correct.

However, you do not have to use mpic++. You can actually use any compiler and to find out about the needed compiler and linker flags a “mpic++ --show-me” will tell you those options.

Does that make sense now?

What is then still needed is the boost MPI which interfaces from C++ to the usual C implementations of MPI.

How about I rename the target “stan-mpi” to “boost-mpi” and I create a pull request with that? Then we can discuss in that pull how to move on.

1 Like

Questions:

  • which of these MPI implementations are you proposing we support?
  • which of these provide “mpic++”?

What do you want to do:

  • do we only support MPI implementations that have “mpic++”?

or

  • do we allow a different compiler with the user specifying flags?

Since we’re on this topic, do we want to support this on Windows, Linux, and Mac? How do we test that it’s worked on each OS x MPI implementation?

No – that clutters the makefile with an additional .PHONY target that’s custom to our build and not something that’s normal in the way make handles it.

The correct way to do it is to use the built library as the target, then specify how that gets built. It’s just a different way of thinking about how to build from the target backwards (which is the way you’re supposed to think about make).

What I’m suggesting is neither easier or harder than what you’re suggesting, but will keep the builds more in line with how make handles it.

All very good questions. I thought we were only going to try to roll this out on Linux initially. I didn’t realize we weren’t supplying an MPI implementation and there wasn’t a standard one on which we can rely. Are they all implementing the same standard?

Do we really want to delegate to mpic++? We provide a lot of control on compiler and linker options. How do we guarantee it doesn’t mess with our options like libraries and optimization?

Yes, they all implement the same standard. It does not make sense to deploy with stan-math any given MPI implementation. We did have this discussion already a while ago. The point is that on any given cluster a MPI implementation is already available and there is no choice in what you use since it is integrated with the hardware installation of the network. For desktop systems you can really easily deploy any of the major MPI installations via Macports or your Linux distribution. I would suggest we support Linux and MacOS. I never got it to work for Windows.

So what we will provide is a C++ interface to MPI which is the boost MPI library. The boost MPI library has been tested with OpenMPI, mpich and IntelMPI. Hence, I would not advise that we test with all different MPI implementations or OS combinations. We should just assume boost MPI works with all of these and quote which ones we use for testing.

Using mpic++ is strongly recommended by OpenMPI, but we do not have to do that. However, you are then guaranteed to get the right compiler and the right options supplied to it which ensure that MPI is working. I have never experienced any trouble when setting CC=mpic++ in the stan-math makefiles - and I also never had problems when using the compiler and linker flags which “mpic++ --show-me” tell you to use.

The reason why OpenMPI recommends to use mpic++ is that they have then some control over the compilation - and this is a good thing. Bear in mind that the MPI installation must know exactly the representation of doubles/ints/whatever on your platform in order to work correctly. So the CPU/OS/compiler details matter to MPI for obvious reasons.

We just need to understand how it interacts with things we specify.

Do we want to pick one? Support them all? I just don’t want huge number of config issues raining down on us from all sorts of MPI libraries.

On my system I am getting this:

[07:37:28][sebi@sebastians-macbook-pro-1:~/work/math]$ mpic++ --show-me
/usr/bin/clang++ -I/opt/local/include/openmpi-mp -L/opt/local/lib -L/opt/local/lib/openmpi-mp -lmpi_cxx -lmpi

So nothing too fancy.

The point of using boost MPI (which is tested with all the ones I mentioned) is that we immediately support all major ones. For testing Stan my suggestion is to just say we test with mpich, for example (or OpenMPI - I don’t care at all). This is a bit like choosing a compiler - you have to make a choice here. I don’t think we need to test with mpich and OpenMPI in this world of limited resources.

I agree with Bob and I’d like to clarify a little bit.

What I don’t know is whether OpenMPI’s instructions work for other MPI implementations. It doesn’t look like mpic++ is even available for mpich (based on their online doc). I’m just really trying to clarify what we’re trying to do. If we’re allowing this only for Linux and OpenMPI, that’s fine and we should be honest about that and be clear that the build instructions are not appropriate for anything else.

If I interpret this correctly, we want the user to first install whatever MPI implementation locally. Then we want the user to set the compiler and additional compiler flags, so we need to provide a place to put those flags. This should be agnostic whether the user wants to use mpic++ or clang++ directly. That’s what I’m trying to get at.

This is dodging the question: do we want to support both or just one? If we want to support both, then we should only rely on the OpenMPI instructions if it applies to both. And it doesn’t look like it does (based on mpich’s documentation). So… it just seems inconsistent.


What I’m trying to suggest:

  • step 0: done by the user. User installs some MPI implementation.
  • step 1: user specifies flags for MPI in make/local
  • step 2: user just goes on and does testing as normal; no need to type make stan-mpi or whatever to set up.

If no MPI installation:

  • step 0: nothing
  • step 1: user specifies whatever compiler flags (but no MPI)
  • step 2: user just goes on and does testing as normal; if MPI is attempted to be used, then this should break.

Is that appropriate? To do that, we use make in the right way and just specify the targets appropriately.

mpich provides the mpicxx command (and OpenMPI seems to have a command which is also mpicxx which should do the same as mpic++).

You are missing in your suggestions the boost MPI library. To me we have these use cases:

  • a cluster installation which runs usually Linux and has already some MPI installation (one of the main ones which are mpich, OpenMPI or Intel MPI). In this case the MPI installation which is there must be used.

  • some user with a desktop machine. That user has to install any MPI implementation - be it mpich or OpenMPI. As operating systems we can support Linux and MacOS here. Windows is beyond me to configure at this stage.

I would treat the MPI implementation as part of infrastructure which we would not need to worry about at all. This is my view as I see MPI just like the OS we are running on - we do not have tests which discriminate RHEL 6.5 or 7.0 or whatever.

The MPI implementations are C libraries and they bring their instructions with them as to how to compile and link programs. The common denominator between OpenMPI and mpich seems to be the mpicxx command.

From the perspective of stan-math we worry about a C++ adapter to the C implementations which is is the boost MPI library (and its dependencies). This is what we package in stan-math. We also package in stan-math the boost build system such that we can hook up any Stan installation into any existing MPI installation easily. Building boost MPI and its dependencies is what is triggered right now by make stan-mpi. BTW, I don’t mind changing the name here; it was just a way to solve a problem.

I hope this clears up a few things. You can certainly have different perspectives on matters (as it seems you have wrt to where stan “starts” / “ends”). I feel like I am starting to repeat things…maybe we should start a wiki page on the matter?

1 Like

I think we’re on the same page. The MPI implementation and C libraries are already installed by the user. Boost is not.

Instead of make stan-mpi, I’m strongly suggesting we replace stan-mpi with the name of the target. That might be something like: make lib/boost_1.66.0/<I don't actually know what it is>/libboost_mpi.so or whatever. The instructions to build that would call boost build.

There would be no need for anyone to call that explicitly. If we add that particular library as a dependency in the correct test, it will be built automatically if it’s not there. If it’s there, it won’t build it.

I don’t know what you mean “starts” / “ends”?

Sounds good to me. Although, I am not sure if we can make this fully automatic. The thing is that boost MPI cannot be build on systems without a MPI installation. However, I did manage to setup MPI tests in a way that they are automatically disabled whenever STAN_HAS_MPI is not defined. This would allow to deploy and compile MPI tests to any system. On systems without MPI tests would be disabled and on systems with configured MPI the MPI test would run as normal. I agree that this is a hack, but I thought this makes it easier to integrate it with our test system.

Maybe this is for cmdstan then: I would prefer if the user makes a specific choice somewhere in order to trigger that MPI is available rather than automating this.

Do we need to test different MPI implementations or not? I would probably not do that - rather I would quote “we tested MPI with implementation XY”.

I just got caught up on this thread - I think one of the main points of disagreement is how to trigger the build of the boost MPI wrapper libraries needed. I think @wds15 is proposing adding a new .PHONY boost-mpi or stan-mpi target to the makefiles because in our proposed user interface, a user must decide that they want to build with MPI support (the system does not figure this out automatically). Is this the correct UI we want? I think it seems reasonable, but it makes sense to get in sync on that first.

If we want the user to specify, then we must provide at least one place for the user to do so. I think maybe we could do what @syclik is saying and use normal makefile targets set up to ensure the boost libraries get built (with the correct boost build incantations), but have some logic in there similar to what @stevebronder put in the GPU makefiles that looks for our primary MPI switch (for GPUs this is adding STAN_OPENCL=true to the make invocation) and then adds the extra dependencies. I am not sure how feasible this is / how closely it matches with the GPU case, but this is one idea.

If that fails, the easiest thing to do could be to have a two-step “button” to turn on MPI compilation / installation. This is still very simple compared to e.g. PyTorch, so might be a good first step if we can’t figure the other thing out?

How does that sound for a plan of action?

The best solution would be some autoconf magic which detects what is on the system available. However, I would not like to go there for now.

GPU and MPI should work the same in terms of building… I was not aware what was doen for the GPU stuff.

So as I understand it, the user puts in the make/local

STAN_HAS_MPI true

And along with it he also makes sure that things can be build wrt to MPI which means to define CXX to be mpicxx or whatever, right?

Once that is done, then the boost mpi will be build as an intermediate target which is hooked into the dependency list if the above is defined. Did I get that right? Works for me.

I think maybe the one trick I glossed over is if we can add dependencies to existing targets at runtime with the makefile. Maybe we would add them to one of the .d dependency files? The GPU case just needs to add extra linker flags, not trigger additional build targets, which is easier. Or I suppose we can just do the actual build commands in the ifdef STAN_MPI block?

1 Like

Yes, the overall UI is correct: a user must decide they want to build with MPI support.

Adding a special make target is at the wrong level. To use MPI, the user must use the map_rect_mpi function instead of the map_rect_serial function.

@wds15, is that right?

We definitely could. We don’t have a build step in GPU. If we did, this is how we would do it.

The difference between GPU and here is that we can skip the building of the libraries. Maybe seeing the overall steps side-by-side would help:

Step MPI GPU
0: done by the user Install some MPI implementation Install OpenCL
1: done by the user specifies MPI flags in make/local specifies GPU flags in make/local
2: user just goes on and does testing as normal no need to type anything special to test against MPI. The MPI libraries are included as dependencies within make/tests; this will trigger the libraries to be built if necessary. The executable is compiled with the right flags and linked as needed. no need to type anything special to test against GPU. The GPU is compiled with the right flags and linked as needed. (The only thing missing is building the library)

Agreed. And same.

Yes! If GPU had to build, it would do what I suggest.


@seantalts, thanks for summarizing.

One thing I want to make clear is that this is for the Math library. We’re not yet talking about how this really affects users at the CmdStan, RStan, and PyStan level. At the CmdStan level, we may want to have a phony target, but not at the math level.

That’s ultimately the way to go. Right now, we’ve been hand-specifying the additional dependencies in make/tests.

If we were to do more fine-grained includes, then we could add a dependency on the header to the libraries and that would work. Right now, our big headers include everything, so we can’t.

When we get to the Stan-level, one thing we can do (and this applies to CVODES as well) is to generate some text that indicates what it depends on. We can use that to link against just MPI or GPU or CVODES or IDAS as necessary. It’ll take work and merits a different discussion; this will need to be generated by stanc or something at that scope.

Minor point but I think in the Stan language the user will just use map_rect and if they have STAN_MPI (or whatever) defined in make/local it will farm to map_rect_mpi (and if not it will call map_rect_serial).

1 Like

Awesome. If that’s the case, then it makes sense to have a macro define just like the GPUs.

Cool. I’m not sure exactly what you mean here - is there some way to add dependencies to make/tests automatically in a ifdef STAN_MPI block in the makefile?

Two things:

  • I always used STAN_HAS_MPI, but you both seem to favor STAN_MPI. I don’t mind either way, just would like to settle on either one and a clear definition. So far I defined that whenever STAN_HAS_MPI is defined, then the code is configured to rely on MPI; the actual value was even ignored. In short: (a) what macro name and (b) is defining enough or does the variable have to be equal to some value in addition?

  • For the tests I envisoned to do the following: All tests which involve MPI would need to include this file. In there I define a MPI_TEST_F macro which disables a given test if MPI is not available and just evaluates the given test in case MPI is to be used. So the definition of that macro depends on the STAN_HAS_MPI variable.

This approach is a bit of a hack, but should allow us to just let the MPI tests be in our source base and we would not need to worry about further changes to our makefiles.

Looks like we are waming up with MPI… great! Although, we still have some iterations to go.