# Boost build into stan-math?

**URL:** <https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765>\
**Category:** Developers\
**Created:** [December 14, 2017, 5:17pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765 "2017-12-14T17:17:02Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![wds15](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wds15/32/908_2.png) [@wds15](https://discourse.mc-stan.org/u/wds15)\
**Post date:** [December 14, 2017, 5:17pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/1 "2017-12-14T17:17:02Z")

</div>

are people ok with having boost build in stan-math eventually?

This would be very useful for the planned integration of boost mpi and boost serialization library.

Pro: Hopefully easy deployment of those two libs  
Con: Lots of extra stuff in stan-math

I am not sure yet what options are best, but before embarking on boost build, I wanted to run this by the team.

---

<div class="post-metadata">

**Author:** ![Krzysztof\_Sakrejda](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Krzysztof\_Sakrejda](https://discourse.mc-stan.org/u/Krzysztof_Sakrejda)\
**Post date:** [December 14, 2017, 8:16pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/2 "2017-12-14T20:16:59Z")

</div>

Is there really another alternative?

---

<div class="post-metadata">

**Author:** ![syclik](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/syclik/32/6_2.png) [@syclik](https://discourse.mc-stan.org/u/syclik)\
**Post date:** [December 14, 2017, 8:51pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/3 "2017-12-14T20:51:59Z")

</div>

The different ways we could do this:

1. (simplest) just include all of boost in the math distribution
2. put things that aren’t used by Stan / Math that we need for conditional builds (MPI / GPU) into a separate folder
3. expect the user to install separately; boost can be installed using things like brew or apt

My preference is to include everything from boost into the math library. I know that means there’s pruning done by RStan and PyStan, but I think that step is already done now. It’ll make it easier for developers if it’s all sitting there.

---

<div class="post-metadata">

**Author:** ![ariddell](https://avatars.discourse-cdn.com/v4/letter/a/57b2e6/32.png) [@ariddell](https://discourse.mc-stan.org/u/ariddell)\
**Post date:** [December 15, 2017, 1:47pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/4 "2017-12-15T13:47:44Z")

</div>

How do other, similar software packages handle this issue?

My sense is that bundling external libraries is generally regarded as  
undesirable. It’s certainly not needed on Linux since you can install  
boost using the system’s package manager.

---

<div class="post-metadata">

**Author:** ![jjramsey](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jjramsey/32/3319_2.png) [@jjramsey](https://discourse.mc-stan.org/u/jjramsey)\
**Post date:** [December 15, 2017, 2:34pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/5 "2017-12-15T14:34:07Z")

</div>

> [@ariddell](#):
>
> It’s certainly not needed on Linux since you can install boost using the system’s package manager.

That’s not necessarily true. On a desktop that one administers, that’s usually true. However, the reason a Boost build is being considered is so that Boost.MPI can be used in Stan, which means that there’s a good chance that someone is going to want to run Stan on a Linux HPC cluster. Users on clusters would generally _not_ be using the system’s package manager. They would either use whatever Boost implementation that the cluster has available (if any), or compile Boost.MPI from source in their home directories.

Furthermore, in my experience, even HPC clusters that have Boost available often don’t have Boost.MPI available. (This is probably because there are usually several compilers and MPI implementations on a cluster, and it would be a pain to have a Boost.MPI implementation for all those compiler/MPI combinations, and only mildly useful to have a Boost.MPI implementation for a subset of them.)

---

<div class="post-metadata">

**Author:** ![Krzysztof\_Sakrejda](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Krzysztof\_Sakrejda](https://discourse.mc-stan.org/u/Krzysztof_Sakrejda)\
**Post date:** [December 15, 2017, 4:16pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/6 "2017-12-15T16:16:05Z")

</div>

> [@ariddell](#):
>
> It’s certainly not needed on Linux since you can install
> 
> boost using the system’s package manager.

I would stay away from relying on the system package manager since Linux systems are still bad at allowing multiple versions of Boost. I certainly _have_ compiled Stan against my system Boost but I wouldn’t foist that mess on anybody… and we don’t have the resources to package properly. That’s not to say we couldn’t be agnostic to the source and we should definitely not make it more difficult to use an independently installed version.

---

<div class="post-metadata">

**Author:** ![syclik](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/syclik/32/6_2.png) [@syclik](https://discourse.mc-stan.org/u/syclik)\
**Post date:** [December 15, 2017, 4:19pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/7 "2017-12-15T16:19:41Z")

</div>

TL;DR: I think we should distribute source. There are incompatibilities with different boost versions.

> [@ariddell](#):
>
> How do other, similar software packages handle this issue?

That’s a good question and I think it’s highly non-standard. I was looking through the [“Trending C++ repositories on GitHub”](https://github.com/trending/c++). Most projects don’t use boost.

These are some that include the source:

- [apple/turicreate: deps/src/boost\_1\_65\_1](https://github.com/apple/turicreate/tree/master/deps/src/boost_1_65_1)
- [mozilla/DeepSearch: native\_client/boost\_locale](https://github.com/mozilla/DeepSpeech/tree/master/native_client/boost_locale)
- [avast-tl/retdec-idaplugin](https://github.com/avast-tl/retdec-idaplugin/blob/802e1b3ea4feeb860c03fd5183d124e36be2eea6/scripts/download-and-build-libs.sh#L104). This technically doesn’t distribute it, but has a script to download it from a mirror and unpack it.

These are some that require installation:

- [bitcoin/bitcoin](https://github.com/bitcoin/bitcoin)
- [BVLC/caffe](https://github.com/BVLC/caffe)

* * *

The real issue we have is that as we’re pushing into MPL, not all versions of boost work. This is actually true for Stan’s `stanc` too. It is much easier relying on one version of boost.

There are (at least) two audiences we care about:

1. developers
2. users of RStan, PyStan, CmdStan (and everyone else that depends on CmdStan)

For the most part, both audiences are aligned. It’s much better to have an easy install rather than require a more complicated process in order to download the right libraries.

To easy development, I’d really suggest we include the whole source.

For release in things like PyStan, I’d suggest we run `bcp` or manually remove the source if that helps. If it’s really a concern (and I don’t want to do this unless there was a very, very good reason), we could possibly break out the third party dependencies into a separate submodule. That would mean Math gets tagged with a version of the dependencies, but the dependencies aren’t included in GitHub’s downloads and it’s easy to just use the tagged version of the repo without any of the libraries. That really adds a lot of overhead for the rest of the process, so I’m really against it, but it is possible.

---

<div class="post-metadata">

**Author:** ![wds15](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wds15/32/908_2.png) [@wds15](https://discourse.mc-stan.org/u/wds15)\
**Post date:** [December 16, 2017, 11:26am UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/8 "2017-12-16T11:26:57Z")

</div>

@jjramsey : Thanks, your post confirms our current plans which is to expect that there is some MPI installation on the target system already while the rest should be contained in stan-math. The MPI installation will already be part of the cluster where MPI is used, but all the rest cannot be expected to be available (let alone in adequate versions). For Mac users a macports/homebrew installation of openmpi or mpich should be straightforward. On Linux the installation of MPI should also work easily; I have no clues if this is doable on windows.

Now, I also think that minimize what gets distributed is a good thing. On the other hand, we already have experienced boost bugs which forced us to do small changes to boost. So having it there is good. How about we include only what we really need. So in addition what we have now, we include in addition

- boost build
- mpi & serialization library

Does that make sense?

---

<div class="post-metadata">

**Author:** ![sakrejda](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/sakrejda/32/845_2.png) [@sakrejda](https://discourse.mc-stan.org/u/sakrejda)\
**Post date:** [December 16, 2017, 3:01pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/9 "2017-12-16T15:01:53Z")

</div>

I haven’t checked lately what the difference is, do you have some idea? All I know is that the math repo recursive download is slow.

---

<div class="post-metadata">

**Author:** ![jjramsey](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jjramsey/32/3319_2.png) [@jjramsey](https://discourse.mc-stan.org/u/jjramsey)\
**Post date:** [December 18, 2017, 1:51pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/10 "2017-12-18T13:51:47Z")

</div>

One thing I wonder about integrating Boost.MPI into stan-math is now you plan to deal with the various quirks in MPI installation that happen on various clusters. Sometimes the available MPI C++ wrapper is simply `mpic++` or `mpicxx`, and IIRC, the Boost.MPI build process handles those almost automatically. However, sometimes there are _multiple_ MPI wrappers; for example, on some clusters, the wrapper for `g++` is `mpicxx`, while the wrapper for the Intel C++ compiler is `mpiicpc`. In some cases, there is no explicit MPI wrapper. For example, on Crays, there is usually a wrapper script for the C++ compiler called `CC`, which often handles both serial and parallel compilation. These issues in turn may smoke out bugs and issues in the build system currently in use in stan-math and CmdStan.

---

<div class="post-metadata">

**Author:** ![wds15](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wds15/32/908_2.png) [@wds15](https://discourse.mc-stan.org/u/wds15)\
**Post date:** [December 18, 2017, 4:37pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/11 "2017-12-18T16:37:54Z")

</div>

Well, these quirks are the reason to go for boost build. As far as I know, you can tell boost build which mpi installation to pickup from a given system. All one needs to do is to tell boost build which mpicxx or mpiWhatever to take. If that is not sufficient, then one even has the choice to tell boost build exactly from where to include what and link against what (the very hard way).

So as long as the user can supply its project-config.jam or user-config.jam (not sure yet), we should be good, right?

---

<div class="post-metadata">

**Author:** ![wds15](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wds15/32/908_2.png) [@wds15](https://discourse.mc-stan.org/u/wds15)\
**Post date:** [December 18, 2017, 4:39pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/12 "2017-12-18T16:39:51Z")

</div>

> [@sakrejda](#):
>
> I haven’t checked lately what the difference is, do you have some idea?

Hmm… I recall some old tweaks for the intel compiler. I am not sure if these are still in there. My point was that it can be convenient to have full control over the sources…cvodes is the same story. For cvodes we have to ifdef out all those stupid printf statements to make CRAN happy.

---

<div class="post-metadata">

**Author:** ![jjramsey](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jjramsey/32/3319_2.png) [@jjramsey](https://discourse.mc-stan.org/u/jjramsey)\
**Post date:** [December 18, 2017, 5:27pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/13 "2017-12-18T17:27:26Z")

</div>

> [@wds15](#):
>
> So as long as the user can supply its project-config.jam or user-config.jam (not sure yet), we should be good, right?

Are you planning on the user supplying the .jam file directly, or do you expect to generate the .jam file based on user input (for example, variables in `make/local`)?

---

<div class="post-metadata">

**Author:** ![wds15](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wds15/32/908_2.png) [@wds15](https://discourse.mc-stan.org/u/wds15)\
**Post date:** [December 18, 2017, 5:52pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/14 "2017-12-18T17:52:23Z")

</div>

I have not yet started to think in detail about it. The more we can automate, the better. I do not think that `make/local` should be translated to a jam file, but a `make/mpi` could be used for that. I hope we can make the „easy“ configurations work out of the box automagically, but also leave things flexible enough for those non-standard cases.

… if you like to contribute to this mpi build problem you are more than welcome to draft something…

---

<div class="post-metadata">

**Author:** ![Krzysztof\_Sakrejda](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Krzysztof\_Sakrejda](https://discourse.mc-stan.org/u/Krzysztof_Sakrejda)\
**Post date:** [December 19, 2017, 6:42pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/15 "2017-12-19T18:42:49Z")

</div>

You got me all wrong, I’m all for including the sources.

---

<div class="post-metadata">

**Author:** ![Krzysztof\_Sakrejda](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Krzysztof\_Sakrejda](https://discourse.mc-stan.org/u/Krzysztof_Sakrejda)\
**Post date:** [December 19, 2017, 6:45pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/16 "2017-12-19T18:45:27Z")

</div>

> [@wds15](#):
>
> … if you like to contribute to this mpi build problem you are more than welcome to draft something…

I really appreciate all the time you’ve put into this but unfortunately puts you into the position of knowing better than most people what needs to be done. Are there specific tasks you want help with? For example if there’s a branch that builds by following some instructions, I could do some comparisons of how difficult it would be to provide Boost in different ways but I don’t want to get into figuring out how to build a branch/example from the beginning.

---

<div class="post-metadata">

**Author:** ![wds15](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wds15/32/908_2.png) [@wds15](https://discourse.mc-stan.org/u/wds15)\
**Post date:** [December 20, 2017, 8:41am UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/17 "2017-12-20T08:41:03Z")

</div>

Hi!

Help would be great and much appreciated. Since the boost mpi 1.64 is broken, the first thing anyone could do is upgrade boost to 1.65.1.

On stan-math I am working on the branch feature/concept-mpi-2 which includes at the top level a MPI\_NOTES file which details steps to get a working system. The final tweaks to get the mpi tests compiling require a make/local file. The one which I have setup for myself is in the cmdstan branch feature/proto-mpi.

The mpi tests in stan-math are under prim / mat / functor .

If you want to try to get those tests up and running that would be great. In case of any issues, I am happy to help out.

Alternativley, I could wipe my system and see if the steps detailed in the file are sufficient. I think they contain all what is needed, but I would know once I clean things.

---

<div class="post-metadata">

**Author:** ![Krzysztof\_Sakrejda](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Krzysztof\_Sakrejda](https://discourse.mc-stan.org/u/Krzysztof_Sakrejda)\
**Post date:** [December 20, 2017, 2:45pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/18 "2017-12-20T14:45:35Z")

</div>

> [@wds15](#):
>
> If you want to try to get those tests up and running that would be great. In case of any issues, I am happy to help out.

I can set it up locally, I’ll just ask separately if there’s an issue.

---

<div class="post-metadata">

**Author:** ![Bob\_Carpenter](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bob_carpenter/32/9230_2.png) [@Bob\_Carpenter](https://discourse.mc-stan.org/u/Bob_Carpenter)\
**Post date:** [December 21, 2017, 9:50pm UTC](https://discourse.mc-stan.org/t/boost-build-into-stan-math/2765/19 "2017-12-21T21:50:37Z")

</div>

> [@ariddell](#):
>
> My sense is that bundling external libraries is generally regarded as undesirable.

It’s a tradeoff. Bundling is easier for users as they don’t have to manage dependencies. You can see what a pain this is from something like the Tensorflow install instructions. Bundling is wasteful in that it can lead to multiple installs of the same software.

Given that our users aren’t unix sysadmins for the most part, I tried to err on the side of making it as easy as possible for them to install things. So we bundle the OS libraries on which we depend into Stan. You don’t have to use them for PyStan if it’s easier to do some other way.

> [@ariddell](#):
>
> How do other, similar software packages handle this issue?

For CmdStan: bundled, but you can use a command-line directive to use a different version than the bundled ones

For RStan: bundled (has to be for CRAN), but also has the option of pointing to different library version

Because CRAN forces RStan to remain consistent with the CRAN BH package (Boost headers), it really restricts what we can do. This is the main drawback with forced external linkage. But it’s not an intrinsic property of external package management.

Lots of packages on GitHub include other packages as submodules because Git makes that easy.

Obviously there’s a trend toward bundling with things like Docker (but I believe you can have a partial Docker image that links externally).

Tensorflow uses external dependencies, so their instructions on how to deal with all the management that entails is rather long: [https://www.tensorflow.org/versions/r0.12/get\_started/os\_setup](https://www.tensorflow.org/versions/r0.12/get_started/os_setup)  
We’ll be going down that route for MPI and GPUs, I’m pretty sure.

Hmm—looks like Tenosrflow only works with Nvidia GPUs, just like MxNet.