One Compiler Per OS


No, not at all. Rather, the MPI standard only guarantees source compatibility, so if one is compiling one’s parallel code with a particular compiler, then in general, one should use an MPI implementation that uses the same compiler. There may be exceptions to this here and there, but that depends on the vendor of an MPI implementation.

I would not say that GCC is the only legitimate compiler for MPI. On the clusters, MPI implementations usable for at least a couple different compilers are available, usually Intel’s and GNU’s. What I would say is that I have yet to see a cluster with clang on it.

I’ve seen clusters that run some enterprise version of Red Hat or SUSE, and I’ve seen Cray clusters, which these days use their own custom version of Linux. I haven’t seen an Ubuntu-based cluster, so far.


If you get off CRAN you loose not only statisticians but also many biologists/ecologists and biostats people (and they do a lot more bespoke models than some fields), if we’re not testing the CRAN compilers we will likely need to just resolve those failures at release time.


Can you tell me more about this (maybe in a PM to not derail too much)? I don’t really know much about why those fields would need a package to be on CRAN but obviously need to learn about it!


I must say missing out linux + gcc + mpi is not a good idea. For MPI tests we should at least have this setting, even better with also linux + intel + mpi.


Sounds like we should run the MPI tests with GCC (or even the intel compiler - @yizhang why do you say that is a better target for MPI?). That’s a separate part of this combinatoric build matrix we can configure individually. But maybe it’s best to use that same compiler for the rest of the Linux tests, just to address problem (3) above and keep #(compilers) low overall.


Didn’t notice this has been pointed out by @jjramsey. I agree with him: linux + gcc/icc + mpi is ubiquitous for HPC infrastructure. For MPI tests, an ideal matrix would be gcc/icc x intel mpi/openmpi/mpich.


Right, but can you prioritize given we have limited CI and engineering resources? If you had to force rank the cells of that matrix…?


I’d say let’s include linux + gcc + mpich first.


If I had to force it, I’d pick GNU over Intel, since the former is ubiquitous on both clusters and normal Linux installations.

As for a choice of MPI implementation to test with, that’s trickier. I’d be tempted to choose OpenMPI simply because I’ve had more problems with it, but a lot of MPI implementations, including IntelMPI and Cray MPI, are based on MPICH, and they share some preprocessor macros used for working around certain issues, like MPICH_IGNORE_CXX_SEEK.


My opinion boils down to at least one gcc in Linux, and I’d prefer at least one clang (though if we only go with one gcc is my vote). I’d be comfortable with an MPI build using gcc and a non-MPI build using clang as like a minimum effort bit of testing. It convolutes the tests a bit, but whatevs. I’m neutral on non-Linuxes. @bgoodri what’d you put as a minimum set of test combos?

I just addressed an issue today where gcc -O3 (7.3.0) segfaulted but clang worked fine. CI caught it. I assume this would swing the other way if I was developing in gcc.


I would also like to see a test with debug build as -g -O0 or -g -Og. This often gives me happy surprises.


One of my points in this thread (echoing the Chrome team’s point here) is that including GCC does not actually make the code better - the errors at this point vs. clang seem to be mostly spurious or compiler bugs like the one you’re describing. Catching that does not make our code better in and of itself. So I would only suggest we use a buggy compiler like this if it’s necessary for something or someone we care a lot about (like MPI).

This thread is actually about limiting what we test on CI 😅


Anything I can do to make your life harder ;)


In addition to responding (see below), I really think we may be able to have our CRAN cake and eat it, too.

I think it’s better to think about what we want it to be, not what it
is now. For instance, we might be low on Python users because of compatibility and install issues in addition to being not-so-Pythonic.

I think it’s working with clang++ on my Mac. The special build target synchs the MPI version and the rest of the code.

The latest one we have at Columbia is running Red Hat Enterprise Linux 7 and the previous one, which we still have access to, runs Red Hat Enterprise Linux 6.

Ben and Andrew have also said that rstan has to be on CRAN to be taken seriously. Like Sean, I’m curious why everone thinks that. I thought the biostats (or bioinformatics?) people mainly put things on Bioconductor (we looked into that, but they were restrictive like CRAN in slightly different ways).

I hope we’re going to leave Intel compiler support to others (take PRs, but not worry about it ourselves). Of course, if Yi or Sebastian (the two who brought it up) may decide it’s important enough for them to support.

Intel’s been a huge pain because of all the compiler flags controlling the non-standards-compliant arithmetic. When we tried to run on Intel, many of our unit tests failed to achieve the same precision as in GCC and Clang and MSVC, so to keep things passing without ifdefs in unit tests, we lowered test precision in a lot of places.


There are several goals we are shooting here. For QA the proposed plan is good enough to me. On the other hand, we also need to consider users and many other developers. In close-door projects I’ve seen, throwing an outrageous suite of platforms to tests serves

  • meet customer requests
  • help finger-pointing among devs.
  • make manager feel better

Though we don’t officially have “customers”, testing on a few more platforms do help us interact with users and not been overwhelmed by building issues that we didn’t anticipate.


Oops, forgot to elaborate. This goes a bit off topic, but loops back to compiler support.

How many users do we lose to failures to install toolchains on their platforms? I doubt it’s zero. I believe the problem is way worse for PyStan, espcially Windows. The only way I’ve been able to install PyStan on my Mac is through Anaconda, which then messed my PATH to the point where the R terminal no longer worked (I went back and removed Anaconda from the PATH, but many of our users wouldn’t know how to do that).

Would being on CRAN in the same way as JJ Allaire’s tensorflow package or @andrewgelman’s R2WinBUGS package count as being on CRAN in the way that matters to @bgoodri, @andrewgelman, or @sakrejda? Greta is obviously in the same boat, as it depends on the tensorflow package.

The way the TensorFlow package works, the driver’s on CRAN, but there’s external software to download through an R script that’s part of the package, and then a lot of the actual running of Stan models is external to R rather than communicating in memory. The way R2WinBUGS works is by having WinBUGS installed separately and then linked in through the R package.

We’re already asking people to go beyond CRAN on Windows and Mac to install C++ compiler toolchains. For Windows, CRAN supplies an external installer for the Windows toolchain, rather than building it into the CRAN downloads themselves or into R itself.


If it wasn’t clear, I wasn’t saying that MPI can’t be built with clang, only that the Linux clusters I’ve seen don’t happen to have that particular compiler.


It’s not that clang cannot work with mpi, it’s for IT to deploy a new compiler on a cluster and interfacing with job scheduler is a pain.


I would say deciding between forcing volunteer Stan developers to maintain compatibility with a difficult compiler forever and giving some one-time pain to a corporate IT department (and they do know how to install packages) seems like an easy choice to me… :P

Agreed 100% both with this point and your “we can be on CRAN in the same way tensorflow is” point.

All this said, for now I think maybe I’ll try to set up a version of GCC on the linux box (which has currently hung itself trying to use the default gcc 5.4 to run our distribution tests) for just the unit tests (with and without MPI) but not the distribution tests, since those are very intense, require a good compiler, and don’t really mimic the actual way we use Stan other than the instantiation correctness.

Currently we’re testing openmpi on the linux box and mpich on the mac box, and the assignment of the MPI tests depends on the queue so is effectively random as to which box (and thus MPI library) will be chosen. This seems worse than standardizing, though, so maybe I will change them both to mpich?


I should’ve been clearer about the field divisions. In my experience biostats is more likely to have people who don’t know if they can handle a non-standard install path. That seems to be even more true of epidemiology. Those I have less familiarity with and I don’t know if CRAN is really a requirement.

In field biology/ecology/conservation biology (basically anything that’s not molecular/biochem/microbiology) it’s common for students/post-docs to apply bespoke models to complicated field data sets. It’s also common for there to be either no support or very limited support in terms of training or technical knowledge. For better or for worse R is a standard in the field and “I can install it from CRAN” is a litmus test for whether a package is worth trying.

If you could wrap clang into a nice installer with everything else C+±based and then have a thin pure R package on CRAN that interfaced with it you’d be in a similar position to where we are now so maybe “full rstan on CRAN” isn’t a strict requirement but it really needs to be a 1-2 step process.