One Compiler Per OS

Every open-source project should test within its resources, although Stan has more resources than most projects and has trouble spending all of it. I would say RTools on Windows plus anything additional PyStan needs, (pure) clang++-6.0 on Mac, and g++ 7 or 8 on Linux plus some MPI wrapper.

But the details of that isn’t my main concern.

It is pretty much zero outside of Mac / Windows. I am sure it is considerable on Windows, and Mac is somewhere in between.

I don’t understand the TensorFlow analogy to Stan. Would our R package would be a wrapper that downloads a compiled stanc? If so, that is a solution to an essentially non-existent problem because the binary rstan packages that Mac / Windows people download from CRAN parse valid Stan models in >99% of cases. And non Mac / Windows people can generally compile stanc.

Where people run into trouble is that they parse a model but can’t compile it because their toolchain is non-existent or misconfigured. I am also having trouble seeing how the proposed solution aligns with the problem. Are we going to create a toolchain installer for Windows and Mac that is better than the RTools that Jeoren Ooms (for Windows) and Coatless (for Mac) have been developing?

I agree with you that should basically be the policy, but that is not what Sean proposed

(and @bgoodri, I tried to clear this up in the discourse post, but no one is proposing WONTFIXING anything - just not testing against these compilers and not obligating anyone in particular at Columbia to fix issues that do arise)

R package maintainers are obligated to fix issues that do arise on the compilers CRAN tests and I am not happy about the prospect that other paid Stan developers at Columbia would not help me with that. I also do not agree that a reasonable solution to this problem is to pull Stan off of CRAN.

Debian package maintainers are obligated to fix issues that arise on the compilers they test and I am not happy about the prospect that other paid Stan developers at Columbia would not help an unpaid Debian developer who maintains Stan. You can substitute any downstream project for Debian in that sentence.

I believe Sean has been convinced that we should not only fix bugs but also test a version of g++ on Linux in order to make sure MPI works well with Stan. I don’t think he and others at Columbia are convinced we should have any obligation for g++ bugs that apply to a later version of g++, even one that is the current stable release. That is essentially unheard of in open-source projects, except for things like OpenBSD.

I defer to Ben and Jonah on this one! But I agree with Bob that installation issues are real. It just came up recently with a reviewer of one of our papers!
A

The problem we’re trying to solve right now isn’t that CRAN users can’t download binaries, it’s that we have too many compilers to support. Being on CRAN the way we are imposes (at least) three dependencies:

  1. We have to be compatible with any compiler CRAN is compatible with. (This is the immediate problem being addressed in this topic)

  2. We have to use the the version of Boost packaged with BH.

  3. We have to use the version of Eigen packages with RcppEigen.
    `

What came up?

What came up was that we had some rstanarm code in a paper, and a reviewer said that he tried to install rstanarm on his computer but couldn’t succeed in doing so. We just removed the code from the paper, that was the simplest option.
A

Just to clarify: When I said, “I defer to Ben and Jonah on this one,” I was responding to Bob’s question: “Would being on CRAN in the same way as JJ Allaire’s tensorflow package or @andrewgelman’s R2WinBUGS package count as being on CRAN in the way that matters to @bgoodri, @andrewgelman, or @sakrejda?” I saw that question but not the rest of the thread, perhaps because Bob included my name in the question?

Any way, I don’t really understand exactly what does it mean to be on CRAN. Ben and Jonah know more about this than I do.

You must have email alerts set up or something. Glad to kow we can get your attention by typing “@andrewgelman” into a thread.

I don’t mean to be rude, but I think you’re overestimating a cluster user’s ability to change what’s installed system-wide on a cluster. First, clusters often have optimized proprietary MPI implementations, and the corporate IT staff probably don’t have access to their sources in order to compile them with a new compiler. Second, more likely than not, a request to either IT staff or a cluster vendor to install a new compiler and compile an MPI implementation with it (or pretty much any request that isn’t fixing something broken) will likely go down a black hole, if it can be made at all.

Just something to keep in mind.

1 Like

I’m not convinced the segfault was a GCC bug. I used Eigen in an undocumented way. I moved to a documented use case and the segfault went away.

I assume we’re breaking some CRAN rules already. Aren’t they more like suggestions?

So that’s
Rtools, Windows, vanilla (vanilla meaning no MPI/GPU)
clang, Mac, vanilla
g++, Linux, vanilla

On top of this we’d need
g++, Linux, MPI
g++, Linux, GPU

And I’d really really like to throw in
clang, Linux, vanilla

Is that a workable list? In terms of turnaround, is there some way to stage these tests? Like by default a pull only tests against one of the Linux targets, and then before merging we unleash the full array of tests?

What’s your reasoning here? Clang will run on a Mac, giving you its error messages that way… Do you think it will have different failure modes on Linux?

The tests are staged and take about 7 hours now (I think without anything else running at the same time) for each commit to a PR. I like @ChrisChiasson’s idea to run the much more marginal tests (e.g. we don’t really expect non-MPI tests to catch anything the MPI tests didn’t catch) only on merge to develop (well, he suggested pre-merge to master, but I like the idea of running them on merge to develop so that we know which merge caused the breakage). If they don’t catch anything for a while that way we could always change that to daily/weekly/pre-release.

Not rude at all! I didn’t realize that we’d actually be trying to support cluster manufacturer’s proprietary MPI implementations. Can we even hope to do that without access to those same implementations and compilers? Is that a significant chunk of our potential academic MPI users? What about industrial?

[edit] And do they typically have the option to run a less optimized open source MPI implementation with GCC?

If you are saying we are currently obligated to help with that, I’m fine with continuing to do so 👍

I do think the tensorflow-style CRAN option (or whatever you want to call it) is extremely attractive and deserves its own conversation, but is essentially unrelated to this thread at this point.

This I’m a little less sure about - you’re saying if someone posts an issue about Stan Math failing on GCC 9 (or whatever-we-test+1) that it should be a priority for the four of us at Columbia? So far I haven’t really noticed any system for prioritizing issues so I’m having trouble operationalizing what you’re proposing (specifically around what it means for us to have an obligation here - do we have an SLA? Do we prioritize compiler bugs above other kinds of bugs? Do we prioritize bugs over new features?).

I think that depends on what you mean by “support.” If there’s a problem with the proprietary implementation, I would hardly expect Stan developers to be able to do anything more than recognize that there is a problem with that implementation and point users to the support staff for the cluster. However, you’ll have to expect that users of Stan on those clusters will almost certainly be using a proprietary implementation.

I’ve noticed that on a cluster, there’s usually a “canonical” MPI implementation provided by the cluster’s vendor. For SGI clusters, that’s SGI MPT, and for Crays, that’s Cray MPI. Whether there’s an alternative to that implementation depends upon the cluster. On SGI clusters, I’ve noticed that there is usually at least one alternative implementation which works at least as well as MPT. However, on those clusters, that implementation is often Intel MPI, which is also proprietary. On Crays, using anything but the default MPI implementation is dicier. There is a so-called “cluster compatibility mode” that allows for alternative MPI implementations to be used, but in my experience, it is both cranky and slow.

In my experience, I’ve also found that support staff of clusters either highly discourage or outright forbid users from using an MPI implementation that isn’t already installed on the cluster.

FWIW, my experience is with clusters provided by some sort of supercomputing center that provides remote access to its clusters.

I suggest that you might want to look at what other open-source projects that use MPI already do. You may also be able to get some degree of informal help from those who already use a cluster, though of course that depends on how busy they are, whether they’re working on something that would exercise Stan’s MPI capabilities, etc.

It really depends. However, so far, the proprietary implementations I’ve seen do support GCC.

1 Like

How about we take the stepwise approach further and have three main branches in the future:

  • master
  • staged
  • develop

Before a release we merge develop into staged and hunt down all remaining bigs with a bigger test suite. Staged then is merged as new release into master (and develop) once everything is good. This is like debians stable / testing / unstable thing. Just a thought.

Btw, on Linux we need gcc. Ideally we support the main compilers of the main enterprise distros… rhel, suse & Ubuntu lts. But that will be difficult as those are usually old … rhel is at gcc 4.8.5+ (it has lots of additional patches). These compilers are easiest to get access to on clusters usually.

Well that’s really good news. Seems like for proprietary MPI issues a decent course of action might be to wait until someone comes to us with a problem and has an incentive to work with us to get it fixed and ideally tested somehow. Do you recommend any particular open source project that tests MPI support?

@wds15 To me develop is that interstitial branch, haha. But we could make another one so that develop has basically the same standards as master. Just requires telling everyone to make PRs to that branch (and then I can make Jenkins run some tests selectively when attempting to merge from that branch into develop).

Well, I’ve encountered a few open-source projects over the years that use MPI. The ones that cross my mind that I think still have publicly available mailing lists are PETSc, ABINIT, and Quantum ESPRESSO, though it’s been a while since I’ve used them.

I agree with what Sean said below that this tensorflow-like installer is tangential to Sean’s main point in this thread about limiting the number of compilers we do unit tests with and to my main point of not limiting the number of compilers we fix bugs with to those that we do unit tests with.

But it is a red herring. You asked that about it in the context of “How many users do we lose to failures to install toolchains on their platforms?” And having a tensorflow-like installer for rstan would do nothing to address the difficulty of installing a toolchain. And having a tensorflow-like installer for rstan would do nothing to address the versioning thing with Boost and Eigen because users would still need BH and RcppEigen to compile their Stan programs.

You have been harping on Stan being forced to match with the versions of BH and RcppEigen for years. And I have never been convinced this has had much merit. Boost.MPI is the only time Dirk said he wouldn’t accommodate us and that is because it is not header-only; it does not have anything to do with whether it is 1.65, 1.66, 1.67, 1.68, etc… Eigen rarely does an official release so the one in RcppEigen is almost always the current one.

I always have been willing to preempt what is in BH with newer or patched versions of Boost libraries that live in the inst/boost_not_in_BH directory of rstan, which I deleted yesterday because the seven commits it contained since I first uploaded to CRAN are no longer necessary when C++14 is required.

We would be well within the norms to not fix bugs in very new compilers that are not part of a major distribution, particularly for compiler bugs that just crash the compiler. In that case filing a solid issue is good for us so I would push the user who finds the crash to produce a complete bug report to gcc/clang. We should track those bugs because we push c++ templating further than a lot of other projects and without a bug report from us it may not get fixed before it becomes our problem. Sometimes that means that smaller distributions (like Arch linux that I’m using) get left out in the cold and that’s part of the reason they don’t have as many packages as, e.g., red hat or Ubuntu. So taking that to its logical conclusion we’d be fixing bugs and finding work-arounds for a block of compiler versions starting with whatever ancient Red Hat Enterprise Linux compiler there is up through the latest default in Fedora or non-LTS Ubuntu.

Re: clusters, the couple of clusters I’ve had experience with are all decent at helping coordinate bug-fixes for their standard installed software so if they have users with Stan and their default proprietary compiler crashes with it, they can poke the vendor for a solution because they have a service agreement. That might come back to use as a suggested compiler-specific temporary fix but I wouldn’t assume it’s always a black hole.

I think the reason to try to cover (do easy fixes, track the ones we can’t do) a pretty broad range of default configurations is that they come with users and devs who are willing to squash bugs for their particular config. That’s part of my reaction here, I want us to avoid the bunker mentality where it seems like supporting platform X is just a liability. So sure, let’s prioritize, but I’d like to think (and post a separate thread) about how to take advantage of our users and package maintainers. For example CRAN can be irritating, but ultimately they do test a broad range of compilers for us (maybe we shouldn’t test all those, but we should squash the bugs they bring up essentially in exchange for the service.).