One Compiler Per OS

Hey all,

We’re looking to reduce the number of compilers that we test to one per OS. We’re not saying we will close issues that people make for other compilers, but that we won’t include the compilers in our CI test suite (and will recommend people use one of our tested compilers and even eventually provide install support for that situation on our 3 OSes).

Here are the proposed compilers:
Mac: clang 5
Linux: clang 5
Windows: gcc 4.9.3 (following the Rtools version)

There has been some existing discussion here because I made the issue after everyone seemed to agree in the meeting, but it seems like it could use a little more discussion.

I have a PR open to begin the move. Most of the contention seems to be whether we should instead choose a modern GCC flavor on linux, my argument against being that it’s best to limit the # of different compiler versions (so having clang 5 on both linux and mac helps us speed up development and testing), clang is much faster and more stable than GCC, and that GCC warnings are often far more spurious (see this post from the Chrome team). The opposing side makes some compelling claims that gcc is the normal compiler for many Linux distributions and that it would be bad if we did not adhere to this norm, though we notably crash gcc-7 with at least one of our math tests (internal compiler error).

Thanks for moving to Discourse.

Also, I should have written this in the original, but I think we need to write why we’re talking about this in the first place. Testing and development have slowed down considerably on Stan and some of this is because of the time it takes to work around compiler bugs in versions of compilers that we have no reason to think our users are using. So there is a lot of extra work being done and we don’t know if anyone even cares. So one of our main goals here is to more closely align the scant resources we have with high ROI projects. Another one is to encourage more open source contributions from people who don’t necessarily have time to address obscure bugs in outdated compilers. We want people to be able to easily install and use Stan and we’re moving towards that separately, with the goal eventually being installers that package our one supported compiler per OS in a way that doesn’t require root access. It will be a larger package than a current Stan install, but it will categorically avoid many of the issues we’ve had around compilers and installation.

Bob, did you want to add stuff here? I’m sure I’m missing something.

I can’t think of anything you missed.

For Linux, not supporting GCC is a very bad idea. GCC is ubiquitous among Linux distributions; clang, not so much. Furthermore, if you are intending to support MPI, going with clang is probably even worse. I cannot recall a Linux MPI cluster where GCC was not an option, but I have yet to see one with clang.

1 Like

Does your analysis here change if we’re including clang as part of our install?

Not really. Again, think about the clusters. When they provide GCC, they also provide an implementation of MPI compatible with it. (I would not recommend trying to provide both MPI and clang; especially since cluster maintainers may not support an MPI implementation that isn’t already installed on the cluster.)

1 Like

I was asked to summarize my counter-proposal on Discourse, which is

  1. Test within our resources
  2. Paid Stan developers should fix known bugs with open-source compilers >= a minimum version

There are several reasons for this. One is that we need Stan to be on CRAN, CRAN tests a handful of compiler / OS combinations

https://cran.r-project.org/web/checks/check_flavors.html

and if a package fails, then the maintainer is expected to fix them. If Stan tests less than CRAN does, then some issues will (and have) inevitably get through. That is okay. However, those issues are going to get fixed, and if I am the only one fixing them, then … after 5 minutes I still don’t have a civil way to conclude this sentence.

I would also point out that essentially zero open-source projects don’t feel obligated to fix bugs involving gcc. And many projects that have fewer paid developers and less funding do fix bugs involving either gcc or clang. Finally, do we really have the audacity to say that unpaid maintainers of downstream open-source projects should have to maintain a fork of Stan that fixes gcc issues that paid Stan developers do not feel obligated to fix?

1 Like

Have you tried installing clang? From source?

Can you be more specific about which compilers on which OSes you’re proposing?

I’m not saying we’d close compiler bug tickets, just that we’d have a slightly smaller set of compilers we test with CI constantly (and not allow anything through that breaks them). There’s no forking involved - we review and accept PRs that fix compiler bugs.

The MPI stuff is pretty compelling as well - does anyone want to figure which / if any modern version(s) of GCC don’t have this segfault regression bug on our code? Then we can see how bad gcc’s spurious warnings are and evaluate if it’s worth fixing them all, or backing off from our zero tolerance policy towards warnings.

Who are you including in this list?

I would say the tests should run with a recent g++ on Ubuntu, at a minimum. If we don’t have the capacity to test clang++ on Linux, that is unfortunate but hopefully testing with a recent enough clang++ on Mac gets us most of the way there.

My main complaint with this proposal is not with the amount of testing, it is with the idea that we are only obligated to fix bugs that we notice with our test infrastructure (even if we accept volunteer fixes in other cases). Who does that besides Free/Open/foo/-BSD?

You, me, Mitzi, Sean, Jonah (although I don’t think Jonah can help that much with C++ issues), and whoever accepts our offer for the open dev-ops position.

I think I’m starting to understand your perspective - for what it’s worth, my norms here were developed in industrial settings where we worked to create situations where we had control over the compiler rather than working to support many at once. I also don’t totally understand the notion of obligation or support here - I have no idea how that really works in an open source project. Like it seems fine if we allow the community to sort of self-regulate in this area - if the issue crops up and many users complain, likely someone will feel enough pressure to take a crack at fixing it.

My understanding of the norm with open-source projects is that you err on the side of assuming too much responsibility. And people who are paid to develop something should rarely be pushing responsibility onto unpaid people.

For example, with CRAN. If their checks find a problem with one of our packages, do we expect CRAN to send us a PR fixing it? Hell, no. CRAN has 12752 packages as of this morning to deal with and expects maintainers to agree to its policies (including, among other things, fixing bugs) if they want CRAN to distribute the package.

I suspect the sense of “obligation” at play here is that Ben thinks we have to support all modern compiler versions in order to not put off a bunch of users.

I think it might be helpful to start thinking in terms of the bigger picture and the opportunity costs of supporting multiple compilers. If we spend time supporting a range of compilers, operating systems, and interfaces, it’s time we can’t spend on improving new features.

I’d for instance like to prioritize Python installation more heavily than we have. So far, everybody fails with PyStan users at courses and they have to wind up using RStan.

I barely have time to code any more. But I think I still count as a developer spending all my time in design discussions like this one.

1 Like

What would the costs be to us if we dropped CRAN support and moved to GitHub-based installs?

1 Like

More on norms. Take Andreas Tille, who has never posted on Stan’s Discourse / Google Group / GitHub, but is packaging StanHeaders, rstan, brms, etc. for Debian, which is something we thought would be a good idea since at least August 2013.

I am not going to tell (people such as) him that he is on his own for any compilers other than clang++-5 (which is not the primary compiler on the 20+ platforms Debian builds on and not even the version of clang++ that Debian tests with) and then ask him if he could split Stan out into a separate package so that it could be used by a PyStan Debian package.

Users would have to install from source, which would not work on some of the more memory-constrained laptops, and we would lose a dozen or so packages like rstanarm that do not require the user to have a toolchain.

All open-source projects face this issue. Which do not accept responsibility for gcc?