RStan dependencies management

Developers
1

Hi!

RStan at the moment seriously suffers from CRAN requirements caused by down-stream R packages when we change Stan version numbers. So what happens is

  • StanHeaders 2.x is on CRAN
  • RStan 2.x is on CRAN which depends on StanHeaders >= 2.x
  • all packages with compiled Stan models depend on StanHeaders 2.x and Rstan 2.x

When we update now Stan to 2.x+1 then the preferred route is

  1. Bump StanHeaders to 2.x+1 => Then we have on CRAN StanHeaders 2.x+1 and Rstan 2.x
  2. Next bump Rstan to 2.x+1

BUT: During step 1 everything needs to still work! So all packages must keep compiling on CRAN as a requirement during step 1. This requires that the Stan services of version 2.x+1 must work with 2.x. Same holds true for Stan-math. Usually this is not a problem as things stay sufficiently compatible, but lately this has been the root cause of huge delays.

We need a solution for that.

Here is a proposal from my side as a solution/workaround:

  • We introduce in StanHeaders two new functions (motivated by what RcppParallel does)
    • StanHeaders::CxxFlags(stan_version="2.x")
    • StanHeaders::LdFlags(stan_version="2.x")
  • Then we instruct all down-stream packages that they have to add this into their Makevars:
    • PKG_CPPFLAGS += $(shell "${R_HOME}/bin/Rscript" -e "StanHeaders::CxxFlags(stan_version="2.x")")
    • PKG_LDFLAGS += $(shell "${R_HOME}/bin/Rscript" -e "StanHeaders::LdFlags(stan_version="2.x")")

This will give us control over the dependend packages compiler and linker flags. Thus, when StanHeaders now moves to 2.x+1, we can instruct the CxxFlags + LdFlags function to emit compile flags which point to sources compatible with 2.x when called with the Stan_version set to 2.x.

To make it concrete - suppose 2.x+1 Stan services break the API wrt to 2.x. Then the call to CxxFlags() of StanHeaders 2.x+1 will work for the case of breaking changes like

  • CxxFlags(stan_version="2.x") => -I StanHeaders/include/v2_x
  • CxxFlags(stan_version="2.x+1") => -I StanHeaders/include/v2_x+1

In case there are no breaking changes we can emit things like

  • CxxFlags(stan_version="2.x") => -I StanHeaders/include/v2_x+1 (because we know it works)
  • CxxFlags(stan_version="2.x+1") => -I StanHeaders/include/v2_x+1

Or something like that. I hope that the logic is clear.

A risk is that all Rstan dependent packages need to follow this convention. We can probably force the StanHeaders dependent packages into this logic by deploying our code in subfolders which carry the version. This way the packages must make either their Stan version explicit in their compile time flags or they use our utility function. In both cases things will still work just fine. It’s the package maintainer’s duty then to use our utility functions to keep getting the latest and greatest or he must be specific about the version - which is what we want, I think.

Moreover, we would require that package maintainers keep bumping the Stan version level every now and then as we would drop support for older versions eventually (probably x-1, x, x+1 needs to be there).

Other ideas?

Sebastian

3 Likes
2

I’m likely missing something, but anyway – couldn’t this sort of stanheaders version dependent logic simply be incorporated into rstan itself in some way? I’m fine to do the mods and super happy for whatever improves the situation, just seems surprising (to my very limited understanding of such mechanics) that it needs to be handled on an individual package level.

3

The other packages compile Stan models in ways we do not control - that’s the issue.

4

I think having StanHeaders::CxxFlags and StanHeaders::LdFlags would help. They certainly wouldn’t hurt. This release is the first time in a while where we have had to add new flags, with the previous notable one being when we started requiring CXX14. In the past, we have mostly relied on rstantools to create an appropriate Makefile, but it would be better if the appropriate Makefile was mostly just calling StanHeaders::CxxFlags and StanHeaders::LdFlags`.

What is different is that rstantools 2.x will by default generate the .cc and .h files in the src/ directory when the package is built. After 2.21, we need to get a bunch of packages to update to the new build process, but historically speaking that will take a lot of time and PRs.

5

The additional suggestion here is to pack things into versioned sub-folders so that we will have the ability to easily deploy different stan services, for example.

How about we coordinate the 2.21 update hell with via rstantools to also include StanHeaders::CxxFlags + StanHeaders::LdFlags ?

I suggest to handle this with a minor StanHeaders version bump.

6

I don’t think CRAN will want us putting multiple copies of everything into v1/, v2/, etc. Also, at the moment there are packages expecting src/ etc. to be at the top level of the include directory of StanHeaders.