Rstan -- maintenance / future prospects?

Sounds good to me! Maybe @bgoodri or @rok_cesnovar will have a reason not to do that, but I can’t think of one.

1 Like

By recent developments do you mean 2.27? If yes then I think this is a good idea

How difficult is the backwards compatibility? Should we think about rstan3?

Yes, not only https://github.com/stan-dev/rstan/pull/887 but also targeting the new branch in other PRs. That PR is getting huge and more difficult to review. After merging it into an experimental branch, we can build on it and review smaller PRs to address the open issues or add new features.

It isn’t difficult at all, but it may take some time if more reverse dependencies need patches. The last time we checked, StanHeaders 2.26 supports rstan 2.21 by hsbadr · Pull Request #912 · stan-dev/rstan · GitHub addressed the backward compatibility issues except for OpenMx which required a patch (Support Stan v2.26+ / Math v4.0+ by hsbadr · Pull Request #308 · OpenMx/OpenMx · GitHub) that is now merged and is on CRAN. If we decided to upgrade to 2.27, csr_matrix_times_vector2 now uses csr_matrix_times_vector by SteveBronder · Pull Request #514 · stan-dev/rstanarm · GitHub should be merged and might need to be applied to other packages.

I don’t think that we need to rename the package unless we want to merge both StanHeaders and rstan into a standalone package. In that case, we can actually keep the name as RStan since R is case sensitive. RStan would be a new package to CRAN, but all other resources remain the same including the GitHub repo. It’s actually called RStan, not rstan, in almost all references. Reverse dependencies can then take their time to upgrade and link to RStan instead of rstan and StanHeaders that will be also available on CRAN. I’m not saying that this is the best approach, but it’s the fastest way to reset the release issues and easily plan for future releases of a standalone package.

PS: Packages should be named in a way that does not conflict (irrespective of case) with any current or past CRAN package (the Archive area can be consulted), nor any current Bioconductor package.

2 Likes

Yes, definitely, lets do that!

1 Like

@bgoodri Reverse dependencies (e.g., OpenMx) now need to link to RcppParallel for Intel TBB. Should we inform the maintainers and wait or take the easiest route and release a new package rstan3 with bundled StanHeaders and let them take their time to upgrade (from rstan + StanHeaders to rstan3)? I can create rstan3 v2.27 that’s ready for CRAN submission. Otherwise, we’ll have to wait until the affected dependencies release a new version linking to RcppParallel.

1 Like

Not sure what is the deal with CRAN and versions, but just came here to comment about RStan3

PyStan (PyStan3) now follows the new API. Also, at v2-> v3 step, we upgraded the package structure (PyStan=frontend, HTTPStan=backend).

PyStan interface is very light and handles only Python side of things.

HTTPStan only handles Python-Stan C++ interface and is kept simple.

It might be that in future HTTPStan is replaced with some implementation of ServerStan.

Does this include packages which depend on both rstan and StanHeaders (i.e. most packages) or only packages which depend only on StanHeaders i.e. OpenMx?

@hsbadr if the rstan + stanheaders formulation is still desired, wouldn’t an easy solution be to make them distinct, ie so that rstan doesn’t depend on stanheaders?

1 Like

Thanks @ahartikainen! I meant by RStan3 a new standalone package based on the experimental branch, which will make it easier to upgrade and release to CRAN while resetting the reverse dependencies (and removing StanHeaders dependency with bundled headers). It can take any name, but rstan3 is in sync with stanc3.

This is in the context of releasing a transition version of StanHeaders that works with the old version of rstan. If you maintain a package, test it with StanHeaders v2.26.2 (StanHeaders_2.26 branch) and the CRAN version of rstan.

That’s what I meant by RStan3. Packages that currently depend on rstan may be linking to StanHeaders too, which currently provides conflicting headers. If a new package isn’t desired, we’ll have to coordinate with the maintainers.

2 Likes

@Charles_Driver @jeffreypullin I’ve updated the source and binary packages for the transition. If you maintain a package that depend on StanHeaders, it’d be helpful if you can test it with the combination of StanHeaders v2.26.2 + rstan v2.21.2 and StanHeaders v2.26.2 + rstan v2.26.2.
You may install StanHeaders v2.26.2 using:

install.packages("StanHeaders", repos = c("https://mc-stan.org/r-packages/", getOption("repos")))

With just the stanheaders upgrade, I get a pile of errors building ctsem. Wouldn’t it be worthwhile including a test build of some of the packages using rstan in more complex ways, so we don’t go through the same back and forth as in January though? :) Can’t post error here (too large) so will add to the long issue on github…

StanHeaders 2.26.2 + rstan v2.21.2 works with rater (my package).

More warnings are shown however, and compilation does feel a little slower.

Do you have USE_STANC3 defined? If so, the C++ code is generated by stanc2 and you try to access the new incompatible headers.

Great! I think the main test here is to pass R-CMD-check for CRAN submission; that could be done by revdepcheck::revdep_check(). Then, we need to make sure that the transition won’t affect the subsequent upgrade of rstan.

1 Like

Ok. with -DUSE_STANC3 removed, then new stanheaders + old rstan is fine. But to get new stanheaders + new rstan working, I still require -DUSE_STANC3.

That’s exactly the purpose of USE_STANC3. The new version of rstan uses stanc3.

We don’t understand each other somehow :) I don’t see how the transition will work. If I patch ctsem so that it works with the new stanheaders and old rstan, it’s then broken when rstan gets upgraded.

Check the discussion here. You need to define USE_STANC3 only if (utils::packageVersion("rstan") >= 2.26).

Discussion didn’t help me. Sorry if I’m being slow. As a package maintainer, I am not in control of what utils::packageVersion("rstan") returns, yet it seems that you expect me to appropriately modify the package makevars.

I see. If you’re using GNU Make, you may add the following, or a variant of it, to the compiler flags:

$R_HOME/bin$R_ARCH_BIN/Rscript -e "cat(ifelse(utils::packageVersion('rstan') >= 2.26, '-DUSE_STANC3', ''))"

You’ll need to get just about every package using rstan to do this, you realise? My makevars has the following lines I couldn’t figure out a working variant of what you posted.

STANHEADERS_SRC = $(shell "$(R_HOME)/bin$(R_ARCH_BIN)/Rscript" -e "message()" -e "cat(system.file('include', 'src', package = 'StanHeaders', mustWork = TRUE))" -e "message()" | grep "StanHeaders")

PKG_CPPFLAGS = -I"../inst/include" -I"$(STANHEADERS_SRC)" -DBOOST_DISABLE_ASSERTS -DEIGEN_NO_DEBUG -DRCPP_PARALLEL_USE_TBB=1 -DUSE_STANC3
PKG_CXXFLAGS = $(shell "$(R_HOME)/bin$(R_ARCH_BIN)/Rscript" -e "RcppParallel::CxxFlags()") $(shell "$(R_HOME)/bin$(R_ARCH_BIN)/Rscript" -e "StanHeaders:::CxxFlags()")
PKG_LIBS = $(shell "$(R_HOME)/bin$(R_ARCH_BIN)/Rscript" -e "RcppParallel::RcppParallelLibs()") $(shell "$(R_HOME)/bin$(R_ARCH_BIN)/Rscript" -e "StanHeaders:::LdFlags()")

CXX_STD = CXX14