Software I develop and work with depends on rstan. rstan has had a dependency on an older version of withr for quite some months now, causing headaches for installation. This dependency is now breaking testthat – I can either choose to have stan able to compile, or testthat able to test. I understand there are hassles involved in getting rstan releases out, but I just wonder if there is much sense of how things look for the future – is rstan likely to lag further and further behind stan, will bugs like this thing with withr be likely to hang around for many months at a time, or is there some optimism re possible improvement? Wondering how seriously to start thinking about a dependency shift in my software…
8 Likes
As another maintainer of an R package which depends on {rstan} I second this question/sentiment.
2 Likes
The current lag is mostly related to the switch to the new Stan-to-C++ translator (stanc3) and once we have a rstan version that support stanc3 on CRAN the lag should be much smaller (at least until the next change of this magnitude - we dont anticipate one at this point though) and contained.
There is a huge PR that should bring rstan to 2.25 or rather now 2.26 (https://github.com/stan-dev/rstan/pull/887). And then CRAN. I am optimistic once this is squashed the release cycle should improve.
@bgoodri can any of us help you with this process in any form? Testing or anything.
There will always be some lag between the Stan and rstan version just because of the tight coupling of C++ and R and also the CRAN limitations.
9 Likes
ok great. Is an encouraging PR :)
1 Like
@Charles_Driver @jeffreypullin @Stephen_Martin would you be willing to be alpha testers if we build binary versions of rstan 2.26 and host them on our R package repo?
I do not have the time right now to thouroughly test it and I honestly dont know all the ins and outs of rstan. I tagged your three because you expressed written interest for it. I know that there are a lot of users/devs with similar thoughts to yours but I want to keep this limited to collect the first round of feedback.
By alpha testing I mean check if its works for what you typically do.
With this huge of a PR as we have fell behind so much, we will need some help for the first round of testing. It would speed this up a lot.
1 Like
Sure.
1 Like
Yes, I’d be willing to. I currently have a few packages built around rstan (LMMELSM, MIRES, ICCier, omegad; all on my github), and I’d like to ensure they still work of course. No better time than pre-release to do so.
Edit: Also, I can be a build tester; I use linux as my daily driver, so I don’t mind compiling rstan myself also.
1 Like
I’m willing to!
1 Like
Ok, here is a first try:
Run
remove.packages(c("StanHeaders", "rstan"))
install.packages("StanHeaders", repos = c("https://mc-stan.org/r-packages/", getOption("repos")))
install.packages("rstan", repos = c("https://mc-stan.org/r-packages/", getOption("repos")))
This should install binary versions if you are using R4.x on Windows/Mac. On Linux it installs from source as usual.
Disclaimer for anyone else coming to this topic:
Please do not try this if you do not have experience with installing complex R packages and how to debug installation issues and you currently have a working setup that you need for work/research.
This is very much experimental.
2 Likes
Where do you want the bug list? As separate issues or just here? I didn’t get to try compiling yet because stanc has (incorrectly I believe) changed behaviour and requires a file specified, even though model_code is still available as an argument.
2 Likes
Please post that here: Issues with the experimental version of 2.26 from PR #887 · Issue #899 · stan-dev/rstan · GitHub
1 Like
Some time down the line, any chance of an update? Is there really some plan to keep rstan (or some similar functionality) working, or should I have jumped ship when I was thinking about it earlier?
1 Like
I think a large part of the problem is that the current maintainership status of rstan.
Officially, bgoodri is the maintainer, but he doesn’t seem to have the bandwidth to perform needed maintenance at the moment.
Therefore, I think a critical first step in continuing the maintenance of rstan would be to create a group of people who officially share in the responsibility for it’s maintenance.
Just my thoughts :)
Best,
Jeffrey
1 Like
I’d say, yes. We already have an experimental version that works. Though, the conflicting headers between the old and new Stan/Math would delay the CRAN submission. This shouldn’t take more than one minor release cycle for some of the dependencies that might need minor patches to switch (e.g., adopting USE_STANC3 macro based on the installed version of rstan and/or checking STAN_MATH_MAJOR).
Agreed. While I’d like to see the new version on CRAN, I understand why @bgoodri doesn’t want to pull the trigger without a smooth transition. This is a major update with breaking changes that should be handled strategically for the future of rstan. Though, it really took much longer that it requires. The slowest solution, which involves a coordination with the maintainers of the reverse dependencies, would’ve been completed early this year, in less than a couple months.
@bgoodri @jonah @rok_cesnovar May I create a new branch experimental and maintain it (and target it in the current PRs)? This will allow us to consolidate the recent development and then we can merge it into develop when we all agree that it’s ready.
3 Likes
Sounds good to me! Maybe @bgoodri or @rok_cesnovar will have a reason not to do that, but I can’t think of one.
1 Like
By recent developments do you mean 2.27? If yes then I think this is a good idea
How difficult is the backwards compatibility? Should we think about rstan3?
Yes, not only https://github.com/stan-dev/rstan/pull/887 but also targeting the new branch in other PRs. That PR is getting huge and more difficult to review. After merging it into an experimental branch, we can build on it and review smaller PRs to address the open issues or add new features.
It isn’t difficult at all, but it may take some time if more reverse dependencies need patches. The last time we checked, StanHeaders 2.26 supports rstan 2.21 by hsbadr · Pull Request #912 · stan-dev/rstan · GitHub addressed the backward compatibility issues except for OpenMx which required a patch (Support Stan v2.26+ / Math v4.0+ by hsbadr · Pull Request #308 · OpenMx/OpenMx · GitHub) that is now merged and is on CRAN. If we decided to upgrade to 2.27, https://github.com/stan-dev/rstanarm/pull/514 should be merged and might need to be applied to other packages.
I don’t think that we need to rename the package unless we want to merge both StanHeaders and rstan into a standalone package. In that case, we can actually keep the name as RStan since R is case sensitive. RStan would be a new package to CRAN, but all other resources remain the same including the GitHub repo. It’s actually called RStan, not rstan, in almost all references. Reverse dependencies can then take their time to upgrade and link to RStan instead of rstan and StanHeaders that will be also available on CRAN. I’m not saying that this is the best approach, but it’s the fastest way to reset the release issues and easily plan for future releases of a standalone package.
PS: Packages should be named in a way that does not conflict (irrespective of case) with any current or past CRAN package (the Archive area can be consulted), nor any current Bioconductor package.
2 Likes
Yes, definitely, lets do that!
1 Like
@bgoodri Reverse dependencies (e.g., OpenMx) now need to link to RcppParallel for Intel TBB. Should we inform the maintainers and wait or take the easiest route and release a new package rstan3 with bundled StanHeaders and let them take their time to upgrade (from rstan + StanHeaders to rstan3)? I can create rstan3 v2.27 that’s ready for CRAN submission. Otherwise, we’ll have to wait until the affected dependencies release a new version linking to RcppParallel.
1 Like
Not sure what is the deal with CRAN and versions, but just came here to comment about RStan3
PyStan (PyStan3) now follows the new API. Also, at v2-> v3 step, we upgraded the package structure (PyStan=frontend, HTTPStan=backend).
PyStan interface is very light and handles only Python side of things.
HTTPStan only handles Python-Stan C++ interface and is kept simple.
It might be that in future HTTPStan is replaced with some implementation of ServerStan.