Sharing a function to switch between interfaces to stan in R

Interfaces
1

A comment not a question. In case useful to others, here’s an R function to which you give a path to Stan code, the input it expects, plus a choice of rstan, cmdstanr or cmdstan, and it will compile & run the code and return a dataframe of the sampled parameters. I’ve found it handy when I want to switch between interfaces for their various relative pros & cons (probably best not to have a debate on that here). Source code is here, doc here, install from here (package is work in progress).

(One problem with this function is that if cmdstan finds a pre-existing binary that was compiled by cmdstanr since the source code was last updated, it thinks no compilation is necessary, but it will not allow multi-threading unless you force recompilation. Perhaps this is a bug in one of the two interfaces, perhaps I need to do something different…) Edit: fixed below

2

Thanks for sharing this.

(One problem with this function is that if cmdstan finds a pre-existing binary that was compiled by cmdstanr since the source code was last updated, it thinks no compilation is necessary, but it will not allow multi-threading unless you force recompilation. Perhaps this is a bug in one of the two interfaces, perhaps I need to do something different…)

Do you mean it was compiled with threading enabled (cpp_options = list(stan_threads = TRUE))) but it doesn’t allow you to use it? Or am I misunderstanding? We’re doing a lot of work on how the cpp options are handled for cmdstanr v1.0, so this might behave differently when that is released. If you want you can try out the v1.0 branch on GitHub to see if it changes this (I’ll be announcing a release candidate soon that is based on that branch).

I’d be curious to know in which cases you’d want to run cmdstan by itself rather than via cmdstanr since maybe that points me towards improvements we should make to cmdstanr.

3

Thanks Jonah.

cpp_options = list(stan_threads = TRUE))

That fixed it. It wasn’t needed to enable multithreading in cmdstanr, but was for when cmdstan reused the same binary.

in which cases you’d want to run cmdstan by itself rather than via cmdstanr

Only when my posterior became too big to fit in my laptop’s memory, crashing cmdstanr. Perhaps there’s other ways to fix that, but I wrote it to file instead with cmdstan and then excluded some rows and columns as I later read it in.

4

Glad that fixed it. I would have thought that would be needed for cmdstanr too. I think it will be needed with v1.0, which is coming soon.

In theory it should be possible to do the same thing with cmdstanr like this:

  1. fit the model with cmdstanr
  2. don’t call anything that causes cmdstanr to read the draws into memory (so no print, summary, draws, etc.).
  3. read in the CmdStan CSV files using read_cmdstan_csv(), which allows you to select a subset of the columns/variables if you want. Alternatively, use as_cmdstan_fit() which creates a fitted model object from the CSVs but lets you select a subset of the columns/variables.

That should avoid having to directly interact with CmdStan, although maybe there’s some reason why that wouldn’t work in your case. If it doesn’t work I’d be curious to know so I can look into it and figure out how to make it work.

5

Thanks, @ChrisWymant.

People sure do love to debate this kind of stuff. Among the developers, the only debate has been whether we can retire RStan and replace the back end of CmdStanR/CmdStanPy with something easier to update and maintain than CmdStan. The answer remains no as long as hundreds of packages depend on RStan. I fear moving cmdstanr to CRAN, which is the current plan, is going to also cement that in place and make it impossible for us to ever retire it.

Going forward developing new samplers, I plan to work outside the Stan ecosystem (like Nutpie did with PyMC and Stan) for two reasons. The main one is because it’s so hard to get anything through the Stan process because it’s so careful/conservative with updates (which absolutely makes sense given how many users we have). The second one is because I want to support approaches to log densities and gradients other than Stan.

@andrjohns is working on a new sampler infrastructure to replace RStan, but the only way this can actually replace RStan is if it can act as the back end to all the packages that depend on RStan.

6

I wasn’t planning on commenting on these things, but since Bob went down this road… I guess I’ll take the bait ;) I definitely don’t intend to start a debate, and I don’t really disagree with anything Bob said, I just want to share some context that might clarify some things.

This is also something that gives me pause, and why we haven’t finalized the decision yet. The first step is v1.0 on GitHub (coming soon!) and then we’ll see about the CRAN decision.

There are tradeoffs here. In general if lots of users and other packages depend on a package then it’s harder to retire that package, but that means lots of people are using the package, which is great! There are plenty of CRAN packages that do get retired, or the developers keep them on CRAN but announce that there will be no further development. If we’re ever in the situation where we want users to transition to a different interface, there’s nothing preventing us from stopping new cmdstanr development work and saying that going forward access to new features will require some other interface (and you can stick with the older interface if you’re fine with no new developments).

And if there ever is a replacement for CmdStan, one option would be to just swap out calling CmdStan behind the scenes with calling the new interface behind the scenes. There just isn’t a replacement for it right now.

Anyway, with cmdstanr this is all much easier than for RStan because no tests that actually run CmdStan can be run on CRAN (in cmdstanr or packages that depend on cmdstanr). That means that from CRAN’s perspective changes to CmdStan (or a CmdStan replacement that we swap in behind the scenes in cmdstanr) can’t break anything CRAN can test.

I also wonder if sometimes in Stan development discussions there’s a misunderstanding about CRAN’s policies, which are not as restrictive as they may seem (see below).

This is true, but it really depends on what you mean by “replace RStan”. Hypothetically, we could announce that there will be no new versions of Stan added to RStan ever again and all future Stan versions will be released through a new interface. Users are free to stay with the frozen RStan or change. There would be no more maintenance burden on RStan related to updating hundreds of reverse dependencies because Stan syntax changed or any other change. RStan would just stay as it is now forever, except maybe occasional minor fixes as needed (e.g. if a new Rcpp requires a small change to RStan), but nothing close to the burden now of trying to get hundreds of dependencies to update for a Stan release. I’m not saying we should do that necessarily, but it seems doable.

Of course then the burden shifts to a new interface, but it will be a better designed interface and also that’s just the deal when you have lots of people who want to write R packages that depend on your software. And it’s great for Stan that there are lots of those people!

Another option is rip out the guts of RStan and replace them what @andrjohns is making or something else, call it RStan v3 and then accept possibly breaking other packages as part of a major version bump.

CRAN’s policy is that you can make breaking changes, you just need to give other maintainers enough advance notice of the breaking change so they can update their own packages. With as many reverse dependencies as we have it’s also customary to coordinate with CRAN. We’ve been overly cautious about this because we prefer not to break packages, so @andrjohns and others have worked hard to try to make the required fixes in those other packages. But CRAN does not have a no breaking changes allowed policy. In fact the recent release of loo broke something in another package, we notified the maintainer well in advance and even provided the fix in a PR, but they didn’t merge our PR in time. CRAN still allowed the release of loo because we had followed protocol. With RStan it’s more involved because there are many packages, but if we give CRAN and package maintainers enough advance notice then it’s doable. Packages with way more reverse dependencies than RStan have successfully gone through major updates that affect lots of packages.

Anyway, my point is not that we should necessarily do any of these things, just that there are actually quite a few options if we ever want to go down that road. And I can think of several other options beyond the ones I’ve mentioned.

Yeah, in general I agree it makes sense to work on new speculative algorithms outside of Stan. Then we can put the ones that pan out into Stan if we have developers interested in doing the implementation.

Relatedly, it seems like there are some clear improvements to adaptation that have widespread support and are just waiting for someone to implement them. So in that specific case I don’t think it would be too onerous to get it through the process.

7

How long could a package like cmdstanr survive on CRAN without our touching it? Sounds like indefinitely if no tests would break.

I thought they have a very minimal size restriction, which is why we couldn’t just vendor all the dependencies like Boost and Eigen and Stan headers and had to spread Stan out over several packages, some of which we don’t control and have to go through the baroque Rcpp interfaces.

I also thought they do not allow enforcing old version requirements. Let’s say Stan (a), Stan headers (b), Boost (c), and Eigen (d) packages are all working together rand I want to make a change to Stan (a + 1) and Stan headers (b + 1) that requires a new Boost version (c + 1). I thought this was challenging because version a + 1 of Stan would have to work with the {b, b + 1} x {c , c + 1} x { d }. And the new version of Stan headers would have to work with {a, a + 1} x {c , c + 1} x d, and so on. My understanding was that CRAN was helping us through that process manually, but have stopped, and this is why it’s now a challenge to update RStan.

Please correct the above if I misunderstood (this is very hard to write about!).

I mean write something new that does everything RStan does so that we can retire RStan and remove it from CRAN. Otherwise, we have two Stan packages on CRAN that people will be asking us about, filing bugs against, etc. I suppose we can just ignore all the questions and send a form letter politely saying “no.”

Will the better design make it easier to update all the dependent packages? If so, I think this should be the main design goal.

This would be very easy if CRAN allowed all of those packages to depend on version a of Stan and then allowed Stan to update to a + 1. Then all those packages could decide if they wanted to do the work to move to be compatible with Stan a + 1 and it wouldn’t be a burden on the Stan developers. But again, if I’m misunderstanding CRAN policy, please correct me. I’ve never dealt with CRAN myself.

Clear enough, I think. It’s going to hurt some models and help many more models. I think it’d be worth making the change. I just don’t want to have to touch any of our convoluted MCMC and the services layer in Stan, or anything in CmdStan code to do it.

the Stan language and math back end are very modular and easy to access externally through foreign function interfaces with BridgeStan. I recently built a whole new sampler that outperform’s Stan with the only external dependency being Eigen (flatironinstitute/walnutpie repo) and Brian wrapped it up in a BridgeStan-style ctypes wrapper in Python. And it works for models outside of Stan, which seems like a much better design than just building something to only work with Stan. I licensed it so that the Stan team could port it into Stan as is. The main advantage there is getting the Stan “stamp of approval” and having a deeper bench committed to maintaining it.

8

When I said CRAN’s policies weren’t as restrictive as they seem I was mainly referring to the policies on breaking other packages. You’re right that they are restrictive in other ways like you mentioned, including package size and depending on minimum versions of other packages rather than pinning specific versions (i.e. pkg >= version and not pkg == version). These wouldn’t really affect CmdStanR (since it doesn’t include CmdStan itself), but they do affect RStan.

Indefinitely if tests don’t break. Maybe an occasional minor tweak if something in R itself or a dependency changes, but we’ve made it a priority to only depend on very stable and widely used packages.

Yeah we’d just politely say “no” and announce that the old package still exists but is no longer under active development. There might have to be very occasional fixes if something in R changes, which isn’t ideal, but we wouldn’t have to update any packages that depend on it or implement any new features or fix bugs, etc.

I think maybe yes, but I’m not entirely sure yet. I agree it should be a goal.

Another benefit of putting things in Stan is that they can be made available through the same interfaces people are used to, as opposed to using a different interface for each sampling algorithm. That’s less of an issue the more people outsource coding to LLMs though. Regardless, walnutpie is very cool, and I’ve enjoyed playing around with it!

9

It does have the benefit of not forcing users to install another package. This will probably help deal with silly corporate policies that approve a package in general but not by version or that insist on things being on CRAN.

Our plan for Walnutpie in R is to write a parallel interface to the Python one. We couldn’t add it to Stan until posterior updates to deal with ragged chains (or you’d be forced to not use the auto stopping). But right now, our plans are tentatively scrapped now that we realize the Walnuts part of Walnutpie is unlikely to ever be useful (this is really bumming me out, but the spectral analysis shows the maximum eigenvalues of the negative Hessian of the log density having very little variation in posteriordb posteriors, so it’s not clear where Walnuts would help). I’m still hoping there’s a bug in my implementations somewhere.

The configurable tuning parameters for Walnutpie or Nutpie are different than those for Stan. So I wouldn’t recommend using the exact same command to sample because it would add 10 or more Nutpie-specific arguments. To make matters worse, each one will have to be documented as to which sampler it goes with and checks put in place to make sure inputs are consistent in not specifying tuning parameters for two different samplers.

I don’t think it’d be much harder to figure out how to use Walnutpie’s sample function than to learn a new variant of an existing sample function or a new sample function in cmdstanr or RStan. These interfaces are very lightweight. Then again, I’ve been very wrong about these things before. What’s easy for me isn’t necessarily easy for our users.