Hangouts Link: https://meet.google.com/gzm-wmum-pfm
Instructions: Ask to attend in the hangouts interface and someone should let you in in the first 10 minutes of the meeting. Email mitzi212@gmail.com
Please add your agenda items in replies.
1 Like
Hi!
While I won’t be able to make it myself, Ben, Steve or Hamada will be at the meeting and we would like to bring up that we discussed to make a public call on the forum for asking R package developers to update their packages such that it will be easier for us as rstan devs to finally update rstan on CRAN. Here is the DRAFT message which we would like to prominently post. This is up for discussion and suggestions to change the message are welcome (sorry for the length, but I hope it’s well structured… and this is a big story unfortunately). The message is below and we wanted to share this before it goes out and hear about any concerns should there be any. Improvements to the draft are always welcome as well, of course.
Here it goes:
Title: R Package Maintainers Using Stan - Please Read
We are really thrilled to have created an explosion of R packages which are using rstan in order to make Stan more easily usable to everyone. Right now we count 84 reverse dependencies on CRAN for rstan and 140 for rstan/StanHeaders when including Bioconductor, which is great. At the moment, rstan is still in version 2.21.1 on CRAN dated from January 2020. This is obviously way outdated in terms of features supported as compared to the current 2.27.0 version (no reduce_sum parallelisation, no variadic ODEs… the list is long). There are many reasons as to why it is so difficult and I will explain below. For now we really urge all R package maintainers who use rstan to do two things if you have not done them already:
- Please use the rstantools package to manage your Stan models. This package will take care of many things for you behind the scenes in order to keep things working magically. Please refer to the vignettes on CRAN for an introduction on how to switch an existing package over. My personal recommendation is to use the command
rstantools::rstan_create_package(“rstanTemplate”, rstudio=FALSE)to create a template for your package and then try once therstantools::use_rstan()command to convert your existing package (and you should really make sure to have yourNAMESPACEfile be ok, see at the very end of this post). - Please put the RcppParallel R package into your Imports and LinkingTo fields of your package. This is required by now as Stan uses the Intel TBB under any circumstances by now. Also ensure that during building things a call is made to
RcppParallel::RcppParallelLibs()accordingly. This is handled by rtsantools if given full control (as recommended). Should your package require more control over Makvars and the like, here is a patch to OpenMX showing steps needed (Link to RcppParallel for stan math · OpenMx/OpenMx@8e0a967 · GitHub)
The above two points will make it a lot easier for us to push new versions of Stan to CRAN. I would herby encourage every R package developer using Stan do the above two things. I would also call out for help for volunteers to support others in need to port over their package should someone need help. Of course, we Stan devs will help as well, but the entire process would be much faster as a community effort obviously.
Hamada was so kind to code up a StanChecks facility, which tests R packages for compliance with all our needed requirements which we need to allow for an update of Stan (more details below). The list is accessible here and will be updated every now and then: rstan/revdep at StanHeaders_2.26 · stan-dev/rstan · GitHub
Background
The fundamental problem we have when updating Stan on CRAN is that we are effectively forced into keeping things compatible on the C++ level between Stan releases due to how things are right now. However, the Stan project only keeps compatibility on the Stan modelling language level - and not on the C++ side of things. A critical decision, which we Stan R developers somewhat regret by now, is that we have split Stan into StanHeaders (BSD licenced stan-math) and rstan (GPLed Stan services, Stan parser). Whenever now an upgrade is to be done, we have to upload one package at a time to CRAN. Hence, we first upload StanHeaders to CRAN. CRAN will then test with a mixed setup of these packages. At the moment we would then have StanHeaders 2.26 and rstan 2.21 on CRAN. Since CRAN will check if all reverse-dependencies (your R package) will still build and compile just fine in this setup. However, that is problematic as then we mix things in a way which the Stan project never intended to support. Your package will still contain C++ code generated under the old parser 2.21 and this version will be combined with a new Stan-Math library part of 2.26 StanHeaders. Before CRAN introduced their fully-automatic build pipeline it was possible to ask for an exception from the usual test one at a time package rule and upload StanHeaders together with rstan. Doing so allowed to solve a few problematic instances of Stan updates, but now this is not anymore an option.
The need to link to RcppParallel is required my now as the Intel TBB is a must have for Stan programs. The use of rstantools gives us Stan developers a lot more control over the entire build process. For example, the rstantools package will upon R package compilation always recreate the C++ sources using the Stan model as basis. Doing so allows us to ensure the latest parser on the system is used (avoiding that a super outdated parser is used for getting the C++ sources) and also allowing us to inject some tricks which are sometimes needed to get things to work.
The quoted reverse dependency check created from Hamada does on key thing for testing the “upgradability” of Stan: We will run R CMD check under a mixed setup. That is, one needs to install StanHeaders 2.21 and rstan 2.21 first. Then one needs to install StanHeaders 2.26, done via
install.packages("StanHeaders", repos = c("https://mc-stan.org/r-packages/", getOption("repos”)))
, and then run in this mixed setup (StanHeaders 2.26 & rstan 2.21) the R CMD check command for your package.
I hope the above is clear enough in explaining matters sufficiently detailed. We really hope to release Stan 2.26 ASAP, but we do need your help.
Best,
Ben, Hamada, Steve, Sebastian & the entire Stan team
The NAMESPACE file of your package whenever you upgrade a package to the new rstantools logic must include these statements:
import(Rcpp)
import(methods)
importFrom(rstan, sampling)
useDynLib(your-great-package, .registration = TRUE)
Otherwise generating the roxygen2 documentation won’t work.
6 Likes
to summarize discussion in today’s meeting:
@wds15 proposal was discussed and there was some consensus that this is preferable to alternatives. the question remains how best to document this - should this be in the Stan math doxygen pages? somewhere else in the Stan math repo - wiki? homepage?
items for next week’s agenda:
-
making good on our promise to deprecate language features, e.g. assignment via
<-. -
slack / zulip / forums /wikis - especially w/r/t developers who want to contribute code to the core Stan libraries - how to find answers to C++ related questions
-
how to insure that the official Stan docs get updated as new features are added to Stan? also, how to insure that contributions to the docs render properly in both HTML and PDFs.
4 Likes
It’s out there now… maybe a forum super user can pin the message for now to stick around at the top?
We should probably dump these instructions onto the rstan wiki, for example?
Done.
I think the intention was to pin this thread, not the current one
2 Likes