Would anyone object to me putting together and posting (on stan-users I guess?) a set of recommendations for Stan users who are developing their own packages based on RStan? We’ve been seeing more and more of these lately and, while I don’t think it’s our job to be tracking them all down and monitoring their development on github (not that we have the time anyway), and while we certainly can’t fully prevent anyone from doing anything stupid in their package, I do think it’s important that we make some recommendations. And better to do it now while there still aren’t so so many of these packages.
Anyway, unless there are objections for some reason, I’ll go ahead and put something simple together.
Absolutely no objection. Please do it. If you need a tag, create one. Do what makes it easy. Please post on discourse too. Eventually the users will move too.
At some point we may need our own blog. I’ve been writing something about Stan that will end up on my blog cause I don’t think it’s right for Andrew’s blog.
Sounds like a good idea to me. A lot of developers need
more of a leg up on these things. I think someone like me
might be rash enough to try to develop an R package on
top of RStan if you made the instructions easy to follow :-)
Should we get a domain like stan-blog.org and put it on WordPress?
It sucks as a platform, but I know how to use it and it supports
LaTeX markdown (though not well in comments given lack of preview).
Jekyll through GitHub would be cooler, but I have no idea how to configure
that or setup logins.
Here is a draft for the R package guidelines that I just wrote up. Please let me know if you have anything to add or think any of it should be removed or altered.
Dear Stan users,
One of the coolest things about working on a project like Stan has been seeing some of our users begin to develop tools for making Stan more accessible to audiences that may otherwise not benefit from what Stan offers. In particular, recently we have started seeing a growing number of R packages that provide high-level interfaces to Stan, using the rstan package for estimating models without requiring that the user be familiar with the Stan modeling language itself.
This is a great development and we would like to support such efforts going forward, but to-date we have made little effort to coordinate the development of these packages. To avoid a Wild West, so to speak, of Stan-based R packages, we think it is important that developers make every effort to adhere to certain guidelines in order to ensure these packages are of the highest possible quality and provide the best possible experience for users. To that end, in this post we present a set of recommendations for the development of R packages that interface with Stan. These recommendations are based on software design principles we value as well as many things we are learning as we continue developing our rstanarm package and review packages being developed by others.
Of course anyone is free to develop a package based on Stan, even if they ignore these recommendations. However, both because we feel strongly about these recommendations and because our time is unfortunately limited, we will loosely enforce these guidelines by (1) prioritizing giving feedback to developers who follow them over those who choose not to, and (2) refraining from promoting packages that ignore them. That said, we are open to some limited exceptions (e.g., the brms package is a sanctioned exception to one of guidelines about Stan code). If you think your package should be an exception definitely let us know.
Guidelines for R packages providing interfaces to Stan
General package structure and development
The rstan package provides the rstan.package.skeleton function, which you should use to create the basic structure of your package. If you have trouble setting up your package using rstan.package.skeleton please consult the rstanarm repository and use it as a template and/or ask us for help on the Stan users forum.
Use version control (e.g., git).
Unless you are developing proprietary private software, organize your code in a repository that is public on GitHub (or a similar service, but preferably GitHub). It should be public
even at early stages of development, not only when officially released.
Unit testing is essential. There are several R packages that make it relatively easy to write tests for your package. rstanarm uses the testthat package for this purpose, but if you prefer a different testing framework that’s fine. The covr package is useful for calculating the line coverage of your tests, and we recommend at least 90% coverage. Good line coverage does not guarantee high quality tests, but it’s a good first step.
Stan code
All Stan code for estimating models should be included in pre-written static .stan files that are compiled when the package is built (see the exec directory in the rstanarm repo for examples). You can also use the inst directory to include code chunks to be used in multiple .stan files (again see the rstanarm repo for examples). If you set of your package using rstan.package.skeleton this structure will be created for you.
This means that your package should NOT write a Stan program when the user calls a model fitting function in your package, but rather use only Stan programs you have written by hand in advance (if you are working on a model for which you don’t think this is possible please let us know). There are several reasons for this. One reason to use pre-compiled Stan programs is that models will run immediately when called, avoiding compilation time. Second, if you want us to give advice on your Stan code (which we are happy to do) it will be easier for us if you write your programs following this recommendation. And most importantly, although pre-written static Stan programs can still contain bugs, it is much easier to test programs that do not change depending on actions taken by the user. To provide flexibility to users, your Stan programs can include branching logic (conditional statements) so that even with a number of .stan files you can still allow for many different specifications to made by the user (see .stan files in rstanarm for examples).
Use best practices for Stan code. If the models you intend to implement are discussed in the Stan manual or on the Stan users forum then you should follow any recommendations that apply to your case. If you are unsure whether your Stan programs can be made more efficient or more numerically stable then please ask us on the Stan users forum. Especially ask us if you are unsure whether your Stan programs are indeed estimating the intended model.
Relatedly, prioritize safety over speed in your Stan code and sampler settings. For example, if you can write a program that runs faster but is potentially less stable, then at a minimum you should make the more stable version the default. This also means that, with rare exceptions, you should not change our recommended MCMC defaults (e.g. 4 chains, 1000+1000 iterations, NUTS not static HMC), unless you are setting the defaults to something more conservative. rstanarm even goes one step further, making the default value of the adapt_delta tuning parameter at least 0.95 for all models (rather than rstan’s default of 0.8) in order to reduce the step size and therefore also limit the potential for divergences. This means that rstanarm models may often run a bit slower than they need to if the user doesn’t change the defaults, but it also means that users face fewer situations in which they need to know how to change the defaults and what the implications of changing the defaults really are.
R code and documentation
Functions that provide useful post-estimation functionality should be given the same names as the corresponding functions in rstanarm (if applicable). For example, posterior_predict to draw from the posterior predictive distribution, posterior_interval for posterior uncertainty intervals, etc. To make this easier, in the next release of rstanarm, functions like these will be converted to S3 generics. rstanarm will provide methods for the stanreg objects it creates, and your package can import the generics from rstanarm and provide methods for the fitted model objects returned by your model-fitting functions. For some other functions (e.g. as.matrix) the generics are already available in base R or core R packages.
To be clear, we are not saying that the naming conventions used in rstanarm are necessarily optimal. (If you think that a function name used by rstanarm should be changed please let us know and suggest an alternative. If it is a substantial improvement we may consider renaming the function and deprecating the current version.) Rather, this guideline is intended to make function names consistent across Stan-based R packages, which will improve the user experience for those users who want to take advantage of a variety of these packages. It will be a mess if every R package using Stan has different names for the same
functionality.
We will soon be releasing a package to serve as a back-end for plotting for our R packages (rstan, shinystan, rstanarm) to reduce code duplication and provide a core set of plotting functions for MCMC diagnostics, posterior predictive checks, and other useful graphical displays after fitting Bayesian models (preview development at jgabry/bayesplot). We hope developers of Stan-based R packages will also use it. For any plot you intend to include in your package, if it is already available then we recommend using the available version or suggesting (or contributing) a better version. If it’s not available then we’re interested in adding it if it’s a plot of general use. We will post a follow-up soon when this is available, and we welcome your feedback and contributions to improve it.
Provide thorough and clear documentation. The documentation won’t be perfect
(we’ve found many holes in our rstanarm documentation already and there are
certainly more), but you should make every effort to provide clear and thorough
documentation, using decent grammar and actual sentences wherever possible. The
poor quality of the documentation in many (though certainly not all) R packages
is insulting to users and an impediment to high quality research. In no way is
the current state of R package documentation a reason to insufficiently document
your own package. It is unfortunate that this even needs to be said, but even a
cursory review of core and contributed R packages immediately reveals the
necessity for guidelines like this one. Some of our own documentation does not
always meet this high standard, but we are consistently making efforts to
improve this.
Recommended resources
Hadley Wickam’s book on R packages. If you are interested in developing an R package that interfaces with Stan but are not an experienced package developer, we recommend Hadley’s book, which is free to read online. Even if you are an experienced developer of R pacakges, Hadley’s book is still a good resource.
These guidelines are not set in stone. We expect them to evolve and we very much appreciate comments or suggestions for how they can be improved.
– The Stan Development Team
Ok, that's what I've got so far. Any comments/criticisms/suggestions? I realize I used somewhat strong language in the part about high quality documentation, but I feel strongly about it and I'm just so sick of the lousy documentation in R.
How about you mention the covr package to calculate test coverage and recommend a minimum 90% test coverage (along with your test paragraph)? High test coverage does not guarantee good tests, but it is a first step and covr is so simple to use.
Thanks Sebastian, yeah I’ll add a bit about line coverage for tests. Unfortunately getting Travis + covr to work consistently with rstanarm has been a bit of a pain. It seems like we need to reconfigure something weekly. But people can check test coverage with covr locally if there are Travis issues.
Ahh… did not know about the Travis problem. But out of my own experience I find covr extremely useful. I thought to have good test coverage for one of my packages, but it was indeed crappy. Covr was really amazing to spot this and upping the test coverage immediately revealed issues; so I think this is an essential tool for R package development. travis should make covr analysis possible at some point.
I’m working on some functions for calculating pdf’s in Stan and I have 2 requirements: 1) they should live in the functions block not be hard-coded into a Stan model; and 2) they should be available in the package namespace as R functions (so that I can run them through unit tests and also make them available in the package for plotting etc…).
So far my solution involves some hacky stuff with the inst subdirectory of the package (more or less like rstanarm) and the expose_stan_functions function but I imagine that’s going to get ugly. Any better suggestions?
I’m working on some functions for calculating pdf’s in Stan and I have 2 requirements: 1) they should live in the functions block not be hard-coded into a Stan model; and 2) they should be available in the package namespace as R functions (so that I can run them through unit tests and also make them available in the package for plotting etc…).
So far my solution involves some hacky stuff with the inst subdirectory of the package (more or less like rstanarm) and the expose_stan_functions function but I imagine that’s going to get ugly. Any better suggestions?
If you know what the functions are at package compile time,
then you could use Stan to generate the C++ for the functions, pull
the function into its own file (and namespace), then expose it
through Rcpp.
Or you could just write the functions in C++ and add them to
Stan. What are the pdfs?
So far it’s four generalized Gamma parameterizations. They were sort of confusing so I coded them up in an R package with unit tests showing they meet some of the requirements for a pdf/lpdf/cdf/lcdf, that the (l)pdf matches the (l)cdf (using R’s 1-d integrate function), and that they match up with two of the parameterizations from the R flexsurv package. The unit tests generate random parameter values for the tests and (for the tests that don’t integrate over the (l)pdf) run them across a range of input values.
I was hoping to make it easy to verify that when I translate the R code to Stan code I don’t introduce bugs now or in the future by re-using the unit-test functions I created. I was also going to have the related Stan survival models in the R package so I want to avoid manual steps in the build process.
Thanks, i think the link is just to the rstanarm package no tto the issue you meant to point to but I found the underlying Stan issue about code generation. I think you’re right that having code generation of all-doubles functions would make this trivial (via sourceRcpp or similar).
The code generation generates templated code. If you
need all doubles, I’d suggest just writing a function
with all double arguments that calls the templated
version with the templates instantiated.