Stan 2.20.0 to be released July 18th 2019

Developers
1

Hey all,

Just a friendly reminder that we’re switching to a quarterly release system. The last release (2.19.1) was on April 18th, so we’ll be releasing 2.20.0 on July 18th. This release likely won’t include the new compiler but I project that the one after that (which will be October 18th 2019) will have us switching over completely.

The way we’re going to run these releases is that the latest commit to develop in each repo that has passed completely through Jenkins testing by 9am GMT will be the commit we use for the release. Just as a heads up, this means that if one were to merge something to master in Math at 5am GMT July 18, it probably won’t make it in to 2.20.0 because the Jenkins jobs that test and percolate the commits upstream take more than 4 hours.

Thanks!
Sean

9 Likes
2

Sounds great… we are just right now stuck on Windows problems with our Jenkins. Any idea when that will settle? I am out of ideas as to what the problem there is/could be.

3

Try again - I turned off the EC2 windows machines until they’re fixed and tested.

4

That’s unfortunate timing for compilation speedup, which our users almost certainly won’t get until November given how backlogged our review process is.

5

I was under the impression it was basically done already - what’s left to do?

6

My guess from past performance on issues this large is that we’ll miss the 18 July deadline by a week or two based on review and discussion time for issues this large that are coupled across repos.

For past releases, Jenkins has also been a bottleneck.

The PR for cmdstan still needs makefile work, so I’m not sure when that’ll even be ready to submit. Maybe today.

I’m OK with feature delays.

7

I’ll make sure Jenkins isn’t an issue this time - we should be in a much better place than we were historically with the autoscaling EC2 linux instances, though we are doing much more Windows testing than we used to and the autoscaling EC2 Windows nodes have been super troublesome and aren’t working yet (of course), so we just have the one physical Windows machine. Fingers crossed!

8

In case it wasn’t clear, I think regularly quarterly releases are a good idea. Even if some of the release times turn out to be suboptimal.

The overhead of deciding what got in/out was very high as was the time for waiting for one more thing before release.

4 Likes
9

Any idea about the availability of these shiny new Stan(s) though an R interface ? rstan is still at 18.2, 19.x is unheard of…

5 Likes
10

2.19 is in process at CRAN. After Ben described the process, Andrew drew an analogy to Sisyphus.

Yes. It’s on the roadmap, but it won’t be started, much less released, until someone volunteers to design and build it.

The Python version’s already feature complete for default NUTS and should be released in a full-featured form soon. The plan may be to just copy that interface into R to simplify the development process.

2 Likes
11

Here https://github.com/stan-dev/rstan/tree/develop/StanHeaders you find a file install-github.R, which explains how use the latest versions of stan and math on GitHub when installing StanHeaders from source. (In the last step you might also have to specify 'type=“source”).

2 Likes
12

I sort-of got an answer : rstan 2.19 is available on CRAN as of this morning…

I don’t get (yet) what needs to be done on rstan to “talk to” the forthcoming Stan 2.20. I’ll try to find Ben’s description.

Thanks for your attention !

1 Like
13

Thanks Emmanuel for checking but rstan 2.19 still ships with StanHeaders 2.18.1 not sure if we have access to all Stan 2.19 features.
Thanks

14

Thanks Emmanuel for checking but rstan 2.19 still ships with StanHeaders 2.18.1 not sure if we have access to all Stan 2.19 features.

Indeed :

> sessionInfo()
R version 3.6.0 (2019-04-26)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 10 (buster)

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3.10.3
LAPACK: /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3.10.3

locale:
 [1] LC_CTYPE=fr_FR.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=fr_FR.UTF-8        LC_COLLATE=fr_FR.UTF-8    
 [5] LC_MONETARY=fr_FR.UTF-8    LC_MESSAGES=fr_FR.UTF-8   
 [7] LC_PAPER=fr_FR.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=fr_FR.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] StanHeaders_2.18.1-10

loaded via a namespace (and not attached):
[1] compiler_3.6.0 tools_3.6.0   
> library(rstan)
Le chargement a nécessité le package : ggplot2
rstan (Version 2.19.2, GitRev: 2e1f913d3ca3)
For execution on a local, multicore CPU with excess RAM we recommend calling
options(mc.cores = parallel::detectCores()).
To avoid recompilation of unchanged Stan programs, we recommend calling
rstan_options(auto_write = TRUE)
> sessionInfo()
R version 3.6.0 (2019-04-26)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 10 (buster)

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3.10.3
LAPACK: /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3.10.3

locale:
 [1] LC_CTYPE=fr_FR.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=fr_FR.UTF-8        LC_COLLATE=fr_FR.UTF-8    
 [5] LC_MONETARY=fr_FR.UTF-8    LC_MESSAGES=fr_FR.UTF-8   
 [7] LC_PAPER=fr_FR.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=fr_FR.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rstan_2.19.2          ggplot2_3.2.0         StanHeaders_2.18.1-10

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.1         magrittr_1.5       tidyselect_0.2.5   munsell_0.5.0     
 [5] colorspace_1.4-1   R6_2.4.0           rlang_0.4.0        dplyr_0.8.3       
 [9] tools_3.6.0        parallel_3.6.0     pkgbuild_1.0.3     grid_3.6.0        
[13] gtable_0.3.0       loo_2.1.0          cli_1.1.0          withr_2.1.2       
[17] matrixStats_0.54.0 lazyeval_0.2.2     assertthat_0.2.1   tibble_2.1.3      
[21] crayon_1.3.4       processx_3.4.0     gridExtra_2.3      purrr_0.3.2       
[25] callr_3.3.0        ps_1.3.0           inline_0.3.15      glue_1.3.1        
[29] compiler_3.6.0     pillar_1.4.2       prettyunits_1.0.2  scales_1.0.0      
[33] stats4_3.6.0       pkgconfig_2.0.2

Suggestions ?

15

rstan and stanheaders typically have to be released in a staggered manner, which is why the first working version of RStan is not X.XX.0 but rather X.XX.1. I believe that this process has not yet been completed as there has not been an official announcement of a 2.19 RStan release.

16

The RStan 2.19 source is on CRAN, but I typically announce when binaries are available. Although StanHeaders is versioned 2.18.1-10, it has Stan Math 2.19 and the Stan language for 2.19. However, stanc is a mix of 2.18 and 2.19 to make it compatible with both RStan 2.18.x and 2.19.x.

2 Likes
17

Nothing’s going to break. But if RStan wants to take advantage of faster compile times, it’s going to need to decouple compilation of the services and the models into separate compilation units the way it’s being proposed for CmdStan 2.20.

Warning: CmdStan 2.20 doesn’t exist yet. My changes for improved compilation speed are still being reviewed.

18

I find the divergence of versioning numbers between Stan Math and Stan Language on the one side and StanHeaders on the other side confusing.

Maybe one could add a NEWS page to the StanHeaders page on cran, which describes from which current stan/math and stan/stan versions StanHeaders is built and add a link to the releases page for the repositories in github?

2 Likes
19

It is confusing but the basic rule is to use the current versions of whatever is on CRAN.

20

With confusing I don’t mean that one can’t deal with it, but that complicates things seemingly unnecessary.
A problem would for example arise if someone thought a bug is fixed in StanHeaders 2.19, because the bug was fixed in Stan/math 2.19.
For example, this means that the bug fix for the vectorized version of the ordered logistics lpmf will be later available in StanHeaders than the naive reader (who doesn’t know that versioning numbers of StanHeaders and Stan/math are not consistent) would think.

1 Like