October 2.25 release?

Developers
62

@wds15 I did more benching.

for i in {1..100}
do
    ./blrm241 sample num_warmup=500 num_samples=500 data file=blrm2.data.R output file=241.$i.csv
    ./blrm24 sample num_warmup=500 num_samples=500 data file=blrm2.data.R output file=24.$i.csv
    ./blrm33 sample num_warmup=500 num_samples=500 data file=blrm2.data.R output file=33.$i.csv
    ./blrm34 sample num_warmup=500 num_samples=500 data file=blrm2.data.R output file=34.$i.csv
done

33 is 2.25 with Math 3.3, 34 is 2.25 with Math 3.4, 24 is 2.24.0, and 241 is Cmdstan commit #9396d19 (v2.24.1 and v2.24 are pointing at the same commit for some reason so I picked the first working build after Aug22 for my v2.24.1 (Cmdstan 2.24.1 is released - #9 by rok_cesnovar)).

Results were:

[1] "241"
20% 80% 
200 220 
[1] "24"
20% 80% 
198 219 
[1] "33"
20% 80% 
198 216 
[1] "34"
20% 80% 
201 222

I don’t even know how to interpret this now. The timings I gave earlier of 33 and 34 were the same binaries and the same computer and those were substantially faster than these.

library(rstan)

for(version in c("241", "24", "33", "34")) {
  timings = c()
  for(i in 1:100) {
    fit = read_stan_csv(paste0(version, ".", i, ".csv"))
    timings = c(timings, get_elapsed_time(fit)[1, "sample"] / sum(get_num_leapfrog_per_iteration(fit)))
  }
  
  print(version)
  print(round(1000000 * quantile(timings, c(0.20, 0.80)), 0))
}

Then maybe this is a platform thing?

I don’t know what it is given I’m not reproducing the benches I got from yesterday. I checked the md5sums and I’m not running the same binary at least lol (edit: I mean the same binary across these four tests).