I think the number one priority should be getting the right answer. The problem is that how close you get to the right answer depends on number of iterations.
Changing number of iterations from 2000 to 200 would change the run time by an order of magnitude. Running in parallel on 8 cores rather than on a single core (Stan’s default) would be another order of magnitude. These matter.
In other words, I think knowing how long it takes to run the default settings is meaningless without a quantification of the accuracy of the answer.
People do this all the time and report that JAGS finishes the same number of iterations faster than Stan. What they forget to do is adjust for effective sample size (i.e., precision).
I don’t know how to solve this problem. Maybe we need to implement the timed runs that Andrew is advocating and run them until we get to an effective sample size of 20 by default.