Now we have, finally, bumped posteriordb to the next beta version 0.2. Since 0.1, the main focus has been to include all the very good comments we got last time. A good presentation of the new posteriordb by @avehtari can be found here:
One of the main focus has been to simplify for the Stan community to contribute new stan posteriors. Here is some information on how to add a posterior using R:
Besides, we would be very happy for any suggestions, comments, and thoughts on how this could be improved to be of use to the community. The next two steps we are currently discussing are:
Adding more probabilistic programming frameworks (PyMC3, Tensorflow Probability, and Pyro) with tests that the models are identical
Nice! One thing that might also be nice for this. For some of the models that go off in terms of 10ths of a second their timing stability can be unreliable. For example the garch model in our performance benchmarks happens in less than a quarter of a second and can be +/- 20% for any individual run. It could be nice to do something like google benchmark does where they use a little heuristic to deduce how many times to run a benchmark in order to get a stable time estimate. Even for things like the SIR model that take a few minutes to run you probably want to run it 30 times or so.
I agree, for timing purposes many models needs to be run multiple times. The question is to decide which ones and for how long. I could run it on my machine and get a rough idea of the speed. Although that is probably not a good long term solution if people start to supply posteriors. Any thoughts?
Oh woof, I’m not sure! The code I posted above has some of what google uses. I think you essentially want a hard limit for each model’s max time and number of iters it can run (say maybe like 15 minutes and like 30K iterations)? There’s def way more complicated things you can do but I think some simple’ish heuristic would work.
