Hi all!
Nice discussion and very good suggestions for further development.
@ahartikainen It should be great to set up a connection to get posteriors, data, and models directly from Github through Python. Hopefully, this can also enable quick access to a RESTful API further on. Unfortunately, connections to Github are not implemented for Python yet and I’m not sure when it will be. I’ll set it up as an issue, and it would be great with some help from the Python people here. I don’t think it is much more work that is needed.
@betanalpha @avehtari @paul.buerkner:
I agree with the expectations and that parameter expectations and variances could be computed using 100 000 draws and stored separately in addition to the smaller sample. If we get a “real” database up we could extend the number of samples to 100 000 further on.
Regarding what expectations to use, I think, as @paul.buerkner noted, this should probably be implemented more generally in the posterior package. Now the samples for posteriordb are returned as a posterior draws object to facilitate quick analysis of the draws. Hence implementing estimators should preferably be included in the posterior package, the idea would be available more generally in the Stan R ecosystem.
I also think that SBC should be included as a slot on each model and framework (Stan in this case) to test the specific model at hand. I put that on the todo as well as including more details regarding:
- R/Python, the makevars file, the systemInfo() (and equivalent for Python)
Is there anything more that should be included for reproducibility?
Regarding diagnostics. Currently, we only check divergencies, Rhat and ESS (bulk and tail). Besides we should add that E-FMI is higher than 0.3. Is there anything else that should be checked for references posterior draws?
The next step is also to try to find a way to compare two distributions based on samples and try to find some measure of the distance between distributions. I know @breckbaldwin would ideally want to have one measure that could be used. Any thoughts or pointers there @betalalpha? We have been discussing maximum mean discrepancy (MMD) loosely but have not yet got started with this: http://www.jmlr.org/papers/volume13/gretton12a/gretton12a.pdf
@andrewgelman Thank you for the link! I may get back to you with questions on this.