This is our plan with posteriordb. We have been thinking whether the testing part would be part of posteriordb or separate package bayesbench, but no final decision has been made yet. posteriordb repo has instructions how to access list of posteriors, each model+data combination with golden standard estimates. @bbbales2 has used posteriordb to test warmup, but not yet in cluster. If you have suggestions for cluster aware scripts, please start a new thread, add to existing posteriordb thread Beta-release Bayesian Posterior Database, or make an issue in posteriordb repo.
Yes. posteriordb project is independent from warmup. We don’t have yet that many models, but we have a student working part time filling the database with models. It would be great if we get more people involved. One missing part is easy to configure (e.g. what information is collected) and use cluster-computing aware scripts. See also PR for use cases for posteriordb https://github.com/MansMeg/posteriordb/pull/120