I gave a student presentation yesterday that focused on what some of Andrew’s thoughts regarding forking paths, multiverse analysis, and type M/S errors mean for the work being done at our school and as a result have been thinking how I can do more to integrate these ideas into my own work.

The multiverse paper was written from a p-value focused mindset, and I’m wondering about how I can best adapt it to the estimation focus I prefer to work in from a decision analysis standpoint. I’m wondering if anyone here thinks it makes sense at all to combine the posteriors from a multiverse analysis in a similar way to how we would combine multiple datasets in multiple imputation (that is, concatenate the posteriors together)? My goal would be to contrast the results of a primary analysis against results that would arise from other credible data processing decisions. Normally in my field we approach this through sensitivity analysis and show each analysis separately, but the numbers here are obviously much larger.

Edit: Added the multiverse paper

http://www.stat.columbia.edu/~gelman/research/published/multiverse_published.pdf