Suggestions to justify use of stan/bayesian MLM (to journal editors/reviewers)

As usual there’s a lot going on with an editorial reply, and though this isn’t my field at all I’d guess that there’s a good chance that politics and gatekeeping are playing a disproportional role in the final assessment and decision – i.e. if you used an objectively inferior model that is common for that kind of research nobody would question it, but if you try to use a more sophisticated one you would have to justify it, and even if it would be done weakly it’s still better than no justification. (The exception is if the paper is actually about the method, in which case a weak justification wouldn’t be acceptable, but the inferior model would not be a paper at all.)

Part of the problem is what you are up against, and from the statement above it is probably researchers who are experts in their fields but who lack the expertise to assess statistical methods but feel entitled to do so anyway. There is no such thing as “The Bayesian Premise”, the only technical different between an MLE and a MAP estimate is that the former must assume flat priors (see for instance a recent discussion here). Parameters do have distributions, you just have to compute them, that is true of frequentist approaches as well, that is how confidence intervals are established, and nobody rejects the use of CIs on the basis of it requiring accepting the bayesian premise.

On the other hand, it is true that you could put some figures into the performance of multilevel models vs something else, and justify other aspects in a more objective way, but again the truth is you shouldn’t have to justify the choice of bayesian vs frequentist inference any more than you have to justify the model itself (up to the priors, but if anyone insists on the discussion that “priors are subjective” were kind of back in the 80s).

All of that said, my personal opinion is that you shouldn’t promote your method as “Bayesian” or the use of Stan in a way that requires explicit justification, but rather describe well and justify the model itself regardless of the inference method, because in principle you could do it within either a bayesian or frequentist formulation (again, up to the priors) and using any package. By downplaying the Bayesian aspect of your analysis and deemphasizing the specific software I think you may fly below the radar of nonexpert nitpicking, but by still stating that the analysis is indeed bayesian and that you are using Stan anyone who has some expertise in statistics will know that you are using a well-justified approach with a state-of-the-art inference implementation, so you’d get the best of both worlds. I’d leave the Bayesian-Frequentist wars for other settings where it won’t happen behind closed doors with an arbitrary power asymmetry and affect publication decisions.

But that’s just like, my opinion, you or others may want to take this on in every possible battefield.

3 Likes