I have a really small doubt regarding “Bayesian power analysis” and I would be grateful if you could help me. I have a sample with 105 participants, and a path model (tested using brms) with 8 variables. After submitting the paper, one of the reviewers asked the following:
The authors mention their sample size is small. It would be helpful to include a power analysis to allow for some estimation of the dependability of the statistical models and results.
So, I would like to know the following:
Is it common to present a power analysis for Bayesian models when sample sizes are around 100?
In the affirmative case, do you know how can I perform it using R?
Is there any rule about the number of variables in a Bayesian model depending on the sample size?
I think that if you have a good sense of where you expect your coefficients to land in terms of magnitude then it can be informative to do a design analysis as described by Gelman and Carlin, but the trick is that anything based on your observed estimates is pointless. If your reviewers are insisting that’s what you should do then you can reference above as well as refer to some of papers here.
Agree with Ara however that a standard power analysis doesn’t really make a lot of sense from a bayesian standpoint, although it can sometimes be nice to know the frequentist properties of your model. Krushcke describes a process based on simulating from priors here.
Do you know what the reviewers meant by “power analysis”? We don’t tend to do power analyses with Bayesian models based around statistical significance (or Bayes factors, for the same reason), but you can do it for other aspects of modeling.
You have to choose values for (hyper)parameters, then you can simulate multiple data sets, fit them, and measure things like variance of posterior estimands.
I don’t have any more specific recommendations, but I’m pinging @andrewgelman and @lauren, who have been working on power-type analyses for multilevel regression and poststratification around survey sampling and can hopefully point you in the right direction.
I think my response is pretty much the same as others here - simulate data with different effect sizes and look to see whether the model can detect them/suggests they are there when they are not there.
Hi–I recommend you take a look at chapter 20 of my book with Jennifer Hill, Data Analysis Using Regression and Multilevel/Hierarchical Models, as we have some discussions and examples of power analysis there. The computations in that chapter are not fully Bayesian, but we do have some simulations so that should give the general idea.