Meta-analysis and treatment of standard errors as known

General
1

Hi everyone,

I’ve noticed that @bgoodri and some others are very concerned with issues that arise when standard deviation/standard error are treated as known (I know of at least one mention in regards to 8 schools and more recently in this thread.

I wonder if anyone can point to resources regarding what this might mean for meta-analysis, where essentially every application I have seen treats study means errors as known. Does anyone know why this is, and what the consequences (if any) may be?

2

Why? Most papers only report the point estimate and standard error and don’t make the individual data available, so often there is little alternative but to ignore the (considerable) uncertainty in those estimates when doing the meta analysis.

3

A more robust meta analysis would model the error as observations of some true error. This provides a principled approach to heuristics that are used in a few fields where the quoted errors are inflated until the experiments agree with each other.