Estimating statistical power, posterior probability and publication bias of psychological research using the observed replication rate

2017-11-27T08:44:59Z (GMT) by Michael Ingre Gustav Nilsonne
In the present paper, we show how Bayes’ theorem can be used to better understand the implications of the 36% reproducibility rate of published psychological findings that was reported by the OSC (Open Science Collaboration, 2015). We demonstrate a method to assess publication bias, and show that the observed reproducibility rate was not consistent with an unbiased literature. We estimate a plausible range for the prior probability of this research, suggesting that statistical power in the original studies was 42%—73% and producing (positive) findings that were likely to be true 43% to 69% of the time. Publication bias was large, assuming a literature with 90% positive findings, indicating that negative evidence was observed 60—100 times before one negative result was published. These findings imply that even when studied associations are truly NULL, we expect the literature to be dominated by statistically significant findings.