How does underreporting of negative and inconclusive results affect the false positive rate in meta-analysis? A simulation study. R code
The aim of this project was to perform simulations investigating the impact of a higher publishing probability for statistically significant positive outcomes on the validity of meta-analysis. The type I error rate for the test of the mean effect size (i.e., the rate at which meta-analyses showed that the mean effect differed from 0 when it in fact equaled 0) was estimated. Additionally, the power and type I error rate of publication bias detection methods based on the funnel plot were estimated. The simulations show that a higher probability of including statistically significant positive outcomes causes a severe increase of the false positive rate in meta-analysis. Moreover, a one-sided selection process based on the statistical significance of a sufficient magnitude to dramatically bias meta-analysis conclusions is poorly detectable by publication bias methods based on the funnel plot when the mean effect size equals 0. An annotated R program which was used to perform the simulations is available here.