animate3.gif (80.96 kB)
The Garden of Forking Paths
An illustration of how opportunities for false positives can mount up when one has a
large dataset and a flexible approach to analysis. Consider an investigator who
is interested in testing whether handedness is associated with attention
deficit hyperactivity disorder (ADHD), and has access to a large dataset that
has measures of both hand skill and hand preference at 6 years and 10 years, as
well as demographic information. Suppose further that there is no true
association in the population. The green node corresponds to the two-group
comparison, where probability of obtaining a p-value < .05 is 1 in 20. An investigator
who compares ADHD (A) and typical (T) children (green node) may be disappointed
to find the comparison is nonsignificant. But the investigator may be tempted
to do a subgroup analysis, because the association looks different in older (O)
and younger children (Y) (purple nodes). He might then realise that results vary
for measures of hand skill (S) vs hand preference (P) (blue nodes). He could
then decide to subdivide the sample by gender (M vs F) (orange nodes) and
according to whether children are from urban (U) rather than rural (R) areas (khaki
nodes). If all these combinations of possibilities were to be considered, the chance
of a finding at least one ‘significant’ result rises to 1 - .95^16 = .56 (assuming the different choices are independent). When
doing numerous comparisons, there are legitimate methods for adjusting p-values
to guard against false positives, but in practice these are often ignored. If the researcher presents what looks like a
simple two-group comparison (e.g. ADHD and Typical young, urban females differ
on a measure of relative hand skill), which is in reality selected from a wide
range of possible contrasts, then uncorrected p-values can be highly misleading.
The figure title is inspired by Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no 'fishing expedition' or 'p-hacking' and the research hypothesis was posited ahead of time. www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf.
Figure created by Tim Brock of DataToDisplay.com.
The figure title is inspired by Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no 'fishing expedition' or 'p-hacking' and the research hypothesis was posited ahead of time. www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf.
Figure created by Tim Brock of DataToDisplay.com.
