Model recovery analysis in the noise-amplitude grid space.

The noise-amplitude grid was defined so as to cover the range of observed noise levels and signal amplitude, which was slightly different between the two studies. (A) The confusion matrix is perfect at the highest SNR levels (high amplitude and low noise, bottom left) but gets increasingly degraded at lower SNR (low amplitude or high noise, upper right corner of the grid) towards biased model selection. (C) The effect of poor SNR can be quantified using the matrix distance between the confusion matrix and the optimal, identity matrix (left) or the uniform matrix (right). Matrices that are far from the identity matrix are those with low diagonal elements, corresponding to poor recovery of the true model in favor of confounding competitors, but not necessarily random ones. Matrices that are far from the uniform matrix correspond to deterministic identification, i.e. the same model tends to win systematically under a given generative model, but not necessarily the correct one. Light lines indicate isocontours of SNR values. (C) Matrix distances as a function of SNR. At low SNR, the high distance from both the identity and the uniform matrix indicate that confusion matrices become biased rather than random.