S1 Text -

Fig A. Neural correlates of visual and auditory word similarity. Visual word similarity, as modelled by the early layers of the an AlexNet DNN (here: layer 3) predicted activations in bilateral posterior visual cortex, including fusiform cortex (866 voxels, peak: -24/-97/-5, t[18] = 7.89). Auditory similarity of the spoken words was modelled by a speech recognition DNN. Early layers of the network (here: layer 2), because of the correlation of word length and speech duration, also predicted activations in bilateral posterior visual cortex (266 voxels, peak: 24/-76/-8, t[18] = 6.51). By contrast, the last layer of the network (layer 7) predicted activations in left middle temporal gyrus (34 voxels, peak: -60/-13/-8, t[18] = 5.61). Brain maps are thresholded at p_voxel<0.001 (uncorrected) and p_cluster<0.05 (FWE-corrected). Fig B. Individual-word effects. a) Brain-model correlations in left IPC when individual words were deleted from the model RDMs and neural RDMs before performing the analysis. The relatively homogeneous pattern shows that no single word exerted a substantial influence on the correspondence between model and brain. Error margins denote standard errors of the mean. b) Brain-model correlations in left IPC when removing a subset of up to 30 words at random from the RDMs. Results across 100 analyses with random subsets removed reveal that the pattern largely holds for smaller subsets of the stimulus space. All data points are means across all participants. Fig C. Model intercorrelations. Pairwise correlations between the representational dissimilarity matrices (RDMs) constructed for all word2vec model variants used in the study. Fig D. Controlling for word frequencies. a) Whole-brain searchlight analysis when controlling for similarities in word frequency using partial correlations. This analysis yielded results analogous to the original analysis (Fig 1C), with two clusters in right IPC (80 voxels, peak: 45/-46/58, t[18] = 4.91, and 35 voxels, peak: 36/-73/40, t[18] = 4.79) and one cluster in left IPC (163 voxels, peak: -36/-58/46, t[18] = 6.23). b) Region-of-interest analysis in IPC for differently sized training corpora when similarities in word frequencies were controlled for. This analysis revealed essentially identical results to the main analysis (Fig 1E). c) Relative frequencies of the 61 abstract words across the differently sized training corpora. Despite some variations across corpus size, the words that were frequent in large corpora were also more frequent in the small corpora. Words are sorted by frequency in the largest (45m) corpus. Fig E. Direct model comparisons in left IPC. Differences between in brain-model correlations between the full model (45m sentences and all words included) and the trimmed models. Asterisks indicate significant differences to the full model, FDR-corrected for multiple comparisons. The full model itself is omitted from the plot. Color conventions are the same as in Fig 1E. Error bars denote standard errors. Fig F. Whole-brain searchlight with TFCE statistics. Here, we used an alternative statistical test, based on threshold-free cluster enhancement (TFCE; Smith & Nichols, 2009), as implemented in CoSMoMVPA (Oosterhof et al., 2016). Z-scores for TFCE values were obtained by comparing the actual values to values across a null distribution constructed from 10,000 sign permutations. The resulting statistical maps were thresholded at z>1.96 (p<0.05). As in the main analysis (Fig 1C), two clusters emerged in right parietal cortex (503 voxels, peak: 42/-46/58, z = 2.58, and 88 voxels, peak: -6/-58/58, z = 2.11) and one cluster emerged in left parietal cortex (442 voxels, peak: -27/-73/46, z = 2.89). Although also centered on the IPC, TFCE yielded somewhat more liberal results, with clusters extending more into the superior parietal cortices. No other clusters emerged across the brain. Fig G. Searchlight results on cross-sectional images. Coronal brain slices are overlaid with regions that show significant correlations between neural representations and the full word2vec model (as in Fig 1C). Fig H. Unthresholded searchlight results. Coronal brain slices are overlaid with unthresholded t-maps comparing brain-model correlations to zero for the full word2vec model (as in Fig 1C). Slices are spaced continuously between z = -34 and z = 54. Negative t-values are truncated. Table A. Abstract words. The 61 abstract German words used in the experiment and their English translations. Table B. Background contexts. The 10 background contexts used in the experiment and their English translations. The gray text on the bottom was always used during the practice run.

(PDF)