Modality switches occur early and extend late in conceptual processing: Evidence from ERPs (Bernabeu, Willems, & Louwerse, in prep.)

Files: poster, design overview, stimuli, EEG montage used, waveforms, difference topographies, critical statistics, fixed effects of final models, entire modeling, raw data. For best resolution, please download image(s). Also see waveforms interactively within sections of the data, and finally the entire data set.

Abstract The engagement of sensory brain regions during word recognition is widely documented, yet its precise relevance is less clear. It would constitute perceptual simulation only if it has a functional role in conceptual processing. We investigated this in an Event-Related Potential (ERP) experiment implementing the conceptual modality switch paradigm. In each trial, participants verified the relation between a property word and a concept word. Orthogonally, we manipulated the conceptual modality of successive trials, and tested whether switching modalities incurred any processing costs at different stages of word recognition. Unlike previous studies, we time-locked ERPs to the first word of target trials, in order to measure the modality transitions from the beginning, and also to reduce confounds. Further, we included different types of switch—one from auditory to visual modality, and one from haptic to visual—, which were compared to the non-switch—visual to visual. Also, one group of participants was asked to respond quickly (n = 21), and another group to respond self-paced (n = 21), whilst a few others received no constraints (n = 5, n ERPs = 4). We found ERP effects in four typical time windows from 160 to 750 ms post word onset. The overall effect is characterized by a negativity for modality-switching relative to not switching, and it increases over time. Further, the effect arises with both types of switch, and influences both participant groups within anterior and posterior brain regions. The emergence of this effect in the first time window particularly suggests that sensory regions may have a functional role in conceptual processing. The increased effect later on converges with previous studies, and should be further investigated in order to determine the extent of semantic processing therein.

References

Collins, J., Pecher, D., Zeelenberg, R., & Coulson, S. (2011). Modality switching in a property verification task: an ERP study of what happens when candles flicker after high heels click. Frontiers in Psychology, 2.

Hald, L. A., Marshall, J.-A., Janssen, D. P., & Garnham, A. (2011). Switching modalities in a sentence verification task: ERP evidence for embodied language processing. Frontiers in Psychology, 2.

Hauk, O. (2016). Only time will tell—Why temporal information is essential for our neuroscientific understanding of semantics. Psychonomic Bulletin & Review, 23.

Louwerse, M., & Connell, L. (2011). A taste of words: linguistic context and perceptual simulation predict the modality of words. Cognitive Science, 35, 2, 381-98.

Mahon, B.Z., & Hickok, G. (2016). Arguments about the nature of concepts: Symbols, embodiment, and beyond. Psychonomic Bulletin & Review, 23, 941-958.