Modality switches occur early and extend late in conceptual processing: Evidence from ERPs (Bernabeu, Willems, & Louwerse, in prep.)

Files: poster, design overview, stimuli, EEG montage used, waveforms, window 1 close-ups, subtraction topographies, critical results, fixed effects of final models, entire modeling, raw data. For best resolution, download image(s). Also see waveforms interactively within sections of the data, and finally the entire data set.

Abstract This experiment investigates the relevance of perceptual simulation in conceptual processing by revisiting the Conceptual Modality Switch paradigm. In each trial, participants (46) verified the relation between a property word and a concept word. These linguistic stimuli were modality-normed, and covertly created switches in conceptual modality across some trials—specifically, auditory-to-visual and haptic-to-visual switches. These two conditions were compared to the visual-to-visual (no-switch) condition. ERPs were time-locked to the first word in target trials so as to reduce within-trial confounds and to measure any effects on-line. A widespread effect was found, characterized by more negative ERP amplitudes for modality-switching than not switching. It proved significant in four typical time windows from 160 to 750 ms post word onset, showing greater strength after 350 ms and in Posterior brain regions. The results suggest that perceptual simulation may contribute to conceptual processing, but also that the early lexicosemantic stage is relatively amodal/symbolic.

References

Collins, J., Pecher, D., Zeelenberg, R., & Coulson, S. (2011). Modality switching in a property verification task: an ERP study of what happens when candles flicker after high heels click. Frontiers in Psychology, 2.

Hald, L. A., Marshall, J.-A., Janssen, D. P., & Garnham, A. (2011). Switching modalities in a sentence verification task: ERP evidence for embodied language processing. Frontiers in Psychology, 2.

Hauk, O. (2016). Only time will tell—Why temporal information is essential for our neuroscientific understanding of semantics. Psychonomic Bulletin & Review, 23.

Louwerse, M., & Connell, L. (2011). A taste of words: linguistic context and perceptual simulation predict the modality of words. Cognitive Science, 35, 2, 381-98.

Mahon, B.Z., & Hickok, G. (2016). Arguments about the nature of concepts: Symbols, embodiment, and beyond. Psychonomic Bulletin & Review, 23, 941-958.