Congruency of auditory sounds and visual letters modulates mismatch negativity and P300 event-related potentials

A key determinant of skilled reading is the ability to integrate the orthographic and auditory forms of language. A number of prior studies have identified neural markers in adult readers corresponding to audio-visual integration of letters and their corresponding sounds. However, there remains some controversy as to the stage of processing at which this occurs. In the present study, we examined this issue using event-related potentials (ERPs), due to their sensitivity to the timing of perceptual and cognitive processes. Letter sounds were presented auditorily in an unattended mismatch negativity (MMN) paradigm, which is argued to be indicative of auditory sensory memory. Concurrently, participants performed a visual letter identification task. On critical trials, the auditory stimulus was played concurrently with the visual letters. We observed significant MMNs both when the visual letter was congruent with the auditory stimulus, and when it was incongruent. However, the magnitude and scalp distribution of this effect was attenuated in incongruent trials. We also observed a later-going effect of congruency on P300 trials, marked by increased amplitudes and latencies for incongruent compared to congruent trials. The results suggest audiovisual integration of letters and sounds can and does occur during relatively early pre-attentive stages of sensory processing, and that these effects extend to later-going attentional phases of processing as well.