Language/Culture Modulates Brain and Gaze Processes in Audiovisual Speech Perception

Area of Science:

• Neuroscience
• Cognitive Science
• Linguistics

Background:

• Audiovisual speech perception, the integration of auditory and visual speech information, varies across individuals.
• Previous behavioral studies indicate native English speakers (ESs) rely more on visual cues than native Japanese speakers (JSs).
• The underlying neural mechanisms and attentional differences driving these behavioral variations remain largely unexplored.

Purpose of the Study:

• To investigate the neurophysiological and oculomotor differences between ESs and JSs in audiovisual speech perception.
• To examine how language and/or cultural background influence the processing of multisensory speech signals.
• To elucidate the time-varying neural processes and attentional patterns associated with group differences in audiovisual speech perception.

Main Methods:

• Utilized event-related potentials (ERPs) and eye-tracking technology to record brain activity and gaze patterns during audiovisual and audio-only speech perception tasks.
• Compared behavioral responses, specifically reaction times, between native English speakers and native Japanese speakers under congruent and incongruent audiovisual conditions.
• Analyzed ERP components (e.g., P2 amplitude) and eye gaze metrics (e.g., fixation patterns, pre-stimulus gaze bias) to assess differences in speech processing.

Main Results:

• Behaviorally, congruent visual information facilitated speech perception (shortened response time) for ESs but hindered it for JSs.
• Eye-tracking revealed a significant gaze bias towards the mouth in ESs, particularly before auditory stimulus onset, which was absent in JSs.
• ERP analysis showed that ESs processed multisensory speech more efficiently than auditory-only speech, indicated by P2 amplitude, whereas JSs exhibited the reverse pattern.

Conclusions:

• Native language and cultural experiences significantly shape audiovisual speech processing, leading to distinct neural and attentional strategies.
• ESs' heightened visual attention to the mouth appears to facilitate phonetic anticipation, a mechanism less prominent in JSs.
• These findings highlight the adaptive nature of the brain, developing specialized neural systems for multisensory speech perception based on linguistic and cultural immersion.