Eye Can Hear Clearly Now: Inverse Effectiveness in Natural Audiovisual Speech Processing Relies on Long-Term...

Area of Science:

• Neuroscience
• Auditory Perception
• Speech Processing

Background:

• Visual speech input enhances auditory speech comprehension, particularly in noisy environments.
• The principle of inverse effectiveness describes this phenomenon, where benefits are greater under adverse conditions.
• Neural mechanisms of audiovisual (AV) speech integration in noise remain incompletely understood.

Purpose of the Study:

• To investigate the neural mechanisms underlying the inverse effectiveness of audiovisual speech integration.
• To examine how visual speech enhances the cortical representation of auditory speech in noise.
• To identify the temporal dynamics of multisensory integration during speech perception in challenging listening conditions.

Main Methods:

• Human electroencephalography (EEG) recordings were used to measure neural responses.
• Natural speech-in-noise stimuli were employed, optimized to maximize the behavioral benefit of AV speech.
• Analysis focused on neural tracking of the auditory speech signal and crossmodal integration at different frequencies and time lags.

Main Results:

• Visual speech significantly enhanced neural tracking of auditory speech in noise, consistent with inverse effectiveness.
• Background noise reduced early auditory envelope tracking, which was restored by visual speech input.
• Multisensory integration in noise occurred at lower frequencies and predicted behavioral benefits at approximately 250 ms, relying on long temporal windows.

Conclusions:

• Inverse effectiveness in audiovisual speech processing relies on crossmodal integration over extended temporal periods.
• Visual speech plays a crucial role in restoring early neural tracking of auditory signals in noisy environments.
• Disparate neural integration mechanisms contribute to efficient audiovisual speech processing under challenging acoustic conditions.