Congruent audiovisual speech enhances auditory attention decoding with EEG.

Area of Science:

• Neuroscience
• Audiology
• Signal Processing

Background:

• Auditory attention decoding (AAD) uses electroencephalography (EEG) to identify attended speakers in complex auditory scenes.
• AAD has potential applications in hearing aid technology for speech enhancement, improving communication and reducing cognitive load.
• Investigating multimodal sensory input, like visual lipreading, may further enhance AAD performance.

Purpose of the Study:

• To evaluate the impact of additional visual input (lipreading) on auditory attention decoding (AAD) performance in normal-hearing listeners.
• To determine if congruent audiovisual speech stimuli improve AAD accuracy compared to audio-only conditions.

Main Methods:

• Participants selectively attended to one of two audiobook narrators while EEG was recorded.
• Stimuli included audio-only, and audio-visual conditions with congruent or incongruent visual lipreading information.
• AAD was applied to EEG data for each condition to assess performance.

Main Results:

• AAD performance improved by approximately 14 percentage points when visual input was congruent with the attended speaker.
• Congruent audiovisual speech enhanced cortical envelope tracking in auditory and visual cortices.
• The congruent audiovisual condition demonstrated greater AAD robustness, achieving higher accuracy with fewer EEG channels and shorter trial durations.

Conclusions:

• Congruent visual lipreading significantly enhances auditory attention decoding performance.
• Findings support the feasibility of AAD-guided hearing aid design for real-world conversations.
• Results provide guidance for developing efficient, low-density EEG setups for AAD applications.