Spatial alignment between faces and voices improves selective attention to audio-visual speech

Area of Science:

• Auditory Perception
• Visual Perception
• Human-Computer Interaction

Background:

• Visual cues, such as seeing a talker's face, significantly improve speech intelligibility in noisy environments when audio-visual signals are temporally synchronized.
• The role of spatial alignment between faces and voices in multi-talker scenarios remains less understood.

Purpose of the Study:

• To investigate the impact of spatial alignment between faces and voices on selective attention to speech in noise.
• To determine if spatial alignment is a critical factor for audio-visual (AV) speech perception in complex listening environments.

Main Methods:

• Conducted online experiments using a selective attention task in multi-talker noise.
• Manipulated spatial alignment of faces and voices (same vs. different hemifield) and signal-to-noise ratio (SNR).
• Investigated the influence of eye gaze direction on performance in auditory-only conditions.

Main Results:

• Task performance improved when faces and voices were spatially aligned (same hemifield).
• Spatial misalignment between audio and visual speech signals incurred a performance cost.
• The effect of AV spatial alignment was more pronounced at lower SNRs, though lipreading accuracy introduced a floor effect.

Conclusions:

• Spatial alignment between faces and voices is a significant factor contributing to the ability to selectively attend to audio-visual (AV) speech.
• Effective AV speech perception in noisy, multi-talker environments relies on both temporal and spatial congruence of speech cues.