Visual speech segmentation: using facial cues to locate word boundaries in continuous speech

Area of Science:

• Cognitive Science
• Developmental Psychology
• Linguistics

Background:

• Speech acquisition is typically multimodal, but the role of visual cues remains understudied.
• Language learners can use prosodic cues (stress, pitch) for speech segmentation.
• Learners can extract prosodic information from visual facial cues.

Purpose of the Study:

• To investigate if visual prosodic information facilitates speech segmentation in language acquisition.
• To determine the impact of informative versus uninformative facial cues on learning word boundaries.

Main Methods:

• Created an artificial speech stream with minimal segmentation cues.
• Paired the audio stream with synchronous facial displays showing informative or uninformative visual prosody.
• Tested learning across three familiarization conditions: audio alone, visual alone, and audiovisual.

Main Results:

• Learning, specifically identifying word boundaries, occurred only when facial displays provided informative visual prosody.
• No learning was observed with audio-only or uninformative visual-only conditions.

Conclusions:

• Visual prosody from talking faces can significantly aid speech segmentation.
• Facial cues are crucial for overcoming early challenges in language acquisition.
• This highlights the importance of multimodal input in infant language development.