Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language

Area of Science:

• Psycholinguistics
• Auditory and Visual Speech Perception

Background:

• Visual cues from faces enhance auditory speech comprehension.
• Mouth movements convey segmental and durational information, while head and eyebrow movements cue prosody.
• The influence of linguistic experience on visual speech perception is not well understood.

Purpose of the Study:

• To investigate how visual scanning of a talking face predicts speech comprehension accuracy.
• To determine the effect of language familiarity on the use of visual speech cues.
• To differentiate the visual processing of segmental versus prosodic speech information.

Main Methods:

• Eye-tracking was used to monitor participants' visual attention to different facial regions.
• Participants performed a speech comprehension task involving English (native) and Mandarin (non-native) sentences.
• Stimuli varied in segmental and/or prosodic information, with participants judging sentence matching.

Main Results:

• Increased visual attention to the mouth predicted correct responses in non-native language trials.
• Fixation latency to the mouth was delayed for non-native compared to native language trials, particularly when only prosodic information was available.
• Saccade amplitude was greater in native language trials, indicating more focused visual attention in non-native trials.

Conclusions:

• Mouth-looking is more prominent when processing non-native languages.
• Language familiarity significantly modulates the visual processing of speech, especially for prosodic cues.
• These findings highlight the dynamic interplay between auditory and visual information in speech comprehension across different linguistic contexts.