Animated virtual characters to explore audio-visual speech in controlled and naturalistic environments

Area of Science:

• Cognitive Science
• Neuroscience
• Speech Processing

Background:

• Natural speech perception integrates auditory and visual cues.
• The McGurk effect, a perceptual illusion from audio-visual mismatch, is crucial for studying speech processing.
• Existing McGurk effect studies use isolated syllables, limiting ecological validity and comparability.

Purpose of the Study:

• To develop and validate an innovative methodology for audio-visual speech perception research.
• To overcome limitations of previous studies by using computer-generated speech with realistic lip movements.
• To investigate the influence of background noise and audiovisual lag on the McGurk effect within meaningful sentences.

Main Methods:

• Utilized 3D virtual characters with synchronized, computer-synthesized speech.
• Employed affordable and accessible tools to ensure reproducibility and comparability.
• Validated the setup with 24 participants in a perception task involving audiovisual mismatch (/v/ viseme with /b/ phoneme) in French sentences.

Main Results:

• The audiovisual mismatch illusion occurred in 40% of trials.
• The illusion rate increased to approximately 50% with added background noise.
• Controlling for phonetic features further elevated the illusion rate to 66%.

Conclusions:

• Computer-generated speech stimuli offer a viable and controlled alternative to natural speech in research.
• This methodology enhances stimulus control over timing and content for audio-visual speech research.
• The findings support the use of virtual characters for reproducible and comparable studies on speech perception.