Modulation transfer functions for audiovisual speech

Area of Science:

• Speech processing
• Auditory-visual perception
• Human-computer interaction

Background:

• Temporal synchrony between facial movements and speech acoustics is crucial for audiovisual speech perception.
• While low-frequency acoustic envelope fluctuations (below 10 Hz) correlate with facial motion, precise temporal relationships across different facial regions remain unclear.

Purpose of the Study:

• To investigate the precise temporal scales of synchronization between speech envelope modulations and facial motion in natural audiovisual speech.
• To identify distinct temporal patterns in facial movements corresponding to different speech rhythm scales.

Main Methods:

• Utilized regularized canonical correlation analysis (rCCA) to model speech envelope filters.
• Employed advanced video-based 3D facial landmark estimation on a large dataset (∼4000 speakers).
• Learned modulation transfer functions (MTFs) to correlate speech envelope with facial motion across speakers.

Main Results:

• Identified two distinct temporal scales of audiovisual speech synchrony: 3-4 Hz correlated with mouth movements and 1-2 Hz with global face/head motion.
• These timescales emerged specifically in natural audiovisual speech statistics across many speakers.
• Controlled speech tasks revealed only the 3-4 Hz modulations, suggesting slower rhythms are unique to natural speech.

Conclusions:

• Natural audiovisual speech exhibits distinct temporal regularities at syllable (3-4 Hz) and phrase (1-2 Hz) timescales, linked to specific facial motion patterns.
• The emergence of slower 1-2 Hz regularities only in crossmodal statistics suggests a potential motor origin for phrase-level speech timing.