Speaking mode variability in multimodal speech production

Area of Science:

• Speech processing
• Biomedical engineering
• Machine learning

Background:

• Speech acoustics and vocal tract behavior are linked to facial motion during speech.
• Neural networks can estimate facial motion from speech acoustics.
• Current methods struggle with multi-sentence analysis, limiting reliable estimation.

Purpose of the Study:

• Investigate if recording procedures create artificial boundaries affecting neural network generalization.
• Determine the impact of utterance length (sentence vs. paragraph) on estimation accuracy.
• Assess the influence of phonetic content on the reliability of face motion estimation from speech.

Main Methods:

• Recorded identical sentence materials individually and within paragraph-length utterances.
• Trained simple neural networks to estimate facial motion from spectral acoustics.
• Compared network performance across different utterance lengths and phonetic content variations.

Main Results:

• Reliable estimation of face motion from speech acoustics improved with larger sentence training sets when sentences were part of continuous paragraph readings.
• Network estimation reliability was dependent on utterance length.
• Increased phonetic diversity within utterances reduced estimation reliability.

Conclusions:

• Continuous speech recording procedures enhance neural network generalization for face motion estimation from speech acoustics.
• Utterance length and phonetic content are critical factors influencing the accuracy of speech-driven facial motion analysis.
• Optimizing recording methods can improve the performance of neural networks in speech production studies.