Plain-to-clear speech video conversion for enhanced intelligibility.

Area of Science:

• Speech processing and computer vision
• Human-computer interaction
• Audiology and speech-language pathology

Background:

• Clearly articulated speech demonstrably improves intelligibility compared to plain speech.
• Visual speech cues in video-only formats are crucial for speech perception, especially in noisy environments.
• Systematic modification of visual speech features to enhance intelligibility remains an underexplored area.

Purpose of the Study:

• To investigate the systematic modification of visual speech cues to enhance clear-speech features.
• To improve speech intelligibility using synthesized clear-speech videos derived from plain speech.
• To evaluate the effectiveness of these synthesized videos using both AI lip-reading and human intelligibility tests.

Main Methods:

• Extraction of clear-speech visual features from videos of English words with varying vowels.
• Application of extracted features to plain speech videos using an image-warping technique with a 'displacement factor'.
• Synthesis of novel clear-speech videos and evaluation via state-of-the-art AI lip readers and human participants.

Main Results:

• Successfully extracted and applied visual cues to enhance speech intelligibility for AI lip readers.
• Demonstrated that universal, talker-independent clear-speech features can modify visual speech styles.
• Introduced the 'displacement factor' for quantifiable scaling of visual modifications between speech styles.

Conclusions:

• Systematic enhancement of visual speech features can significantly improve AI-based speech intelligibility.
• The findings suggest the potential for talker-independent visual speech modification techniques.
• Generated high-definition videos are suitable for future human-centric intelligibility and perceptual training studies.