Speech-In-Noise Comprehension is Improved When Viewing a Deep-Neural-Network-Generated Talking Face.

Area of Science:

• Audiology and Speech Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Speech comprehension is difficult in noisy environments.
• Visual cues from a talker's face significantly improve speech understanding.
• Deep neural networks (DNNs) can synthesize talking face videos from audio.

Purpose of the Study:

• To quantify the benefits of DNN-generated talking face videos for speech comprehension in noise.
• To compare synthesized audio-visual (AV) stimuli with natural AV stimuli and audio-only conditions.

Main Methods:

• Speech audio was masked at various signal-to-noise ratios (-9 to 0 dB).
• Subjects experienced three conditions: synthesized AV, natural AV, and audio-only.
• Speech comprehension was assessed by typed sentence recall and keyword recognition.

Main Results:

• Synthesized AV stimuli showed significant improvement over audio-only, falling between audio-only and natural AV performance.
• All subjects benefited from the synthesized AV condition.
• Performance improved with increasing signal-to-noise ratios across all conditions.

Conclusions:

• DNN-based synthesized talking face videos meaningfully enhance speech comprehension in noisy environments.
• This technology shows potential as a visual hearing aid.
• Further research can optimize DNN models for even greater comprehension benefits.