NaturalL2S: End-to-end high-quality multispeaker lip-to-speech synthesis with differential digital signal processing.

Area of Science:

• Artificial Intelligence
• Speech Technology
• Computer Vision

Background:

• Visual speech recognition (VSR) models improve lip-to-speech synthesis by providing semantic information.
• Cascade frameworks using VSR models show promise but suffer from a domain gap in mel-spectrograms, degrading synthesis quality.

Purpose of the Study:

• To propose Natural Lip-to-Speech (NaturalL2S), an end-to-end framework to bridge the mel-spectrogram domain gap.
• To enhance the intelligibility and quality of synthesized speech from lip movements.

Main Methods:

• Developed an end-to-end framework (NaturalL2S) jointly training the vocoder with acoustic inductive priors.
• Introduced a fundamental frequency (F0) predictor to model prosodic variations.
• Utilized a differentiable digital signal processing (DDSP) synthesizer driven by predicted F0 for acoustic prior generation.

Main Results:

• Achieved satisfactory speaker similarity without explicit speaker embeddings.
• Significantly enhanced synthesized speech quality compared to state-of-the-art methods based on objective metrics and subjective listening tests.
• Effectively bridged the domain gap between synthetic and real mel-spectrograms.

Conclusions:

• NaturalL2S offers a novel approach to lip-to-speech synthesis by integrating acoustic priors and F0 prediction.
• The framework demonstrates superior performance in speech synthesis quality and speaker similarity.
• This method addresses a key bottleneck in current VSR-enhanced speech synthesis systems.