LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Speech Processing

Background:

• Crossmodal learning leverages natural audio-visual correlations in videos.
• Self-supervised pre-training reduces reliance on human annotations for speech reconstruction.

Purpose of the Study:

• Investigate the effectiveness of crossmodal self-supervised pre-training for video-to-audio speech reconstruction.
• Develop a model (LipSound2) capable of mapping visual facial sequences to audio spectrograms.

Main Methods:

• Utilized an encoder-decoder architecture with a location-aware attention mechanism.
• Pre-trained LipSound2 on multilingual audio-visual data (VoxCeleb2).
• Fine-tuned the model on domain-specific English datasets (GRID, TCD-TIMIT) and Chinese datasets (CMLR).

Main Results:

• Achieved significant improvements in speech quality and intelligibility for English speech reconstruction.
• Demonstrated generalizability across speaker-dependent and independent settings.
• Showcased transferability for Chinese speech reconstruction and state-of-the-art performance in cascaded lip reading systems.

Conclusions:

• Crossmodal self-supervised pre-training is highly effective for speech reconstruction from video.
• LipSound2 offers a robust and adaptable solution for lip reading and speech recognition tasks.
• The approach advances the state-of-the-art in both English and Chinese speech-related AI applications.