End-to-End Video-to-Speech Synthesis Using Generative Adversarial Networks

Area of Science:

• Artificial Intelligence
• Speech Technology
• Computer Vision

Background:

• Traditional video-to-speech methods use multi-step processes with intermediate representations.
• These methods often rely on separate vocoders or waveform reconstruction algorithms, limiting direct audio synthesis.

Purpose of the Study:

• To develop a novel, end-to-end video-to-speech model.
• To achieve direct waveform audio synthesis from raw video input without intermediate representations.

Main Methods:

• An encoder-decoder architecture based on generative adversarial networks (GANs) was employed.
• The model utilizes waveform and power critics with adversarial loss for direct audio synthesis.
• Three comparative losses ensure correspondence between generated audio and input video.

Main Results:

• The model successfully reconstructs speech with high realism on constrained datasets like GRID.
• It is the first end-to-end model to generate intelligible speech for the challenging Lip Reading in the Wild (LRW) dataset.
• Evaluations on seen and unseen speakers demonstrated superior performance across multiple objective metrics compared to prior work.

Conclusions:

• The proposed end-to-end GAN-based video-to-speech model offers a significant advancement in direct waveform synthesis.
• This approach achieves state-of-the-art results in speech reconstruction realism and intelligibility for both controlled and in-the-wild scenarios.