Related Experiment Video
Updated: Jul 27, 2026
12:43
A Protocol for Comprehensive Assessment of Bulbar Dysfunction in Amyotrophic Lateral Sclerosis ALS
Published on: February 21, 2011
34.8K
A Facial Feature and Lip Movement Enhanced Audio-Visual Speech Separation Model
Guizhu Li1, Min Fu1,2, Mengnan Sun1
1College of Electronic Engineering, Ocean University of China, Qingdao 266100, China.
Sensors (Basel, Switzerland)
|November 14, 2023
Summary
This study enhances audio separation for the cocktail party problem by integrating visual cues like facial features and lip movement. The proposed method significantly improves audio-visual speech separation performance.
Area of Science:
- Artificial Intelligence
- Signal Processing
- Computer Vision
Background:
- The cocktail party problem, separating mixed audio signals, is challenging.
- Leveraging multi-modal information, specifically audio-visual cues, offers a promising avenue for improvement.
Purpose of the Study:
- To propose a novel audio separation method that effectively utilizes visual information from facial features and lip movements.
- To enhance the performance of audio separation in complex acoustic environments.
Main Methods:
- Incorporated residual connections in the audio separation module for detailed feature extraction.
- Employed an attention mechanism in the face module to focus on relevant visual information.
- Designed a loss function that maximizes audio-visual similarity.
Main Results:
- The proposed model demonstrated significant improvements in audio separation metrics, including SDR, PSEQ, and STOI.
- Achieved notable performance gains, with up to 4 dB enhancement in SDR on the VoxCeleb2 dataset.
Conclusions:
- Integrating visual cues, particularly facial features and lip movement, substantially enhances audio separation capabilities.
- The developed audio-visual approach offers a robust solution for the cocktail party problem.
