Correlating subword articulation with lip shapes for embedding aware audio-visual speech enhancement

Area of Science:

• Speech processing
• Computer vision
• Machine learning

Background:

• Speech enhancement aims to improve audio quality in noisy environments.
• Existing methods often rely solely on audio features.
• Visual cues from lip movements offer complementary information for speech enhancement.

Purpose of the Study:

• To propose a novel visual embedding approach for embedding aware speech enhancement (EASE).
• To synchronize visual lip frames at phone and articulation place levels for improved speech enhancement.
• To develop a multi-modal EASE (MEASE) framework integrating audio and visual features.

Main Methods:

• Extracting visual embeddings from lip frames using pre-trained phone or articulation place recognizers for visual-only EASE (VEASE).
• Developing a MEASE framework by extracting audio-visual embeddings through information intersection of audio and visual features.
• Utilizing subword-based embeddings and focusing on articulation place level for enhanced visual feature extraction.

Main Results:

• Subword-based VEASE outperforms conventional word-level embedding for speech enhancement.
• Visual embedding at the articulation place level shows superior performance compared to the phone level.
• The proposed MEASE framework significantly enhances speech quality and intelligibility over single-modality systems.

Conclusions:

• Visual embedding synchronization at phone and articulation place levels effectively improves speech enhancement.
• Multi-modal EASE (MEASE) leveraging audio-visual feature complementarity offers substantial gains in speech quality and intelligibility.
• The proposed approach represents a significant advancement in robust speech enhancement technology.