Leveraging Self-Supervised Audio-Visual Pretrained Models to Improve Vocoded Speech Intelligibility in Cochlear...

Area of Science:

• Audiology
• Speech Processing
• Machine Learning

Background:

• Hearing impairments present significant challenges in speech comprehension, especially in noisy environments.
• Cochlear implants (CIs) aim to restore hearing but can struggle with speech intelligibility, particularly with processed (vocoded) speech.
• Audio-visual speech enhancement (AVSE) offers a potential solution by leveraging visual cues like lip movements.

Purpose of the Study:

• To evaluate the effectiveness of a novel Self-Supervised Learning-based Audio-Visual Speech Enhancement (SSL-AVSE) framework for improving vocoded speech intelligibility in cochlear implant (CI) simulations.
• To investigate the performance of SSL-AVSE compared to existing methods.
• To assess the cross-lingual generalization capabilities of the proposed model.

Main Methods:

• Developed the SSL-AVSE framework, integrating visual speech cues (lip/mouth movements) with audio.
• Utilized the AV-HuBERT model for feature extraction and a bidirectional LSTM for refinement.
• Conducted experiments on the Taiwan Mandarin Speech with Video (TMSV) dataset.

Main Results:

• Objective metrics showed significant improvements: PESQ increased from 1.43 to 1.67, and STOI improved from 0.70 to 0.74.
• NCM scores saw an increase of up to 87.2% compared to the noisy baseline.
• Subjective listening tests revealed maximum gains of 45.2% in speech quality and 51.9% in word intelligibility.

Conclusions:

• SSL-AVSE demonstrates superior performance over AOSE and conventional AVSE baselines in CI simulations.
• Statistically significant listening tests confirm the effectiveness of SSL-AVSE.
• The model exhibits cross-lingual generalization, performing effectively on Mandarin speech despite English pretraining, highlighting the robustness of foundation model features.