Multi-Stage Audio-Visual Fusion for Dysarthric Speech Recognition With Pre-Trained Models

Area of Science:

• Speech Recognition
• Machine Learning
• Biomedical Engineering

Background:

• Dysarthric speech recognition is crucial for communication but hindered by data scarcity.
• Existing machine learning models struggle with insufficient dysarthric speech data, leading to poor performance.
• Traditional audio-visual fusion methods often focus solely on lip movements.

Purpose of the Study:

• To enhance the accuracy of dysarthric speech recognition.
• To address the challenge of limited dysarthric speech data for model training.
• To propose an effective audio-visual fusion framework for improved recognition.

Main Methods:

• Developed a Multi-stage Audio-Visual HuBERT (MAV-HuBERT) framework.
• Utilized convolutional neural networks to encode motor information from all facial speech function areas.
• Employed AV-HuBERT for pre-training, fusing audio and visual information to mitigate overfitting.

Main Results:

• Achieved a 13.5% reduction in word error rate (WER) for moderate dysarthric speech compared to baseline.
• Obtained a WER of 6.05% for mild dysarthric speech.
• Reduced WER by 2.72% and 4.02% for severe dysarthric speech compared to wav2vec and HuBERT, respectively.

Conclusions:

• The proposed MAV-HuBERT framework effectively improves dysarthric speech recognition accuracy.
• Fusing comprehensive facial motor information with acoustic data enhances model robustness.
• This method offers a promising solution for better communication accessibility for individuals with dysarthria.