End-to-end feature fusion for jointly optimized speech enhancement and automatic speech recognition

Area of Science:

• Signal Processing
• Artificial Intelligence
• Machine Learning

Background:

• Real-time speech enhancement (SE) and automatic speech recognition (ASR) are critical for clear communication and accurate transcription.
• Traditional SE methods can distort speech, negatively impacting downstream ASR performance.
• Existing joint SE-ASR models often use only enhanced features, potentially losing valuable information.

Purpose of the Study:

• To develop an SE network that suppresses noise while minimizing speech distortion.
• To propose a dynamic fusion approach integrating enhanced and raw noisy speech features for improved ASR.
• To create a joint training framework for robust end-to-end ASR.

Main Methods:

• An attentional codec model with a causal attention mechanism for SE.
• A modified Gated Recurrent Unit (GRU) in the SE network, using an attention-based rectified linear unit (AReLU).
• A GRU-based fusion network combining enhanced and raw noisy features, feeding into an ASR system.

Main Results:

• The proposed SE model achieved superior speech quality, intelligibility, and noise suppression compared to baselines in matched and unmatched conditions.
• Significant improvements in STOI (19.81%) and PESQ (28.97%) were observed in matched conditions.
• The joint training framework reduced character error rate (CER) from 32.99% to 13.52% for ASR.

Conclusions:

• The dynamic fusion approach effectively mitigates speech distortions and preserves crucial details from noisy signals.
• The proposed SE and joint training framework significantly enhance the robustness and accuracy of real-time ASR systems.
• This integrated approach offers a promising solution for challenging acoustic environments.