Two-Step Joint Optimization with Auxiliary Loss Function for Noise-Robust Speech Recognition.

Area of Science:

• Machine Learning
• Speech Processing
• Signal Processing

Background:

• Pipeline models in speech processing often involve separate optimization of individual components, potentially leading to suboptimal overall performance.
• Existing joint optimization methods may not effectively align the goals of front-end and back-end models in a pipeline.

Purpose of the Study:

• To propose a new two-step joint optimization approach for training pipeline models composed of distinct front-end and back-end components.
• To enhance the performance of a speech enhancement and automatic speech recognition (ASR) pipeline by optimizing both models concurrently.

Main Methods:

• A novel two-step joint optimization strategy is introduced, building upon the asynchronous subregion optimization method.
• The first step uses a new loss function to align the front-end model's objective with the back-end model's goal.
• The second step jointly trains all parameters of the combined pipeline model.

Main Results:

• The proposed approach was applied to a pipeline combining a Deep Complex Convolutional Recurrent Network (DCCRN) for speech enhancement and a Conformer-Transducer for ASR.
• Evaluated on the LibriSpeech corpus, the joint optimization significantly reduced character error rate (CER) and word error rate (WER) compared to separate optimization.
• The method outperformed conventional two-step joint optimization, achieving lower CER and WER under both matched and mismatched noise conditions.

Conclusions:

• The proposed two-step joint optimization approach effectively trains cascaded speech enhancement and ASR models.
• This method leads to superior performance in noisy environments, outperforming both separate and conventional joint optimization techniques.
• The optimized pipeline demonstrates improved robustness and accuracy for automatic speech recognition.