VB-Adapter: Variational Bayesian Adapter for Cross-Domain Speech Representation Learning.

Area of Science:

• Artificial Intelligence
• Speech Processing
• Machine Learning

Background:

• Current speech models generalize well due to extensive pretraining.
• Domain shift between pretraining and fine-tuning data presents challenges for real-world speech scenarios.
• Unfamiliar speech data can lead to performance degradation in existing models.

Purpose of the Study:

• To propose a novel method for cross-domain speech representation learning during fine-tuning.
• To address the performance gap caused by domain shift in speech recognition.
• To enhance the robustness of speech models when encountering novel speech data.

Main Methods:

• Developed a variational Bayesian adapter (VB-Adapter) incorporating a latent variable model.
• Constructed a posterior distribution to bridge source and target domain gaps.
• Introduced an adaptive objective maximizing mutual information and contrastive learning for optimization.

Main Results:

• VB-Adapter demonstrated effectiveness in dysarthric speech recognition (DSR).
• Applied to Whisper-encoder and Llama for Mandarin speech recognition (MSR), showing significant improvements.
• The method successfully modeled uncertainties arising from domain shift.

Conclusions:

• VB-Adapter enhances the robustness of speech representations in cross-domain scenarios.
• The proposed approach effectively mitigates performance degradation due to domain shift.
• This work offers a promising solution for adapting pre-trained speech models to diverse real-world applications.