Emphasizing Domain Differences Through Interactive-Augmented Prompts in Continual Audio-Visual Speech Recognition.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Speech Processing

Background:

• Audio-Visual Speech Recognition (AVSR) excels with complementary acoustic and visual data but struggles with real-world domain shifts.
• Heterogeneous data distributions in AVSR lead to catastrophic forgetting and reduced generalization.
• Existing AVSR models often fail in dynamic, non-stationary environments.

Purpose of the Study:

• Introduce the Continual Audio-Visual Speech Recognition (CL-AVSR) problem formulation.
• Establish a benchmark for CL-AVSR with scenarios reflecting real-world challenges (noise, video degradation, speaker variation).
• Develop a novel framework to enhance AVSR adaptability and mitigate catastrophic forgetting.

Main Methods:

• Proposed the Interaction-enhanced Multimodal Prompt learning (IMP) framework using a pre-trained AV-HuBERT backbone.
• Integrated task-relevant soft prompts with cross-modal and cross-task interactions for knowledge transfer.
• Employed contrastive regularization and a multi-modal prompt selection strategy for dynamic adaptation.

Main Results:

• IMP demonstrated substantial improvements over strong baselines on the LRS2 dataset.
• Achieved new state-of-the-art performance across all CL-AVSR experimental scenarios.
• Validated the framework's effectiveness in handling domain shifts and diverse data streams.

Conclusions:

• The IMP framework significantly enhances continual learning capabilities for AVSR systems.
• IMP offers a robust solution for adaptable multi-modal speech recognition in real-world applications.
• This work paves the way for more resilient AVSR systems facing dynamic environmental and data variations.