Robust Audio-Visual Question Answering with Missing Modality in Training and Testing

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Audio-Visual Question Answering (AVQA) typically requires complete audio and visual data for reasoning over dynamic scenes.
• Real-world scenarios often involve missing modalities due to signal loss or privacy concerns, challenging existing AVQA methods.
• A new setting, Training-time Modality-Missing AVQA (TM-AVQA), is proposed to address AVQA with potentially incomplete data during training and testing.

Purpose of the Study:

• To develop a robust AVQA framework capable of handling missing audio or visual modalities during both training and inference.
• To introduce novel techniques for cross-modal reconstruction and dependency modeling under modality-missing constraints.
• To evaluate the effectiveness of the proposed methods on established AVQA datasets adapted for modality-missing scenarios.

Main Methods:

• A two-stage AVQA framework is proposed, incorporating a reconstruction network and an answer prediction stage.
• Stage-I features a reconstruction network with Dense Temporal-scale Reconstruction (DTR) and Multimodal Dependency Modeling (MDM) modules.
• Novel learning objectives, Cross-Modal Relation-based Contrastive Learning (CMR-CL) and Cross-Sample Relation-based Pseudo-label Learning (CSR-PL), are adapted to supervise reconstruction with missing modalities.

Main Results:

• The proposed TM-AVQA framework demonstrated consistent and robust improvements across various AVQA backbones and missing-modality conditions.
• Experiments on modified MUSIC-AVQA, MUSIC-AVQA-R, and AVQA datasets confirmed the effectiveness of the approach.
• Ablation studies validated the contributions of task-specific designs, reconstruction quality, and efficiency.

Conclusions:

• The developed framework effectively addresses the challenge of missing modalities in AVQA, outperforming existing baselines.
• The proposed reconstruction and learning strategies enable robust performance even when one modality is entirely absent.
• The approach shows potential for transfer learning to related audio-visual recognition tasks.