JMSC: Joint Spatial-Temporal Modeling with Semantic Completion for Audio-Visual Learning.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Audio-visual learning integrates auditory and visual cues for holistic scene understanding.
• Previous methods often involved computationally expensive full model fine-tuning.
• Parameter-efficient tuning methods are increasingly used for adapting large models to audio-visual tasks.

Purpose of the Study:

• To address challenges in leveraging complementary semantics between audio and visual modalities.
• To improve comprehension of dynamic video context, including evolving spatial and temporal attributes.
• To propose a novel framework, Joint Spatial-Temporal Modeling with Semantic Completion (JMSC), for advanced audio-visual learning.

Main Methods:

• JMSC employs cross-modal latent reconstruction to deeply integrate semantic information from different modalities.
• It encourages reconstructing one modality's summary from a masked counterpart, moving beyond shallow correlations.
• The framework jointly models spatial attributes and temporal changes under audio guidance for unified representation.

Main Results:

• JMSC achieves state-of-the-art performance on multiple downstream audio-visual tasks.
• The method demonstrates improved semantic comprehension and accurate localization/tracking in dynamic scenes.
• JMSC maintains high computational efficiency compared to existing approaches.

Conclusions:

• JMSC offers an effective solution for integrating audio-visual information, overcoming limitations of prior methods.
• The framework enhances understanding of complex dynamic video content.
• JMSC provides a computationally efficient and high-performing approach to audio-visual learning.