Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptation.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Current audio-visual learning research prioritizes task-specific models using complex multimodal fusion.
• Recent advancements focus on universal audio-visual embedding networks for diverse downstream tasks.
• Parameter-efficient fine-tuning of large pretrained transformers is common but memory-intensive due to deep backbones.

Purpose of the Study:

• To introduce Meta-Token Learning (Mettle), a novel, memory-efficient method for adapting large pretrained transformer models to audio-visual tasks.
• To address the high training memory consumption of existing parameter-efficient fine-tuning techniques.
• To enhance accessibility for researchers with constrained computational resources.

Main Methods:

• Mettle employs a lightweight Layer-Centric Distillation (LCD) module to distill intact audio/visual features from each transformer layer into compact meta-tokens.
• The distillation process balances pretrained knowledge preservation with task-specific adaptation.
• A Meta-Token Injection (MTI) module is introduced for fine-grained segmentation tasks, guiding earlier layer adaptation using top-layer distilled meta-tokens.

Main Results:

• Mettle significantly reduces memory usage and training time compared to existing methods.
• The approach maintains parameter efficiency.
• Competitive accuracy is achieved across various audio-visual benchmarks, including event localization, video parsing, and segmentation tasks.

Conclusions:

• Mettle provides a simple and memory-efficient solution for adapting large-scale pretrained transformer models for audio-visual tasks.
• The method democratizes access to advanced audio-visual learning by lowering computational barriers.
• Mettle demonstrates strong performance across diverse audio-visual benchmarks, highlighting its versatility and effectiveness.