Style-Aware Contrastive Test-Time Adaptation: A Dual-Cache Model for Robust Vision-Language Alignment.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Test-time adaptation (TTA) is crucial for vision-language models (VLMs) facing real-world distribution shifts.
• Existing TTA methods struggle with a trade-off between efficiency and accuracy, and suffer from discriminative collapse in fine-grained tasks.

Purpose of the Study:

• To introduce a novel framework, Style-aware Contrastive Test-Time Adaptation (SCTTA), to overcome limitations in current TTA methods.
• To jointly address modality misalignment and discriminative collapse in VLMs during test-time adaptation.

Main Methods:

• Style-aware Embedding Adaptation (SEA) refines text embeddings using domain-specific styles for improved visual-textual alignment.
• Fine-grained Contrastive Adaptation (FCA) enhances feature separation via contrastive learning with adaptive prototypes.
• Dual-Cache Model (DCM) employs multimodal caches (visual and textual) for accumulating adaptation knowledge without additional overhead.

Main Results:

• SCTTA achieves state-of-the-art performance across 15 datasets for both fine-grained and out-of-distribution benchmarks.
• The framework demonstrates continuous improvement with accumulating test samples, validating its sustainable adaptation capacity.
• SCTTA effectively mitigates modality misalignment and discriminative collapse.

Conclusions:

• SCTTA offers a robust solution for enhancing VLM performance under distribution shifts.
• The proposed methods (SEA, FCA, DCM) effectively address key challenges in test-time adaptation.
• SCTTA provides a computationally efficient and continuously adaptive approach for real-world VLM applications.