Unified Modality Separation: A Vision-Language Framework for Unsupervised Domain Adaptation.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Unsupervised domain adaptation (UDA) aims to generalize models from labeled source domains to unlabeled target domains.
• Pre-trained vision-language models (VLMs) show promise in UDA by aligning vision and text embeddings, but the modality gap hinders performance.
• Direct UDA with VLMs often transfers only modality-invariant knowledge, leading to suboptimal results.

Purpose of the Study:

• To propose a unified framework for UDA that addresses the modality gap in VLMs.
• To disentangle and separately handle modality-specific and modality-invariant components for improved adaptation.
• To introduce a modality discrepancy metric for sample categorization and targeted adaptation.

Main Methods:

• A unified modality separation framework is proposed to disentangle VLM features into modality-specific and invariant components.
• Modality-adaptive ensemble weights are determined at test time to maximize component synergy.
• A modality discrepancy metric categorizes samples, enabling targeted use of modality-invariant samples for alignment and uncertain samples for annotation.

Main Results:

• The proposed framework achieves up to a 9% performance gain in UDA tasks.
• The method demonstrates a 9-fold increase in computational efficiency compared to existing approaches.
• Experiments across diverse settings validate the framework's efficacy and robustness.

Conclusions:

• The unified modality separation framework effectively bridges the modality gap in VLMs for UDA.
• The approach enhances cross-domain generalization by leveraging both modality-specific and invariant knowledge.
• This work offers a computationally efficient and high-performing solution for UDA with VLMs.