Leveraging VLMs for MUDA: Category-specific prompt with multi-modal interactive LoRA.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Multi-Source Unsupervised Domain Adaptation (MUDA) aims to train models using labeled data from multiple sources and unlabeled target data.
• Existing MUDA methods lack effective integration with emerging pre-trained Visual Language Models (VLMs).

Purpose of the Study:

• To develop a novel framework for MUDA leveraging CLIP-based Visual Language Models.
• To address the limitations of current methods in adapting models to target domains using VLMs.

Main Methods:

• A CLIP-model-based framework integrating category-specific prompts and multimodal Low-Rank (LoRA) matrix adaptation.
• Utilizing learnable, class-specific prompts for shared knowledge extraction.
• Employing multimodal LoRA for domain-specific knowledge acquisition and a modality interaction mechanism.

Main Results:

• The proposed method demonstrates significant improvements on standard image classification benchmark datasets.
• Successful adaptation to target domains using a combination of shared and domain-specific knowledge.

Conclusions:

• The novel CLIP-model-based framework offers an effective solution for MUDA.
• The integration of category-specific prompts and multimodal LoRA advances VLM-based domain adaptation techniques.