Training-Free Open-Set Domain Adaptation With Vision-Language Models

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Pre-trained vision-language models like CLIP offer valuable generic knowledge for domain adaptation.
• Existing CLIP-based methods are often restricted to closed-set scenarios, limiting their applicability.
• Open-Set Domain Adaptation (OSDA) is challenging because CLIP requires semantic labels for unknown classes during inference.

Purpose of the Study:

• To propose a novel Semantic-guided Target Adaptation (SemTA) framework for Open-Set Domain Adaptation (OSDA).
• To enable training-free domain adaptation by leveraging both CLIP and a source model.
• To improve the practicality and interpretability of OSDA methods.

Main Methods:

• Developed an unknown semantic discovery module using target data cluster centroids to identify unknown class semantics from a global corpus.
• Integrated CLIP for semantic-based inference and a dual sample attention mechanism for sample-based inference.
• Utilized representative features from both source and CLIP models to enhance task specificity.

Main Results:

• The proposed SemTA framework achieves state-of-the-art performance on four benchmark datasets without requiring any training.
• Demonstrated superior practicality and interpretability compared to traditional OSDA methods that rely on confidence thresholds.
• Successfully adapted models to new domains by effectively handling unknown classes.

Conclusions:

• The SemTA framework presents a significant advancement in Open-Set Domain Adaptation, offering a training-free and effective solution.
• The method's ability to discover and utilize unknown class semantics enhances its applicability in real-world scenarios.
• Future work will involve making the code publicly available to facilitate further research and development.