Incorporating Pre-Training Data Matters in Unsupervised Domain Adaptation.

Area of Science:

• Deep Learning
• Computer Vision
• Machine Learning

Background:

• Pre-trained models are standard in deep learning for downstream tasks.
• Unsupervised domain adaptation (UDA) often uses ImageNet pre-trained backbones, focusing on source-target domain discrepancy.
• The influence of pre-training on UDA effectiveness is under-explored.

Purpose of the Study:

• Investigate the impact of pre-training on unsupervised domain adaptation (UDA).
• Analyze how pre-training affects model performance and error bounds in UDA.
• Propose a novel framework to leverage pre-training data for improved UDA.

Main Methods:

• Analyzed dynamic distribution discrepancies between pre-training, source, and target domains.
• Identified pre-trained knowledge degradation and gradient differences as sources of target error.
• Proposed TriDA, a framework treating UDA as a three-domain problem (source, target, pre-training).
• Developed pre-training data selection and synthesis strategies for efficiency and availability.

Main Results:

• Demonstrated that pre-training significantly impacts UDA performance.
• Showed that target error arises from degenerative pre-trained knowledge and theoretical error bounds.
• TriDA effectively maintains pre-trained knowledge and improves error bounds by incorporating pre-training data.
• Achieved state-of-the-art performance on multiple benchmarks in both vanilla and source-free UDA.

Conclusions:

• Pre-training is a critical factor influencing UDA, not just a initialization step.
• The TriDA framework offers a novel approach to enhance UDA by explicitly considering the pre-training domain.
• The findings provide new insights into understanding and applying domain adaptation techniques.