Token Calibration for Transformer-Based Domain Adaptation.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Unsupervised Domain Adaptation (UDA) transfers knowledge from labeled source to unlabeled target domains.
• Vision Transformers (ViTs) offer fine-grained patch-level representations for UDA, termed Transformer-based Domain Adaptation (TransDA).
• Domain shifts in TransDA can cause misleading patch similarities, degrading adaptation performance.

Purpose of the Study:

• To propose a novel method, Patch-Adaptation Transformer (PATrans), to address the challenge of misleading patch similarities in TransDA.
• To identify and suppress the negative impact of similarity-anomalous patches on domain alignment.
• To enhance the robustness and performance of UDA using ViT architectures.

Main Methods:

• Introduced a Patch-Adaptation Attention (PAA) mechanism replacing standard self-attention.
• PAA integrates a weight-shared triple-branch mixed attention and a patch-level domain discriminator for token calibration.
• Implemented a contrastive attention regularization strategy using category-level information for class-consistent attention distributions.

Main Results:

• PATrans effectively identifies and mitigates the impact of unreliable patch correspondences.
• The proposed method achieves significant improvements over existing state-of-the-art UDA methods.
• Demonstrated strong performance on four benchmark datasets, reaching 89.2% on VisDA-2017.

Conclusions:

• PATrans offers a robust solution for Transformer-based Domain Adaptation by effectively handling domain shifts.
• The novel attention mechanism and regularization strategy enhance feature learning and domain alignment.
• The approach sets a new benchmark for unsupervised domain adaptation using Vision Transformers.