SCADA: Sparse cross attention for domain adaptive semantic segmentation

Unsupervised domain adaptive (UDA) semantic segmentation intends to perform satisfactory dense prediction on unannotated target (real-world) images by leveraging a learned model trained on annotated source (synthetic) images. Although many models have achieved remarkable performance by utilizing attention mechanisms, they rarely focus on large-region semantic categories that typically occupy a large portion of an image. This leads to repeated computations on massive pixels with the same class, resulting in additional and unnecessary computational resources. To address this problem, we propose a sparse cross attention (SCA) block. This innovative block generates discontinuous yet effective sparse cross attention maps by aggregating comprehensive contextual information from all pixels in their horizontal and vertical directions. Consequently, it demands fewer computational resources, leading to a significant reduction in both time and space complexity. In addition, we also perceive that most UDA methods neglect the intrinsical connection between training data, which is crucial for enhancing pixel discrimination. Therefore, we perform pixel-wise contrastive learning on a latent space with extracted features, thereby promoting intra-class compactness and inter-class separability of pixel representations within and across domains. Further, we demonstrate that using SCA rectified features for contrastive learning can additionally improve the performance. Plentiful experiments validate the effectiveness of our proposed method, showcasing significant advancements on three widely-used UDA benchmarks: GTA  →  Cityscapes, Synthia  →  Cityscape, and Cityscapes  →  Dark Zurich. Moreover, our SCA block can be seamlessly integrated with other UDA methods to further boost their performance.