Semi-supervised semantic segmentation by self-training with ambiguity-driven online refinement

We propose a semi-supervised semantic segmentation framework based on a novel self-training method with Ambiguity-Driven Online Refinement (ADOR). Compared to recent methods based on consistency regularization that only use a limited amount of unlabeled data due to the need to filter noisy pseudo-labels, our approach leverages a larger amount of unlabeled data for training while being able to dynamically refine pseudo-labels to mitigate noise. To achieve this, we introduce Spatial Ambiguity Resolution (SAR) that selectively filters and refines the ambiguous regions in pseudo-labels produced by CNNs and ViTs. The proposed method leverages both local and global confidence as a robust estimate for label quality, and a stochastic correction disrupts the accumulation of errors. In addition, we present Semantic Ambiguity Learning (SAL) to adaptively enhance class separability based on varying levels of pairwise class confusion. Specifically, we adjust the pairwise similarity between a query feature and its positive and negative class prototypes, in order to place more emphasis on classes that are prone to confusion. Experimental results on both the Pascal VOC and Cityscapes datasets demonstrate the efficacy of our method with both CNNs and ViTs. Code will be made publicly available at https://github.com/LlistenL/ADOR.