Static-dynamic class-level perception consistency in video semantic segmentation

Video semantic segmentation (VSS) has been widely employed in lots of fields, such as simultaneous localization and mapping, autonomous driving and surveillance. Its core challenge is how to leverage temporal information to achieve better segmentation. Previous efforts have primarily focused on pixel-level static-dynamic contexts matching, utilizing techniques such as optical flow and attention mechanism. Instead, this paper rethinks static-dynamic contexts at the class level and proposes a novel static-dynamic class-level perceptual consistency (SD-CPC) framework. In this framework, we propose multivariate class prototype with contrastive learning and a static-dynamic semantic alignment module. The former provides class-level constraints for the model, obtaining personalized inter-class features and diversified intra-class features. The latter first establishes intra-frame spatial multi-scale and multi-level correlations to achieve static semantic alignment. Then, based on cross-frame static perceptual differences, it performs two-stage cross-frame selective aggregation to achieve dynamic semantic alignment. Meanwhile, we propose a novel window-based attention map calculation method that leverages the sparsity of cross-frame attention points and the Hadamard product, which reduces the computational cost of cross-frame attention aggregation. It is worth noting that the proposed method achieves a 51.1 mIoU on the VSPW dataset using MiT-B5, and 81.6 mIoU and 78.2 mIoU on the Cityscapes and CamVid datasets, respectively, using ResNet101. These results surpass those of other existing state-of-the-art methods. Our implementation will be open-sourced on GitHub.