Covariance Attention for Semantic Segmentation

The dependency between global and local information can provide important contextual cues for semantic segmentation. Existing attention methods capture this dependency by calculating the pixel wise correlation between the learnt feature maps, which is of high space and time complexity. In this article, a new attention module, covariance attention, is presented, and which is interesting in the following aspects: 1) covariance matrix is used as a new attention module to model the global and local dependency for the feature maps and the local-global dependency is formulated as a simple matrix projection process; 2) since covariance matrix can encode the joint distribution information for the heterogeneous yet complementary statistics, the hand-engineered features are combined with the learnt features effectively using covariance matrix to boost the segmentation performance; 3) a covariance attention mechanism based semantic segmentation framework, CANet, is proposed and very competitive performance has been obtained. Comparisons with the state-of-the-art methods reveal the superiority of the proposed method.