Lightweight audiovisual attention cross-perception network for video saliency prediction

Audio signals, as a crucial component of the human perceptual system, can assist video saliency models in semantic localization. However, integrating audio signals requires rich information representations, which may increase computational costs. Additionally, the inherent differences between auditory and visual features can lead to information confusion during the fusion process. To address this issue, this paper proposes a lightweight audiovisual attention cross-perception network (LAAC-Net) for video saliency prediction. Specifically, we introduce a lightweight two-stream audiovisual feature encoder to reduce computational overhead. To integrate multi-modal features, including temporal, spatial, and auditory features, we design an audiovisual cross-perception fusion (ACPF) module that uses multi-head self-attention (MSA) and multi-head cross-attention (MCA) to model inter-modality correlations. In the decoding stage, we employ three multi-level dilated perception (MLDP) modules to aggregate multi-scale spatiotemporal features, where the convolution layers with different dilation rates are cascaded to extend the receptive fields. Experimental results on six challenging audiovisual eye-tracking datasets show that LAAC-Net achieves superior results compared with the evaluated lightweight methods. Moreover, our model has fewer parameters than the fastest competing model, while maintaining comparable running speed.