A Multimodal Saliency Model for Videos with High Audio-Visual Correspondence

Area of Science:

• Computer Science
• Cognitive Science
• Signal Processing

Background:

• Most visual attention prediction models overlook audio information, despite its known influence on human visual attention.
• Psychological studies confirm that sound significantly impacts visual attention and can aid in localizing sound sources.

Purpose of the Study:

• To develop a novel multi-modal saliency (MMS) model for videos with high audio-visual correspondence.
• To integrate audio cues into video attention prediction, addressing limitations of current visual-only models.

Main Methods:

• Utilized a free energy principle for visual spatial and temporal saliency map detection.
• Introduced cross-modal kernel canonical correlation analysis for audio saliency map detection by localizing moving-sounding objects.
• Developed a two-stage adaptive audiovisual saliency fusion method to combine visual and audio saliency maps.

Main Results:

• The proposed MMS model effectively captures the influence of audio, a factor absent in many deep learning saliency models.
• Combining deep saliency models with the MMS model via late fusion yielded an average 5% performance gain.
• Models incorporating audio cues demonstrated significant superiority over state-of-the-art visual-only saliency models.

Conclusions:

• Audio information plays a crucial role in visual attention prediction for audio-visual scenes.
• The novel MMS model and its fusion with deep learning approaches offer a substantial advancement in video saliency prediction.
• Future research should continue exploring multi-modal approaches for more comprehensive attention modeling.