Spherical Vision Transformers for Audio-Visual Saliency Prediction in 360$^{\circ }$∘ Videos

Area of Science:

• Computer Vision
• Virtual Reality
• Human-Computer Interaction

Background:

• Omnidirectional videos (ODVs) offer immersive virtual reality (VR) experiences with a full field-of-view (FOV).
• Predicting visual saliency in 360° environments presents unique challenges due to spherical distortion and the integration of spatial audio.
• Existing datasets lack comprehensive audio-visual data for 360° saliency prediction.

Purpose of the Study:

• To extend saliency prediction to 360° video environments by addressing spherical distortion and spatial audio integration.
• To develop and evaluate novel models for audio-visual saliency prediction in ODVs.
• To introduce a new dataset, YT360-EyeTracking, for training and evaluating 360° saliency prediction models.

Main Methods:

• Curated the YT360-EyeTracking dataset comprising 81 ODVs with varying audio-visual conditions.
• Proposed SalViT360, a vision-transformer model with spherical geometry-aware attention for ODVs.
• Developed SalViT360-AV, an extension incorporating transformer adapters conditioned on audio input.

Main Results:

• SalViT360 and SalViT360-AV significantly outperform existing methods on benchmark datasets, including YT360-EyeTracking.
• Demonstrated the effectiveness of incorporating spatial audio cues for enhanced saliency prediction accuracy.
• Validated the models' ability to predict viewer attention in complex 360° scenes.

Conclusions:

• Integrating spatial audio is crucial for accurate saliency prediction in omnidirectional videos.
• The proposed SalViT360 and SalViT360-AV models represent significant advancements in 360° visual attention prediction.
• The YT360-EyeTracking dataset facilitates further research in audio-visual saliency for immersive media.