Saliency-Guided Foveated Video Encoding for Low-Latency and Immersive Cloud VR

Area of Science:

• Computer Vision
• Virtual Reality
• Video Encoding

Background:

• Cloud virtual reality (VR) faces challenges in delivering high visual quality under limited wireless bandwidth.
• Foveated video encoding (FVE) uses gaze position to reduce bandwidth but is content-independent, leading to suboptimal quality for visual saliency.
• Existing FVE methods fail to model human attention effectively, causing artifacts and reduced perceptual quality.

Purpose of the Study:

• To develop a novel AI-driven streaming framework for cloud VR that incorporates visual saliency cues into video encoding.
• To improve perceptual visual quality and reduce bandwidth consumption in immersive VR streaming.
• To address the limitations of content-independent FVE methods.

Main Methods:

• Proposed a lightweight deep neural network for efficient saliency inference, reducing computational complexity by 48×.
• Integrated a saliency-guided FVE pipeline into an open-source cloud VR gaming platform.
• Conducted comprehensive experiments and an IRB-approved user study for evaluation.

Main Results:

• The AI-driven FVE approach enhanced perceptual visual quality by 22.98% compared to state-of-the-art (SoTA) systems.
• Achieved superior visual quality and spatial smoothness.
• Significantly reduced noticeable artifacts compared to gaze-exclusive FVE.

Conclusions:

• The proposed saliency-guided FVE pipeline effectively improves visual quality and user experience in cloud VR streaming.
• The lightweight saliency inference network meets low-latency demands for immersive VR.
• This AI-driven approach offers a promising solution for bandwidth-constrained VR delivery.