Instant Reality: Gaze-Contingent Perceptual Optimization for 3D Virtual Reality Streaming

Area of Science:

• Computer Science
• Human-Computer Interaction
• Multimedia Systems

Background:

• Edge-cloud computing is used for media streaming in applications like entertainment and design.
• Virtual reality (VR) applications require 3D asset streaming with low latency and high quality for interactive experiences.
• Existing 3D asset streaming solutions struggle with large data sizes and latency, leading to long loading times or reduced quality.

Purpose of the Study:

• To propose a perceptually-optimized progressive 3D streaming method for spatial quality and temporal consistency in immersive interactions.
• To address the challenges of large data sizes and low latency in 3D asset streaming for VR.
• To enhance user experience in VR by improving rendering quality and reducing interaction delays.

Main Methods:

• Estimating perceptual importance in 2D image space using user gaze behaviors (viewing focus and eye movement).
• Mapping estimated importance to 3D object space to prioritize streaming for edge-side rendering.
• Developing a neural network to accelerate the cloud-side scheduling process for efficiency.

Main Results:

• Demonstrated improved visual quality and temporal consistency compared to alternative solutions.
• Evaluated performance under various network conditions (3G to 5G) and edge devices (HMD, traditional displays).
• Subjective studies and objective analysis confirmed the method's effectiveness.

Conclusions:

• The proposed perceptually-optimized progressive 3D streaming method effectively enhances spatial quality and temporal consistency in immersive VR interactions.
• Gaze-based prioritization and accelerated cloud-side scheduling are key to overcoming 3D asset streaming challenges.
• This approach offers a significant improvement for VR applications requiring interactive 3D content delivery.