Audio-Visual Aware Foveated Rendering.

Area of Science:

• Computer Graphics
• Human-Computer Interaction
• Virtual Reality

Background:

• Virtual reality (VR) rendering demands high frame rates, challenging existing foveated rendering methods.
• Current foveated rendering ignores auditory content's impact on human visual perception.
• Auditory stimuli significantly influence visual perception in the human visual system (HVS).

Purpose of the Study:

• To quantify the impact of auditory conditions on visual perception in foveated rendering.
• To develop an audio-visual aware foveated rendering method (AvFR) for enhanced VR experiences.
• To improve VR rendering efficiency without compromising perceived visual quality.

Main Methods:

• Developed an auditory-content-based perceived rendering quality analysis.
• Constructed an audio-visual feature-driven perception model integrating scene audio-visual content.
• Implemented an adaptive foveated rendering cost optimization algorithm guided by the perception model.

Main Results:

• AvFR achieved an average frame rate of 116 FPS in complex VR scenes.
• Demonstrated up to 1.4x performance improvement compared to state-of-the-art methods.
• Maintained high perceived visual quality without lowering visual fidelity.

Conclusions:

• Audio-visual integration in foveated rendering significantly boosts VR performance.
• AvFR offers a novel approach to optimize VR rendering by incorporating auditory cues.
• The proposed method enhances user experience in complex VR environments.