OFERA: Blendshape-Driven 3D Gaussian Control for Occluded Facial Expression to Realistic Avatars in VR.

Area of Science:

• Computer Graphics
• Virtual Reality
• Human-Computer Interaction

Background:

• Existing methods for controlling facial expressions in VR avatars face limitations like increased device weight, discomfort, privacy concerns, and restricted data access.
• Leveraging blendshape signals from commercial VR headsets offers a promising alternative for expression input.

Purpose of the Study:

• To introduce OFERA, a novel framework for real-time expression control of photorealistic Gaussian head avatars for VR users.
• To overcome the limitations of sensor-based and camera-based expression recovery methods.

Main Methods:

• OFERA utilizes blendshape signals from VR headsets as expression inputs.
• Key components include Blendshape Distribution Alignment (BDA) for input space alignment, an Expression Parameter Mapper (EPM) for mapping blendshapes to avatar control, and Mapper-integrated Avatar (MiA) for distributional consistency.
• An end-to-end pipeline integrates sensing, mapping, avatar updating, and real-time rendering within VR.

Main Results:

• The Expression Parameter Mapper (EPM) component demonstrates superior performance over existing mapping methods based on quantitative metrics.
• User studies confirm that the OFERA framework improves expression fidelity and maintains avatar realism.
• OFERA enables real-time, photorealistic avatar expression control, significantly enhancing telepresence in VR communication.

Conclusions:

• OFERA provides an effective solution for real-time expression control of Gaussian head avatars in VR.
• The framework enhances user experience in virtual communication by improving avatar expressiveness and realism.
• This research advances the field of VR avatars and telepresence by offering a practical and privacy-preserving approach to expression control.