HFM-GS: Half-Face Mapping 3DGS Avatar Based Real-Time HMD Removal

Area of Science:

• Computer Vision
• Extended Reality (XR)
• Computer Graphics

Background:

• Enhancing user perception in extended reality (XR) often requires head-mounted display (HMD) removal.
• Current methods for HMD removal in XR exhibit limitations in both processing speed and reconstruction accuracy.

Purpose of the Study:

• To develop a real-time, high-fidelity method for restoring faces occluded by HMDs in XR videos.
• To improve the adaptability to facial deformations and enhance processing speed for HMD removal.

Main Methods:

• Proposed a half face mapping 3D Gaussian splatting avatar (HFM-GS) method for HMD removal.
• Established a mapping field between upper and lower face Gaussians for deformation adaptability.
• Implemented correlation weight-based sampling for improved time performance and Gaussian number variation handling.
• Utilized a Gaussian Segregation Strategy to ensure model robustness.

Main Results:

• The HFM-GS method demonstrated superior quality and time performance compared to two state-of-the-art approaches.
• Achieved real-time and high-fidelity online restoration of complete faces in HMD-occluded videos.
• User studies confirmed a significant improvement in visual fidelity.

Conclusions:

• The HFM-GS method offers an effective solution for real-time, high-fidelity face restoration in XR applications.
• This approach significantly enhances user perception by overcoming HMD occlusions.
• The method provides a robust and efficient alternative to existing HMD removal techniques.