Visibility Constrained Generative Model for Depth-Based 3D Facial Pose Tracking.

Area of Science:

• Computer Vision
• 3D Reconstruction
• Machine Learning

Background:

• Accurate 3D facial pose tracking is crucial for unconstrained environments.
• Existing methods struggle with occlusions and dynamic facial expressions.
• Real-time face model adaptation remains a significant challenge.

Purpose of the Study:

• To develop a unified generative framework for simultaneous 3D facial pose tracking and on-the-fly face model adaptation.
• To enhance robustness against heavy occlusions and arbitrary facial expression variations.
• To improve the adaptability of face models for changing subjects in real-time.

Main Methods:

• Introduced a statistical 3D morphable model for flexible face surface description.
• Implemented an efficient switchable online adaptation for identity capture and model construction.
• Proposed a novel ray visibility constraint to improve pose estimation robustness against occlusions, moving beyond ICP-based methods.

Main Results:

• The proposed framework demonstrates effective unification of pose tracking and model adaptation.
• Experimental results on Biwi and ICT-3DHP datasets show superior performance compared to state-of-the-art depth-based methods.
• The ray visibility constraint significantly enhances robustness to occlusions.

Conclusions:

• The developed generative framework offers a robust solution for 3D facial pose tracking and adaptation in challenging, unconstrained scenarios.
• The novel approach outperforms existing methods, particularly in the presence of occlusions and expression variations.
• This work advances the field of real-time 3D facial analysis.