Perceptually-Aligned Dynamic Facial Projection Mapping by High-Speed Face-Tracking Method and Lens-Shift Co-Axial...

Area of Science:

• Computer Vision
• Graphics and Rendering
• Human-Computer Interaction

Background:

• Dynamic Facial Projection Mapping (DFPM) creates immersive visual experiences by overlaying digital content onto faces.
• Misalignment artifacts between projected images and facial features are a persistent challenge in DFPM applications.
• Existing methods struggle with real-time accuracy and robust alignment, limiting DFPM's potential in industries like entertainment and cosmetics.

Purpose of the Study:

• To propose and evaluate novel methods for reducing misalignment in Dynamic Facial Projection Mapping.
• To develop a high-speed face-tracking system leveraging temporal information for improved accuracy.
• To design a projector-camera setup that enhances optical alignment and minimizes projection errors.

Main Methods:

• Developed a cropped-area-limited, inter/extrapolation-based face detection framework for parallel processing with facial landmark detection.
• Introduced a hybrid facial landmark detection method combining fast Ensemble of Regression Trees (ERT) with auxiliary detection for error recovery.
• Created a novel simulation technique for high-frame-rate video annotations to train the landmark detection model.
• Designed a lens-shift co-axial projector-camera setup ensuring high optical alignment with minimal pixel error across varying depths.

Main Results:

• The proposed face-tracking method achieves rapid detection (0.107 ms) by utilizing temporal information and auxiliary detection.
• The simulated high-frame-rate dataset enabled effective training of the facial landmark detection model.
• The co-axial projector-camera setup maintained sub-pixel alignment error (1.274 pixels) between 1m and 2m.
• The integrated high-speed DFPM system demonstrated nearly perfect alignment, enhancing visual perception.

Conclusions:

• The developed high-speed DFPM system effectively addresses misalignment challenges through advanced face-tracking and optical design.
• The innovations enable more immersive and visually accurate facial projection mapping experiences.
• This research advances DFPM technology for broader applications in entertainment, makeup, and interactive media.