Towards Fully Mobile 3D Face, Body, and Environment Capture Using Only Head-worn Cameras

Area of Science:

• Computer Vision
• 3D Reconstruction
• Human-Computer Interaction

Background:

• Current 3D reconstruction methods often require multiple cameras or controlled environments.
• Egocentric systems face challenges with occlusions and limited viewpoints for near-body capture.

Purpose of the Study:

• To develop a self-sufficient, head-worn system for dynamic 3D scene reconstruction.
• To enable immersive virtual tours and telepresence using egocentric capture.

Main Methods:

• Utilizing near-body and outward-facing cameras for comprehensive scene capture.
• Employing parametric-model-based approaches with Convolutional Neural Networks (CNNs) for body pose and facial expression estimation.
• Re-targeting a pre-scanned model using intermediate reconstructions for high-fidelity results.

Main Results:

• Successful 3D reconstruction of wearer's movements and surrounding environment in diverse indoor/outdoor settings.
• Demonstrated immersive virtual reality experiences from the captured egocentric data.
• Proof-of-concept for a compact, head-worn system without external sensors.

Conclusions:

• The proposed egocentric system offers a viable solution for self-sufficient 3D reconstruction.
• Potential for future integration into augmented reality glasses for widespread use.
• Enables new possibilities for immersive telepresence, virtual tours, and 3D content creation.