UNOC: Understanding Occlusion for Embodied Presence in Virtual Reality.

Area of Science:

• Computer Vision and Machine Learning
• Human-Computer Interaction
• Robotics

Background:

• Accurate 3D body and hand motion tracking is crucial for social and self-presence in augmented and virtual reality (AR/VR).
• Egocentric tracking, using embodied perception, is preferred over traditional 3D pose estimation for AR/VR applications.
• Optimization-based methods for egocentric tracking struggle with frequent occlusions, limiting their effectiveness.

Purpose of the Study:

• To propose a novel data-driven framework for robust egocentric body tracking.
• To address the challenge of omnipresent occlusions in real-time tracking scenarios.
• To enhance the fidelity of embodied poses in augmented and virtual environments.

Main Methods:

• Collected a large-scale motion capture dataset including body and finger motions using optical markers and inertial sensors.
• Focused dataset collection on social scenarios, capturing ground truth poses under self-occlusions and body-hand interactions.
• Simulated head-mounted camera view occlusions using ray casting and trained a deep neural network (DNN) to infer missing body parts.

Main Results:

• The proposed deep neural network effectively infers occluded body parts from simulated egocentric views.
• The framework achieves high-fidelity embodied pose generation in experiments.
• Successful application demonstrated in real-time egocentric body tracking, finger motion synthesis, and 3-point inverse kinematics.

Conclusions:

• The developed data-driven framework significantly improves egocentric body tracking by handling occlusions.
• The method offers a viable solution for enhancing presence and interaction in AR/VR.
• Future work can leverage this approach for more immersive and realistic virtual experiences.