EventEgo3D++: 3D Human Motion Capture from A Head-Mounted Event Camera

Area of Science:

• Computer Vision
• Robotics
• Human-Computer Interaction

Background:

• Monocular egocentric 3D human motion capture is difficult, especially in low light and with fast movements.
• Existing RGB camera methods fail in these challenging conditions common in head-mounted device applications.

Purpose of the Study:

• Introduce EventEgo3D++, the first method for 3D human motion capture using a monocular event camera with a fisheye lens.
• Address limitations of current methods in challenging environments.

Main Methods:

• Utilize event camera data with LNES representation for precise 3D reconstructions.
• Develop a mobile head-mounted device (HMD) prototype with an event camera.
• Capture real and synthetic datasets, including allocentric RGB streams and SMPL body models.

Main Results:

• EventEgo3D++ demonstrates superior 3D accuracy and robustness compared to existing solutions.
• Achieve real-time 3D pose updates at 140Hz.
• Validated in controlled studio and in-the-wild settings.

Conclusions:

• EventEgo3D++ advances the state of the art in egocentric 3D human motion capture.
• Event cameras offer reliable cues for accurate motion capture in challenging scenarios.
• The developed dataset and method provide a comprehensive resource for future research.