DiffCap: Diffusion-Based Real-Time Human Motion Capture Using Sparse IMUs and a Monocular Camera.

Area of Science:

• Robotics
• Computer Vision
• Human-Computer Interaction

Background:

• Human motion capture is crucial for various applications.
• Existing methods often struggle with real-time performance and data fusion challenges.
• Integrating sparse Inertial Measurement Units (IMUs) and monocular cameras offers a promising yet complex approach.

Purpose of the Study:

• To develop a unified framework for fusing IMU and monocular camera data for human motion capture.
• To leverage diffusion models for learning human motion priors and handling multi-modal sensor data.
• To achieve robust and accurate real-time human pose estimation.

Main Methods:

• A diffusion-based solution is proposed to learn human motion priors.
• Sequential visual information is processed as a whole into a condition embedding to handle occlusions.
• Inertial measurement data is concatenated frame-by-frame with noisy pose data for the diffusion model.

Main Results:

• The proposed method demonstrates state-of-the-art performance in pose estimation.
• Experimental results validate the effectiveness of the system design in fusing sparse IMUs and a monocular camera.
• The approach shows robustness to occasional degenerations in visual information.

Conclusions:

• The diffusion-based framework effectively fuses IMU and monocular camera data for human motion capture.
• The method achieves superior accuracy and robustness compared to previous approaches.
• This work advances the field of real-time human motion analysis.