Spatiotemporal Bundle Adjustment for Dynamic 3D Human Reconstruction in the Wild

Area of Science:

• Computer Vision
• Robotics
• 3D Reconstruction

Background:

• Traditional bundle adjustment reconstructs static scenes but fails with dynamic points in unsynchronized videos.
• Estimating temporal alignment between cameras is not inherently supported by standard bundle adjustment.

Purpose of the Study:

• To develop a spatiotemporal bundle adjustment framework for jointly optimizing camera parameters, 3D point triangulation, temporal alignment, and dynamic point trajectories.
• To enable accurate 3D motion reconstruction from multiple uncalibrated and unsynchronized video streams.

Main Methods:

• A novel spatiotemporal bundle adjustment framework integrating physics-based motion priors.
• An incremental reconstruction and alignment algorithm with a divide and conquer scheme for efficiency and accuracy.
• Fitting a statistical 3D human body model to asynchronous video streams for enhanced interpretability.

Main Results:

• Accurate 3D motion trajectories of human bodies reconstructed from dynamic events.
• Sub-frame temporal alignment achieved between unsynchronized cameras.
• Significantly improved 3D motion reconstruction at higher temporal resolution compared to input videos.

Conclusions:

• The proposed spatiotemporal bundle adjustment effectively reconstructs dynamic 3D scenes and human motion from challenging video inputs.
• Integration of motion priors and efficient algorithms enables high-accuracy, high-temporal-resolution motion capture.
• The framework offers a robust solution for real-world dynamic event reconstruction using uncalibrated, unsynchronized cameras.