Every Angle is Worth a Second Glance: Mining Kinematic Skeletal Structures From Multi-View Joint Cloud.

Area of Science:

• Computer Vision
• Human Motion Analysis
• Machine Learning

Background:

• Multi-person motion capture from sparse observations is difficult due to self- and mutual-occlusions.
• Existing 2D-to-3D lifting methods struggle with accurate joint candidate selection and identity association.
• Robust human pose estimation requires effectively utilizing all available 2D joint information.

Purpose of the Study:

• To develop a novel framework for accurate multi-person 3D motion capture from sparse, occluded 2D observations.
• To address the limitations of existing methods in handling joint ambiguity and identity association.
• To improve the utilization of 2D joint detection data for robust 3D pose estimation.

Main Methods:

• Proposed a Joint Cloud by triangulating all same-typed 2D joints across views, regardless of target ID.
• Introduced the Joint Cloud Selection and Aggregation Transformer (JCSAT) with three cascaded encoders.
• Developed an Optimal Token Attention Path (OTAP) module for feature selection and aggregation.

Main Results:

• The JCSAT framework effectively processes redundant 3D joint candidates from the Joint Cloud.
• The OTAP module successfully selects informative features for accurate human motion prediction.
• Achieved state-of-the-art performance on benchmark and a new challenging dataset (BUMocap-X).

Conclusions:

• The proposed JCSAT framework significantly improves multi-person motion capture accuracy, especially under severe occlusion.
• The Joint Cloud and JCSAT effectively handle ambiguities in 3D pose estimation from sparse views.
• The method demonstrates superior performance compared to existing state-of-the-art techniques.