RUMPL: Ray-Based Transformers for Universal Multi-View 2D to 3D Human Pose Lifting.

Area of Science:

• Computer Vision
• Machine Learning
• Human Pose Estimation

Background:

• Estimating 3D human poses from 2D images is challenging due to occlusions and projective ambiguity.
• Existing multi-view methods struggle with generalization due to scarce, constrained datasets.
• Current 2D-to-3D lifting relies on synthetic data and struggles with arbitrary camera setups.

Purpose of the Study:

• To develop a robust and generalizable 3D human pose estimation method.
• To overcome limitations of existing multi-view and synthetic data-based approaches.
• To introduce a camera-agnostic 3D pose lifting framework.

Main Methods:

• Proposed RUMPL, a transformer-based 3D pose lifter utilizing a novel 3D ray-based representation of 2D keypoints.
• Introduced a View Fusion Transformer to aggregate information along rays for enhanced multi-view consistency.
• Developed a camera-agnostic model deployable across arbitrary camera configurations without retraining.

Main Results:

• RUMPL achieved significant performance improvements on standard benchmarks like Human3.6M and CMU Panoptic.
• Demonstrated a 56.6% MPJPE reduction on Human3.6M compared to triangulation methods.
• Exceeded 70% improvement on CMU Panoptic against transformer-based image-representation approaches.

Conclusions:

• RUMPL offers a robust and scalable solution for 3D human pose estimation from multi-view 2D images.
• The proposed 3D ray representation enables universal deployment across diverse camera setups.
• Validated robustness and scalability on challenging in-the-wild multi-view and multi-person datasets.