Estimating a 3D Human Skeleton from a Single RGB Image by Fusing Predicted Depths from Multiple Virtual Viewpoints

Area of Science:

• Computer Vision
• Human Pose Estimation
• 3D Reconstruction

Background:

• Estimating 3D human skeletons from single RGB images is a significant challenge in computer vision.
• Multi-view approaches offer advantages in accuracy but require multiple cameras.

Purpose of the Study:

• To develop a single-view method for accurate 3D human skeleton estimation.
• To leverage virtual viewpoints to enhance depth perception and skeleton accuracy.

Main Methods:

• A two-stage network utilizing a two-stream approach (Real-Net and Virtual-Net) to predict 2D coordinates and relative depths from real and virtual viewpoints.
• Integration of a depth-denoising module, cropped-to-original coordinate transform (COCT), and a fusion module for 2D-to-3D lifting and regression.

Main Results:

• Achieved an average per-joint position error of 45.7 mm, outperforming existing single-view methods.
• Performance is comparable to sequence-based methods that utilize multiple consecutive frames.

Conclusions:

• The proposed single-view method effectively reconstructs accurate 3D human skeletons by fusing information from multiple virtual viewpoints.
• This approach offers a promising alternative to multi-view or sequence-based methods for 3D pose estimation.