Multi-View Hand Reconstruction With a Point-Embedded Transformer.

Area of Science:

• Computer Vision
• 3D Reconstruction
• Human-Computer Interaction

Background:

• Accurate 3D hand mesh reconstruction is crucial for realistic motion capture.
• Existing methods often struggle with generalizability and real-world applicability.
• Multi-view stereo approaches offer potential but require effective feature fusion.

Purpose of the Study:

• Introduce POEM, a novel and generalizable multi-view hand mesh reconstruction (HMR) model.
• Enable practical, user-friendly, and cost-effective multi-view motion capture for hands.
• Improve the fusion of multi-view features for robust 3D hand modeling.

Main Methods:

• Propose embedding static basis points within the multi-view stereo space to represent 3D hand geometry.
• Utilize basis points to fuse features across multiple camera views effectively.
• Employ a training strategy combining five large-scale multi-view datasets with randomized camera configurations.

Main Results:

• The POEM model demonstrates notable generalizability in real-world applications.
• Achieved effective fusion of multi-view image features using 3D basis points.
• Developed a practical, plug-and-play solution for both left and right hand motion capture.

Conclusions:

• POEM offers a novel and effective approach to multi-view hand mesh reconstruction.
• The method provides a user-friendly and cost-effective solution for practical motion capture.
• The generalizability of POEM makes it suitable for diverse real-world scenarios.