Deep Non-Rigid Structure-From-Motion: A Sequence-to-Sequence Translation Perspective.

Area of Science:

• Computer Vision
• Machine Learning
• 3D Reconstruction

Background:

• Traditional Non-Rigid Structure-from-Motion (NRSfM) methods often process 2D frames individually, neglecting the inherent spatial-temporal nature of the problem.
• Frame-by-frame 3D reconstruction pipelines overlook the sequential data structure crucial for accurate NRSfM.

Purpose of the Study:

• To develop a self-supervised, sequence-to-sequence deep learning framework for Non-Rigid Structure-from-Motion (NRSfM).
• To address the limitations of frame-by-frame reconstruction by treating the 2D keypoint sequence as a whole for 3D reconstruction.

Main Methods:

• A sequence-to-sequence translation perspective is employed, utilizing a shape-motion predictor for initial 3D shape and motion estimation.
• Introduction of a novel Context Layer with multi-head attention (MHA) and temporal encoding to impose self-expressiveness regularity and sequence-level constraints.
• Self-supervised learning approach reconstructs 3D keypoint sequences from input 2D keypoint sequences.

Main Results:

• The proposed framework demonstrates superior performance in 3D reconstruction for Non-Rigid Structure-from-Motion (NRSfM).
• Experimental validation across diverse datasets including Human3.6M, CMU Mocap, and InterHand confirms the effectiveness of the approach.
• The method successfully reconstructs 3D sequences by leveraging spatial-temporal information and deep learning constraints.

Conclusions:

• The sequence-to-sequence deep learning approach offers a more effective solution for NRSfM compared to traditional frame-by-frame methods.
• The Context Layer effectively integrates structural characteristics and temporal dynamics for improved 3D reconstruction accuracy.
• The self-supervised framework provides a robust method for reconstructing non-rigid 3D structures from monocular 2D sequences.