3D face-model reconstruction from a single image: A feature aggregation approach using hierarchical transformer with...

Area of Science:

• Computer Vision
• Machine Learning
• 3D Reconstruction

Background:

• Convolutional Neural Networks (CNNs) excel in computer vision but struggle with long-range image dependencies.
• Transformer networks, adept at long-range dependencies, achieve state-of-the-art results in various vision tasks.
• Transformers have not been extensively applied to 3D face reconstruction.

Purpose of the Study:

• To propose a novel hierarchical transformer model for extracting 3D face parameters from a single 2D image.
• To address the limitations of existing models in capturing long-range facial features.
• To achieve state-of-the-art performance in 3D face reconstruction and alignment.

Main Methods:

• A hierarchical transformer model utilizing pre-trained Swin Transformer backbones.
• A feature pyramid aggregation structure for multi-stage feature fusion.
• A semi-supervised training approach combining supervised learning (3DMM parameters) and unsupervised learning (differential rendering).

Main Results:

• The proposed model achieves comparable or superior results to current state-of-the-art methods on two public datasets.
• Demonstrates improved performance in 3D face reconstruction and dense 3D face alignment.
• Subjective evaluations indicate enhanced realism and occlusion resistance compared to previous works.

Conclusions:

• The novel hierarchical transformer model effectively extracts 3D face parameters from single images.
• The approach overcomes CNN limitations by leveraging long-range dependency extraction.
• The method shows significant potential for advancing 3D face reconstruction technology.