CMFAN: Cross-Modal Feature Alignment Network for Few-Shot Single-View 3D Reconstruction

Area of Science:

• Computer Vision
• 3D Computer Graphics
• Machine Learning

Background:

• Few-shot single-view 3D reconstruction aims to generate 3D models from limited data.
• A key challenge is feature misalignment between 2D query images and 3D support shapes.
• Existing methods overlook this cross-modal feature misalignment problem.

Purpose of the Study:

• To propose a novel network, the cross-modal feature alignment network (CMFAN), to address feature misalignment in few-shot 3D reconstruction.
• To introduce effective pretraining and feature fusion strategies for improved cross-modal understanding.

Main Methods:

• Cross-modal contrastive learning (CMCL) for pretraining, aligning global 2D and 3D features of the same objects.
• Cross-modal feature fusion (CMFF) for aligning and fusing local features using cross-attention and feature descriptor concatenation.
• Application of CMFF across multiple feature levels.

Main Results:

• CMFAN effectively aligns global and local cross-modal features, mitigating misalignment issues.
• The proposed CMCL and CMFF techniques significantly enhance 3D reconstruction performance.
• CMFAN achieves new state-of-the-art results on ShapeNet and ModelNet datasets for 1-/10-/25-shot tasks.

Conclusions:

• The proposed CMFAN effectively tackles the feature misalignment problem in few-shot 3D reconstruction.
• CMCL and CMFF are crucial components for achieving superior cross-modal feature alignment and reconstruction accuracy.
• CMFAN represents a significant advancement in few-shot object reconstruction from single images.