Leveraging two-dimensional pre-trained vision transformers for three-dimensional model generation via masked...

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Transformers are standard in NLP but underutilized in computer vision due to domain differences.
• Self-supervised learning, particularly masked autoencoding, shows promise for both NLP and computer vision.
• Limited 3D datasets hinder the development of high-quality 3D feature learning models.

Purpose of the Study:

• To develop a multi-scale masked autoencoding (MAE) prior training architecture for self-supervised learning on 3D point clouds.
• To leverage pre-trained 2D vision models to guide the 3D feature learning process.
• To overcome the challenge of limited 3D data by effectively utilizing 2D image representations.

Main Methods:

• Utilized a pre-trained Vision Transformer (ViT) and a 3D representation model trained on 2D images.
• Developed a 3D masking-based autoencoder with an encoder-decoder architecture for self-supervised pre-training.
• Employed a novel 2D masking strategy to preserve semantically important point tokens during pre-training.

Main Results:

• The proposed method achieved 93.63% accuracy on ScanObjectNN and 91.31% accuracy on ModelNet40 using linear SVM.
• Demonstrated effective generalization across various downstream tasks.
• Showcased that a conventional transformer-based architecture can outperform specialized supervised models.

Conclusions:

• The multi-scale MAE prior training architecture effectively enables self-supervised learning on 3D point clouds using 2D pre-trained models.
• The approach successfully transfers knowledge from 2D vision to 3D representation learning, addressing data scarcity.
• This work highlights the potential of straightforward transformer architectures for high-performance 3D computer vision tasks.