PointVST: Self-Supervised Pre-Training for 3D Point Clouds via View-Specific Point-to-Image Translation

Area of Science:

• Computer Vision
• Machine Learning
• 3D Data Analysis

Background:

• Self-supervised representation learning has advanced significantly in language and 2D vision.
• These advancements have not been fully adopted in 3D point cloud learning.
• Existing pre-training methods for 3D point clouds often rely on generative or contrastive learning.

Purpose of the Study:

• To propose a novel translative pre-training framework, PointVST, for 3D point cloud learning.
• To introduce a self-supervised pretext task involving cross-modal translation from 3D point clouds to 2D images.
• To enhance feature extraction for 3D point cloud analysis.

Main Methods:

• Developed PointVST, a translative pre-training framework.
• Implemented a self-supervised pretext task: cross-modal translation from 3D point clouds to 2D images.
• Incorporated view-conditioned point-wise embeddings and adaptive aggregation of view-specific global codewords for image generation.

Main Results:

• PointVST demonstrated consistent and prominent performance superiority over state-of-the-art approaches.
• The framework showed satisfactory domain transfer capability across various downstream tasks.
• Experimental evaluations validated the effectiveness of the proposed method.

Conclusions:

• PointVST offers a novel and effective self-supervised approach for 3D point cloud representation learning.
• The cross-modal translation strategy significantly improves performance on downstream tasks.
• The framework exhibits strong generalization and transferability.