クロスドメイン点群表現学習のためのクロスモダリティ特徴量集約

Existing methods for learning 3D point cloud representation often use a single dataset-specific training and testing approach, leading to performance drops due to significant domain shifts between training and testing data. While recent crossdomain methods have made promising progress, the lack of inherent semantic information in point clouds makes models prone to overfitting specific datasets. As such, we introduce 3D-CFA, a simple yet effective cross-modality feature aggregation method for cross-domain 3D point cloud representation learning. 3D-CFA aggregates the geometry tokens with semantic tokens derived from multi-view images, which are projected from the point cloud, thus generating more transferable features for cross-domain 3D point cloud representation learning. Specifically, 3D-CFA consists of two main components: a cross-modality feature aggregation module and an elastic domain alignment module. The cross-modality feature aggregation module converts unordered points into multi-view images using the modality transformation module. Then, the geometry tokens and semantic tokens extracted from the geometry encoder and semantic encoder are fed into the cross-modal projector to get the transferable 3D tokens. A key insight of this design is that the semantic tokens can serve as a bridge between the 3D point cloud model and the 2D foundation model, greatly promoting the generalization of cross-domain models facing the severe domain shift. Finally, the elastic domain alignment module learns the hierarchical domain-invariant features of different training domains for either domain adaptation or domain generalization protocols. 3D-CFA finds a better way to transfer the knowledge of the 2D foundation model pre-trained at scale, meanwhile only introducing a few extra trainable parameters. Comprehensive experiments on several cross-domain point cloud benchmarks demonstrate the effectiveness of the proposed method compared to the state-of-the-art methods.