Clustering-based Supervised Training for 3D Perception

Point cloud perception poses a formidable challenge, because of not only the irregular geometries of many millions of unordered points, but also the great variations caused by depth, viewpoint, occlusion, etc. Current studies put much focus on various fancy architectural designs for learning the complex geometries of point clouds, but often place less emphasis on explicitly organizing intra-class variations in the embedding space. This raises a fundamental question: how to learn an appropriate embedding space that is aware of both discriminative semantics and challenging variations? As a response, we propose a clustering based supervised training scheme for 3D perception. Unlike current de-facto, scene-wise training paradigm, our algorithm conducts within-class clustering on the embedding space for automatically discovering subclass patterns that are latent yet representative across scenes. The mined patterns are, in turn, used to refine the corresponding embedding space, so as to respect the underlying structure of the entire training dataset and improve the robustness of variations. To pursue a more robust model with better transferability, we construct separate embedding spaces and conduct the clustering based supervised training scheme within each space. This approach cultivates robust embedding spaces that accommodate the diverse intra-class variations present within complex point cloud scenarios. Our algorithm is principled and readily pluggable to modern point cloud perception networks during training, without extra overhead during testing. With various 3D network architectures (ie, voxel-based, point-based, Transformer-based, automatically searched), our algorithm shows impressive improvements on famous point cloud segmentation datasets (ie, 2.1-3.2% on single-scan and 1.8-2.7% on multi-scan of SemanticKITTI, 1.9-2.4% on S3DIS, 1.6-2.5% on ScanNet v2, 2.3-2.7% on NuScenes, in terms of mIoU). Our algorithm even demonstrates utility in instance embedding space and 3D detection, showing 2.6-3.9% mAP gains on KITTI. Moreover, a larger gain of transfer learning (1.7% vs. 0.5%) in the coarse-to-fine transfer learning is observed for our algorithm, verifying its transferability and superiority.