Robust 3D Semantic Occupancy Prediction With Calibration-Free Spatial Transformation

3D semantic occupancy prediction, which seeks to provide accurate and comprehensive representations of environment scenes, is important to autonomous driving systems. For autonomous cars equipped with multi-camera and LiDAR, it is critical to aggregate multi-sensor information into a unified 3D space for accurate and robust predictions. Recent methods are mainly built on the 2D-to-3D transformation that relies on sensor calibration to project the 2D image information into the 3D space. These methods, however, suffer from two major limitations: First, they rely on accurate sensor calibration and are sensitive to the calibration noise, which limits their application in real complex environments. Second, the spatial transformation layers are computationally expensive and limit their running on an autonomous vehicle. In this work, we attempt to exploit a Robust and Efficient 3D semantic Occupancy (REO) prediction scheme. To this end, we propose a calibration-free spatial transformation based on vanilla attention to implicitly model the 2D-to-3D spatial correspondence. In this way, we robustly project the 2D features to a predefined BEV plane without using sensor calibration as input. Then, we introduce a multi-task training framework to ease the learning of spatial correspondence without explicit spatial priors and enhance the discrimination power of 2D backbones on spatial, semantic, and texture features. Last, we propose a query-based prediction scheme to efficiently generate large-scale fine-grained occupancy predictions. Extensive experiments on three benchmarks (i.e., OpenOccupancy, Occ3D-nuScenes, and SemanticKITTI Scene Completion) demonstrate the superiority of our REO. For instance, our REO achieves 19.8× speedup compared to Co-Occ, with 1.1% improvements in geometry IoU on OpenOccupancy. Our code is available at https://github.com/ICEORY/REO.