AnchorSED: A Segmentation Enhanced Detection Framework Based on Sparse Anchor for Multi-View 3D Object Detection

Existing 3D detection methods typically exploit 2D image information through region-level feature extraction and auxiliary semantic supervision. However, these strategies do not explicitly establish a correspondence between 3D candidates and corresponding 2D instance-level representations, which limits the ability to handle objects with complex shapes or severe occlusions. To address this limitation, we propose an anchorbased framework that establishes an anchor-conditioned 2D-3D interaction for jointly exploiting 2D and 3D annotations. Specifically, each 3D anchor and its sampled points are projected onto multiple image planes to identify anchor-associated regions of interest (ROIs), from which view-specific instance segmentation masks are predicted. These 2D representations are subsequently mapped back to their corresponding 3D anchors and aggregated through geometry-aware cross-modal feature alignment. Our approach extends conventional sparse detectors by introducing three components: (1) multi-view ROIs prediction, which maps 3D anchors and sampled points onto multi-view image planes to predict regions of interest (ROIs); (2) multi-view instance segmentation, generating pixel-accurate masks corresponding to predicted ROIs; and (3) cross-modal feature alignment, which fuses geometrically consistent 2D segmentation features with their associated 3D anchors. Furthermore, to bridge the geometric discrepancy between 2D images and 3D objects, we distill multi-scale visual knowledge from a pretrained large-scale self-supervised vision model into our detector, enhancing geometric consistency. Evaluated on the nuScenes benchmark, our framework achieves state-of-the-art performance while maintaining computational efficiency. The code and auxiliary supervision data are available at https://github.com/HuiAuto/AnchorSED.