TempoCross: instance-aware sparse representation for multimodal temporal fusion in 3D detection

Area of Science:

• Computer Vision
• Autonomous Driving Systems
• Sensor Fusion

Background:

• Rising performance demands in autonomous driving perception necessitate advanced techniques.
• Bird's-eye-view (BEV) perception integrating sensor data and temporal information is a key research area.
• Existing methods often focus on either BEV-level feature aggregation or sparse object-centric temporal modeling, with limited integration for LiDAR-camera detection.

Purpose of the Study:

• To develop a novel 3D detection method for LiDAR-camera systems that effectively integrates multimodal temporal fusion.
• To address the challenge of preserving modality-specific temporal states during fusion.
• To improve the accuracy and robustness of autonomous driving perception.

Main Methods:

• Introduced TempoCross, a 3D detection method utilizing instance-aware sparse representations for multimodal temporal fusion.
• Encoded features from different timestamps and sensor modalities into a unified instance space.
• Implemented a motion compensation module with hybrid motion modeling (rigid-body transformations + learnable deformation residuals) and a temporal-aware enhancement step using lightweight cross-attention.

Main Results:

• TempoCross achieved 74.1% mAP and 75.7% NDS on the nuScenes test set.
• The method outperformed mainstream baseline detectors in LiDAR-camera 3D detection.
• Demonstrated the effectiveness of combining LiDAR-camera fusion with instance-aware sparse temporal modeling.

Conclusions:

• TempoCross successfully integrates multimodal sensor data and temporal information for enhanced 3D object detection.
• The instance-aware sparse representation and hybrid motion modeling contribute to improved accuracy and robustness.
• The proposed approach offers a promising direction for advancing autonomous driving perception systems.