Sparse4D: Sparse-Based End-to-End Multi-Sensor Temporal Perception

Area of Science:

• Computer Vision
• Autonomous Driving Systems
• Machine Learning

Background:

• Perception model design is critical for autonomous driving.
• Existing Bird's-Eye View (BEV) algorithms have limitations in efficiency and performance.
• Sparse perception offers a promising alternative for enhancing autonomous driving capabilities.

Purpose of the Study:

• To propose a novel end-to-end sparse perception framework, Sparse4D, for autonomous driving.
• To enhance both performance and computational efficiency compared to mainstream BEV-based algorithms.
• To validate the framework's effectiveness in 3D detection and multi-object tracking.

Main Methods:

• Developed Sparse4D, an end-to-end sparse perception framework.
• Defined a novel 'instance' concept decoupling implicit features and explicit anchors for core feature fusion.
• Introduced 'deformable aggregation' for spatial modeling and a recurrent structure for efficient temporal modeling.
• Proposed a minimalist joint detection and tracking model based on Sparse4D.

Main Results:

• Sparse4D achieved state-of-the-art performance on the nuScenes benchmark for multi-camera 3D detection and tracking.
• Demonstrated superior training and inference efficiency compared to other algorithms.
• Extended Sparse4D to a multi-modal model, achieving excellent performance and generalization.

Conclusions:

• Sparse4D offers a highly efficient and performant sparse perception framework for autonomous driving.
• The proposed instance-centric approach and novel modeling techniques advance the field.
• Sparse4D shows strong potential for real-world deployment and further multi-modal extensions.