SODFormer: Streaming Object Detection With Transformer Using Events and Frames.

Area of Science:

• Computer Vision
• Neuromorphic Engineering
• Deep Learning

Background:

• Object detection faces challenges with fast motion blur and low-light conditions.
• Existing methods struggle to effectively fuse asynchronous event data and traditional frame data.
• Leveraging temporal cues from multimodal sensing is crucial for robust object detection.

Purpose of the Study:

• To propose SODFormer, a novel streaming object detector integrating event and frame data for asynchronous object detection.
• To develop a spatiotemporal Transformer architecture capable of leveraging rich temporal cues from heterogeneous visual streams.
• To create a large-scale multimodal dataset for neuromorphic object detection.

Main Methods:

• Developed SODFormer, a Transformer-based object detector for continuous, asynchronous detection using event and frame data.
• Constructed the large-scale PKU-DAVIS-SOD dataset with over 1 million manual labels.
• Designed a spatiotemporal Transformer with a novel temporal module and an asynchronous attention-based fusion module.

Main Results:

• SODFormer significantly outperforms four state-of-the-art methods and eight baseline methods.
• The framework demonstrates superior performance in high-speed motion and low-light scenarios where conventional methods fail.
• The asynchronous fusion module effectively integrates heterogeneous sensing modalities for enhanced detection.

Conclusions:

• SODFormer offers an effective solution for asynchronous object detection by fusing event and frame data.
• The proposed method achieves state-of-the-art performance, particularly in challenging environmental conditions.
• The developed dataset and framework advance research in multimodal neuromorphic object detection.