Nighttime traffic object detection via adaptively integrating event and frame domains

Area of Science:

• Computer Vision
• Intelligent Transportation Systems (ITS)
• Sensor Fusion

Background:

• Traditional RGB cameras struggle with nighttime traffic scenes due to limited dynamic range.
• Event cameras offer superior performance in low-light conditions owing to their high dynamic range.
• Multimodal learning shows promise in vision tasks, motivating combined sensor approaches.

Purpose of the Study:

• To develop an adaptive method for joint object detection using both event and RGB camera data for nighttime traffic.
• To address challenges in fusing data from different sensor modalities effectively.
• To improve the robustness and accuracy of nighttime traffic object detection.

Main Methods:

• Proposed an adaptive selection and fusion module for efficient multimodal data fusion via channel-wise feature ranking.
• Developed a multi-level feature pyramid network incorporating multimodal attention fusion.
• Curated a novel dataset of nighttime traffic scenes with synchronized RGB frames and event streams.

Main Results:

• The proposed method significantly outperforms existing state-of-the-art techniques in event-based, frame-based, and fusion-based detection.
• Demonstrated superior performance in detecting nighttime traffic objects compared to single-modality approaches.
• Validated the effectiveness of the adaptive fusion strategy and multimodal attention mechanism.

Conclusions:

• Integrating event and RGB camera data through adaptive fusion substantially enhances nighttime traffic object detection.
• The developed method offers a robust solution for intelligent transportation systems operating in challenging low-light conditions.
• This work highlights the potential of multimodal sensing for improving traffic safety and efficiency.