RGB-FIR Multimodal Pedestrian Detection with Cross-Modality Context Attentional Model.

Area of Science:

• Computer Vision
• Autonomous Driving Systems
• Machine Learning

Background:

• Pedestrian detection is crucial for visual cognition and autonomous driving.
• Existing multimodal YOLO models focus on back-end fusion, potentially missing front-end feature extraction optimizations.
• Current methods overlook the complementarity between RGB and FIR modalities and across different feature scales.

Purpose of the Study:

• To propose a novel RGB-FIR multimodal backbone network framework using a cross-modality context attentional model (CCAM).
• To enhance pedestrian detection by optimizing feature fusion at both front-end and back-end stages.
• To improve the accuracy and robustness of pedestrian detection systems.

Main Methods:

• Developed a multi-level fusion framework incorporating a CCAM at the front-end of parallel RGB-FIR backbone networks.
• Utilized lower-level feature fusion to optimize spatial weights of upper-level features, enabling cross-modality and cross-scale complementarity.
• Combined Channel-Space Joint Attention (CBAM) and self-attention models at the back-end for final feature fusion.

Main Results:

• Comparative experiments on multiple RGB-FIR public datasets demonstrated significant improvements in pedestrian detection.
• The proposed CCAM-based framework outperformed current RGB-FIR multimodal YOLO models in accuracy and robustness.
• The method effectively enhances the representation ability of raw features through cross-modality and cross-scale optimization.

Conclusions:

• The novel RGB-FIR multimodal backbone network with CCAM offers a superior approach to pedestrian detection.
• This framework addresses limitations in existing methods by integrating front-end feature extraction with advanced fusion techniques.
• The findings contribute to more reliable pedestrian detection systems for autonomous driving applications.