Weakly Aligned Feature Fusion for Multimodal Object Detection.

Area of Science:

• Computer Vision
• Machine Learning
• Sensor Fusion

Background:

• Multimodal data (e.g., RGB, thermal, depth) improve object detection accuracy.
• Position shifts between modalities hinder deep learning fusion and training.
• Existing methods struggle with misalignment in real-world scenarios.

Purpose of the Study:

• To propose a general multimodal object detector, aligned region CNN (AR-CNN), addressing the position shift problem.
• To enhance the robustness and accuracy of object detection using aligned multimodal features.
• To introduce a novel multimodal dataset labeling strategy.

Main Methods:

• A region feature (RF) alignment module with adjacent similarity constraint predicts and corrects cross-modal position shifts.
• A region of interest (RoI) jitter strategy improves robustness against unexpected alignment variations.
• A feature reweighting method adaptively fuses multimodal features, prioritizing reliable information.
• A new multimodal labeling approach, KAIST-Paired, is introduced.

Main Results:

• The proposed AR-CNN effectively tackles the position shift problem in multimodal object detection.
• Experiments demonstrate significant improvements in accuracy and robustness across various datasets (2D, 3D, RGB-T, RGB-D).
• The method shows strong performance in aligning cross-modal region features.

Conclusions:

• The aligned region CNN (AR-CNN) offers a robust and effective solution for multimodal object detection with position shifts.
• The developed alignment and fusion strategies enhance deep learning model performance on misaligned data.
• The KAIST-Paired dataset and AR-CNN contribute to advancing research in real-world multimodal perception.